6 Replies Latest reply on Feb 22, 2021 9:50 PM by Dhananjay Singh

    ElasticSearch Index - Incomplete

    Venkatesh Ramasamy Active Member



      I am implementing elasticSearch and testing it currently. I see an issue with some records when created via cleansePut are not searchable. This does not happen to all records. When looked at the Index on the elastic search server, I see that the parent object is missing and just the child objects exists. Say for example, we have Customer (First Name, Last Name and ID) as parent and there are address and email records as child records. When I check the database, the Customer record and the all children are properly created. However, the Index does not have Parent object. Also the '_Version' is also not same for the ones that are working. When I run the below on elastic search directly I get the below results, it's the same cleansePut calls (Total 5 cleansePut calls and one to set consolidation ind)  that created both the records, however, the index is different for both.



      1. Is there a way to troubleshoot the index creation process? has any one seen this behaviour?

      2. How can we Identify how may records have this malformed index and are not searchable? We have thousands of records created every day and few records has this issue. I came to know when the QA Team reported a record was not searchable by Name. So I do not know how may records has this issue.


      This is an example of the one that is working (all elements in the index are correct and searchable using the parent object):

      {"_index": "4d5354312e375053-search_customer","_type": "_doc","_id": "386050001","_version": 6,"found": true,"_source": {"boid": "386050001","url": "Search_Customer|386050001","CUSTOMER_NUMBER": "171239396","rowidObject": "386050001 ","lastRowidSystem": "COMET","hubStateInd": "1","Customer_Type_Code": "INDIVIDUAL","FULL_NAME": "NATHAN LYONS","FIRST_NAME": "NATHAN","consolidationInd": "4","LAST_NAME": "LYONS","Activity_Type_Code": "ORDER BILLING","Email": [{"consolidationInd": "4","Email": "nathonlyons@acb.com","hubStateInd": "1","boid": "99640001","lastRowidSystem": "COMET","url": "Search_Customer|386050001.Email|99640001"}],"Address": [{"State_Code": "WA","Address_Line1": "123 ADELIDE DR","consolidationInd": "4","boid": "361370001","Phone": [{"consolidationInd": "4","Area_Code": "789","hubStateInd": "1","boid": "46070001","lastRowidSystem": "COMET","Phone_Nbr_Extn": "0","Phone_Nbr": "5854685","url": "Search_Customer|386050001.Address|361370001.Phone|46070001"}],"lastRowidSystem": "COMET","Address_Type": "SHIP TO","Country": "USA","Postal_Code": "98127","City": "SEATTLE","rowidObject": "361370001 ","url": "Search_Customer|386050001.Address|361370001"},{"State_Code": "WA","Address_Line1": "123 ADELIDE DR","consolidationInd": "4","boid": "361370002","Phone": [{"consolidationInd": "1","Area_Code": "789","hubStateInd": "1","boid": "46070001","lastRowidSystem": "COMET ","Phone_Nbr_Extn": "0","Phone_Nbr": "5854685","url": "Search_Customer|386050001.Address|361370002.Phone|46070001"}],"lastRowidSystem": "COMET","Address_Type": "BILLING","Country": "USA","Postal_Code": "98127","City": "SEATTLE","rowidObject": "361370002 ","url": "Search_Customer|386050001.Address|361370002"}]}}


      Here's an example of the one that is missing the parent elements,

      {"_index": "4d5354312e375053-search_customer","_type": "_doc","_id": "386050005","_version": 3,"found": true,"_source": {"boid": "386050005","url": "Search_Customer|386050005.Address|361370005.Phone|46070014","Address": [{"boid": "361370005","Phone": [{"consolidationInd": "4","Area_Code": "523","hubStateInd": "1","boid": "46070014","lastRowidSystem": "COMET","Phone_Nbr_Extn": "0","Phone_Nbr": "6012345","url": "Search_Customer|386050005.Address|361370005.Phone|46070014"}],"url": "Search_Customer|386050005.Address|361370005.Phone|46070014"},{"State_Code": "VA","Address_Line1": "5 HENRY AVE","consolidationInd": "4","boid": "361370006","Phone": [{"consolidationInd": "1","Area_Code": "523","hubStateInd": "1","boid": "46070014","lastRowidSystem": "COMET ","Phone_Nbr_Extn": "0","Phone_Nbr": "6012345","url": "Search_Customer|386050005.Address|361370006.Phone|46070014"}],"lastRowidSystem": "COMET","Address_Type": "BILLING","Country": "USA","Postal_Code": "23601","City": "NEWPORT NEWS","rowidObject": "361370006 ","url": "Search_Customer|386050005.Address|361370006"}]}}


      Thanks in advance for your help.


      Venkatesh Ramasamy