Find Last Record in Chain - a Customer Merge Process

Viewed 64

I am importing customer data from a another vendor's system and we have merge processes that we use to identify potential duplicate customer accounts and them merge them if they meet certain criteria - like same first name, last name, SSN and DOB. In this process, I am seeing where we are creating chains - for instance, Customer A is merged to Customer B who is then merged to Customer C.

What I am hoping to do it to identify these chains and update the customer record to point to the last record in the chain. So in my example above, Customer A and Customer B would both have Customer C's id in their merged To field.

CustID FName   LName     CustStatusType  isMerged  MergedTo
1      Kevin   Smith     M               1         2 
2      Kevin   Smith     M               1         3
3      Kevin   Smith     M               1         4
4      Kevin   Smith     O               0         NULL
5      Mary    Jones     O               0         NULL
6      Wyatt   Earp      M               1         7
7      Wyatt   Earp      O               1         NULL
8      Bruce   Wayn      M               1         10
9      Brice   Wayne     M               1         10
10     Bruce   Wane      M               1         11
11     Bruce   Wayne     O               1         NULL

CustStatusType indicates if the customer account is open ("O") or merged ("M"). And then we have an isMerged field as a BIT field that indicates whether the account has been merged and finally the MergedTo field that indicates what customer account the record was merged to.

With the example provided, what I would like to achieve would to have the CustID records of 1 & 2 have their MergedTo record set to 3 - while CustID 3 could either be updated or left as is. For Cust IDs 4, 5, and 6 - these records are find and do not need to be updated. But on Cust IDs 8 - 10, I would like these records to be set to 11 - like the table below.

CustID FName   LName     CustStatusType  isMerged  MergedTo
1      Kevin   Smith     M               1         4 
2      Kevin   Smith     M               1         4
3      Kevin   Smith     M               1         4
4      Kevin   Smith     O               0         NULL
5      Mary    Jones     O               0         NULL
6      Wyatt   Earp      M               1         7
7      Wyatt   Earp      O               1         NULL
8      Bruce   Wayn      M               1         11
9      Brice   Wayne     M               1         11
10     Bruce   Wane      M               1         11
11     Bruce   Wayne     O               1         NULL

I haven't been able to figure out how to achieve this with TSQL - suggestions?

Test Data:

DROP TABLE IF EXISTS #Customers;

CREATE TABLE #Customers
    (
        CustomerID INT ,
        FirstName VARCHAR (25) ,
        LastName VARCHAR (25) ,
        CustomerStatusTypeID VARCHAR (1) ,
        isMerged BIT ,
        MergedTo INT
    );
INSERT INTO #Customers
VALUES ( 1, 'Kevin', 'Smith', 'M', 1, 2 ) ,
       ( 2, 'Kevin', 'Smith', 'M', 1, 3 ) ,
       ( 3, 'Kevin', 'Smith', 'M', 1, 4 ) ,
       ( 4, 'Kevin', 'Smith', 'O', 0, NULL ) ,
       ( 5, 'Mary', 'Jones', 'O', 0, NULL ) ,
       ( 6, 'Wyatt', 'Earp', 'M', 1, 7 ) ,
       ( 7, 'Wyatt', 'Earp', 'O', 1, NULL ) ,
       ( 8, 'Bruce', 'Wayn', 'M', 1, 10 ) ,
       ( 9, 'Brice', 'Wayne', 'M', 1, 10 ) ,
       ( 10, 'Bruce', 'Wane', 'M', 1, 11 ) ,
       ( 11, 'Bruce', 'Wayne', 'O', 1, NULL );

SELECT *
FROM   #Customers;

DROP TABLE #Customers;
3 Answers

For your example soundex() seems good enough. It returns a code, that is based on the word's pronunciation in English. Use it on the first and last name to join the customer table and a subquery which queries the customer table adding the row_number() partitioned by the Soundex of the names and order descending by the ID -- to number the "latest" record with 1. For the join condition use the Soundex of the names, a row number of 1 and of course inequality of the IDs.

UPDATE c1
       SET c1.mergedto = x.customerid
       FROM #customers c1
            LEFT JOIN (SELECT c2.customerid,
                              soundex(c2.firstname) sefn,
                              soundex(c2.lastname) seln,
                              row_number() OVER (PARTITION BY soundex(c2.firstname),
                                                              soundex(c2.lastname)
                                                 ORDER BY c2.customerid DESC) rn
                              FROM #customers c2) x
                      ON x.sefn = soundex(c1.firstname)
                         AND x.seln = soundex(c1.lastname)
                         AND x.rn = 1
                         AND x.customerid <> c1.customerid;

db<>fiddle

I don't really get the concept behind the customerstatustypeid and ismerged columns. As what I understand, they're all derived from whether mergedto is null or not. But the sample data neither the expected result doesn't support that. But as these columns apparently don't change between your sample input and output I guess it's alright, that I just left them alone.

If Soundex proves to be insufficient for your needs you may want to look for other string distance metrics, like the Levenshtein distance. AFAIK there's no implementation of that included in SQL Server but search engines may spit out implementations by third parties or maybe there's something that can used via CLR. Or you roll your own, of course.

Below query finds the latest CustomerID which is match to each customer and returns the id in Ref column

select *
, Ref = (select top 1 CustomerID from #Customers where soundex(FirstName) = soundex(ma.FirstName) and soundex(LastName) = soundex(ma.LastName) order by CustomerID desc)
from #Customers ma

using below update, you can update MergedTo column

;with ct as (
select *
, Ref = (select top 1 CustomerID from #Customers where soundex(FirstName) = soundex(ma.FirstName) and soundex(LastName) = soundex(ma.LastName) order by CustomerID desc)
from #Customers ma
)
update c1
set c1.MergedTo = iif(c1.CustomerID = ct.Ref, null, ct.Ref)
from #Customers c1
    inner join ct on ct.CustomerID = c1.CustomerID

Final data in Customer table after update

Final Result

Recursion can be used for this:

WITH CTE as
(
  SELECT P.CustomerID, P.MergedTo, CAST(P.CustomerID AS VarChar(Max)) as Levels
  FROM #Customers P
  WHERE P.MergedTo IS NULL

  UNION ALL

  SELECT  P1.CustomerID, P1.MergedTo, M.Levels + ', ' + CAST(P1.CustomerID AS VarChar(Max)) 
  FROM #Customers P1  
  INNER JOIN CTE M ON M.CustomerID = P1.MergedTo
 )
SELECT
       CustomerID
     , MergedTo 
     , x           -- "end of chain"
     , Levels
FROM CTE
CROSS APPLY (
    SELECT LEFT(levels,charindex(',',levels+',')-1) x
    ) a
WHERE MergedTo IS NOT NULL

Result:

+----+------------+----------+----+------------+
|    | CustomerID | MergedTo | x  |   levels   |
+----+------------+----------+----+------------+
|  1 |         10 |       11 | 11 | 11, 10     |
|  2 |          8 |       10 | 11 | 11, 10, 8  |
|  3 |          9 |       10 | 11 | 11, 10, 9  |
|  4 |          6 |        7 |  7 | 7, 6       |
|  5 |          3 |        4 |  4 | 4, 3       |
|  6 |          2 |        3 |  4 | 4, 3, 2    |
|  7 |          1 |        2 |  4 | 4, 3, 2, 1 |
+----+------------+----------+----+------------+

Note the string levels is formed by the recursion, and in the manner this is concatenated the first part will be the "end of chain" (see column x). That first part is extracted using a cross apply although using an apply isn't essential.

Available as a demo

Related