SQL Server constraint enforcement being violated for split seconds

Question

I have a table on premise that is about 21 million rows with a primary key constraint and when I search that table, there are no duplicates. This table is in an OLTP application database that is constantly moving.

I have the exact same table in Azure which has the same primary key constraint. This table is not an application table, it's just a copy of the one that is on-premise (the goal is to use this one for ad hoc queries, as a source for other systems, etc.).

When I use Azure Data Factory to select all_columns from table on premise to the table in Azure, it returns a violation of the primary key constraint. No matter how many times I run this data factory pipeline, it comes back with a primary key violation for duplicate keys (the keys are always changing though).

So I dropped the primary key constraint in Azure and ran the pipeline again, and sure enough, duplication exists.

Upon investigation, it appears that the on-premise database is doing an insert new record then update the old record to inactivate it. So for a fraction of a second, there are two active rows that ADF is grabbing to then try to insert into the table in Azure which of course fails because of duplicate primary keys.

Now to the best of my knowledge, this shouldn't be possible. You can't insert a new row that violates the primary key constraint. But ADF seems to be grabbing all the data and some of those rows are mid-flight where the insert has happened and the update to inactivate the old row hasn't happened yet.

For those that are curious, the insert happens and the update of the old row happens within less than a second... it's typically 10-20 microseconds. I don't know how this is possible and I don't know how to fix it (because I can't modify the application code). The database for the on-premise database is a SQL Server 2000 database and Azure SQL is an Azure SQL database.

user_8651601 · Answer

Try with readpast hint. It should not select any rows in locking state.

SELECT * FROM yourtable WITH (readpast)

Since you have create_date and updated_date column then you can select rows older than 5 seconds to avoid duplication.

select * from yourtable where created_date<=dateadd(second,-5,getdate()) and updated_date<=dateadd(second,-5,getdate());

user_15997690 · Answer

Need to enable the Fault tolerance in a Pipeline Azure Data Factory

Copy data from a Source SQL to a Sink SQL database. A primary key is defined in the sink SQL database, but no such primary key is defined in the source SQL server. The duplicated rows that exist in the source cannot be copied to the sink. Copy activity copies only the first row of the source data into the sink. The subsequent source rows that contain the duplicated primary key value are detected as incompatible and are skipped.

To configure Json Definition skip the incompatible rows in copy activity "enableSkipIncompatibleRow": true

Please Refer: https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-fault-tolerance

If possible to modify your application, need to check the Primary key constraint before insert or update using EXISTS() function.

Example:

IF EXISTS(SELECT * FROM Table_Name WHERE primary key condition)
BEGIN
UPDATE Table_Name 
SET Col_Name= value
WHERE condition
END
ELSE
BEGIN
INSERT INTO Table_Name  ( col_Name1,col_Name2,,.. )
VALUES ( ‘’,’’,’’,….)
END

SQL Server constraint enforcement being violated for split seconds

2 Answers