How to update large table with millions of rows in SQL Server?

I encountered this thread yesterday and wrote a script based on the accepted answer. It turned out to perform very slowly, taking 12 hours to process 25M of 33M rows. I wound up cancelling it this morning and working with a DBA to improve it.

The DBA pointed out that the is null check in my UPDATE query was using a Clustered Index Scan on the PK, and it was the scan that was slowing the query down. Basically, the longer the query runs, the further it needs to look through the index for the right rows.

The approach he came up with was obvious in hind sight. Essentially, you load the IDs of the rows you want to update into a temp table, then join that onto the target table in the update statement. This uses an Index Seek instead of a Scan. And ho boy does it speed things up! It took 2 minutes to update the last 8M records.

Batching Using a Temp Table

SET NOCOUNT ON

DECLARE @Rows INT,
        @BatchSize INT,
        @Completed INT,
        @Total INT,
        @Message nvarchar(max)

SET @BatchSize = 4000
SET @Rows = @BatchSize
SET @Completed = 0

-- #targetIds table holds the IDs of ALL the rows you want to update
SELECT Id into #targetIds 
FROM TheTable 
WHERE Foo IS NULL 
ORDER BY Id

-- Used for printing out the progress
SELECT @Total = @@ROWCOUNT

-- #batchIds table holds just the records updated in the current batch
CREATE TABLE #batchIds (Id UNIQUEIDENTIFIER);

-- Loop until #targetIds is empty
WHILE EXISTS (SELECT 1 FROM #targetIds)
BEGIN
    -- Remove a batch of rows from the top of #targetIds and put them into #batchIds
    DELETE TOP (@BatchSize)
    FROM #targetIds
    OUTPUT deleted.Id INTO #batchIds  

    -- Update TheTable data
    UPDATE t
    SET Foo = 'bar'
    FROM TheTable t
    JOIN #batchIds tmp ON t.Id = tmp.Id
    WHERE t.Foo IS NULL
    
    -- Get the # of rows updated
    SET @Rows = @@ROWCOUNT

    -- Increment our @Completed counter, for progress display purposes
    SET @Completed = @Completed + @Rows

    -- Print progress using RAISERROR to avoid SQL buffering issue
    SELECT @Message = 'Completed ' + cast(@Completed as varchar(10)) + '/' + cast(@Total as varchar(10))
    RAISERROR(@Message, 0, 1) WITH NOWAIT    

    -- Quick operation to delete all the rows from our batch table
    TRUNCATE TABLE #batchIds;
END

-- Clean up
DROP TABLE IF EXISTS #batchIds;
DROP TABLE IF EXISTS #targetIds;

Batching the slow way, do not use!

For reference, here is the original slower performing query:

SET NOCOUNT ON

DECLARE @Rows INT,
        @BatchSize INT,
        @Completed INT,
        @Total INT

SET @BatchSize = 4000
SET @Rows = @BatchSize
SET @Completed = 0
SELECT @Total = COUNT(*) FROM TheTable WHERE Foo IS NULL

WHILE (@Rows = @BatchSize)
BEGIN

    UPDATE t
    SET Foo = 'bar'
    FROM TheTable t
    JOIN #batchIds tmp ON t.Id = tmp.Id
    WHERE t.Foo IS NULL

SET @Rows = @@ROWCOUNT
SET @Completed = @Completed + @Rows
PRINT 'Completed ' + cast(@Completed as varchar(10)) + '/' + cast(@Total as varchar(10))

END

I want share my experience. A few days ago I have to update 21 million records in table with 76 million records. My colleague suggested the next variant. For example, we have the next table 'Persons':

Id | FirstName | LastName | Email            | JobTitle
1  | John      |  Doe     | abc1@abc.com     | Software Developer
2  | John1     |  Doe1    | abc2@abc.com     | Software Developer
3  | John2     |  Doe2    | abc3@abc.com     | Web Designer

Task: Update persons to the new Job Title: 'Software Developer' -> 'Web Developer'.

1. Create Temporary Table 'Persons_SoftwareDeveloper_To_WebDeveloper (Id INT Primary Key)'

2. Select into temporary table persons which you want to update with the new Job Title:

INSERT INTO Persons_SoftwareDeveloper_To_WebDeveloper SELECT Id FROM
Persons WITH(NOLOCK) --avoid lock 
WHERE JobTitle = 'Software Developer' 
OPTION(MAXDOP 1) -- use only one core

Depends on rows count, this statement will take some time to fill your temporary table, but it would avoid locks. In my situation it took about 5 minutes (21 million rows).

3. The main idea is to generate micro sql statements to update database. So, let's print them:

DECLARE @i INT, @pagesize INT, @totalPersons INT
    SET @i=0
    SET @pagesize=2000
    SELECT @totalPersons = MAX(Id) FROM Persons

    while @i<= @totalPersons
    begin
    Print '
    UPDATE persons 
      SET persons.JobTitle = ''ASP.NET Developer''
      FROM  Persons_SoftwareDeveloper_To_WebDeveloper tmp
      JOIN Persons persons ON tmp.Id = persons.Id
      where persons.Id between '+cast(@i as varchar(20)) +' and '+cast(@i+@pagesize as varchar(20)) +' 
        PRINT ''Page ' + cast((@i / @pageSize) as varchar(20))  + ' of ' + cast(@totalPersons/@pageSize as varchar(20))+'
     GO
     '
     set @i=@i+@pagesize
    end

After executing this script you will receive hundreds of batches which you can execute in one tab of MS SQL Management Studio.

4. Run printed sql statements and check for locks on table. You always can stop process and play with @pageSize to speed up or speed down updating(don't forget to change @i after you pause script).

5. Drop Persons_SoftwareDeveloper_To_AspNetDeveloper. Remove temporary table.

Minor Note: This migration could take a time and new rows with invalid data could be inserted during migration. So, firstly fix places where your rows adds. In my situation I fixed UI, 'Software Developer' -> 'Web Developer'. More about this method on my blog https://yarkul.com/how-smoothly-insert-millions-of-rows-in-sql-server/

This is a more efficient version of the solution from @Kramb. The existence check is redundant as the update where clause already handles this. Instead you just grab the rowcount and compare to batchsize.

Also note @Kramb solution didn't filter out already updated rows from the next iteration hence it would be an infinite loop.

Also uses the modern batch size syntax instead of using rowcount.

DECLARE @batchSize INT, @rowsUpdated INT
SET @batchSize = 1000;
SET @rowsUpdated = @batchSize; -- Initialise for the while loop entry

WHILE (@batchSize = @rowsUpdated)
BEGIN
    UPDATE TOP (@batchSize) TableName
    SET Value = 'abc1'
    WHERE Parameter1 = 'abc' AND Parameter2 = 123 and Value <> 'abc1';

    SET @rowsUpdated = @@ROWCOUNT;
END