Oracle - Hash Join Buffered Very Slow

Question

I'm facing an issue during the merging of some data.

I've two tables:

CREATE TABLE tmp_table
(
  TROWID ROWID NOT NULL
, NEW_FK1 NUMBER(10)
, NEW_FK2 NUMBER(10)
, CONSTRAINT TMP_TABLE_PK_1 PRIMARY KEY
  (
    TROWID
  )
  ENABLE
)
CREATE UNIQUE INDEX TMP_TABLE_PK_1 ON tmp_table (TROWID ASC)

CREATE TABLE my_table
(
  M_ID NUMBER(10) NOT NULL
, M_FK1 NUMBER(10)
, M_FK2 NUMBER(10)
, M_START_DATE DATE NOT NULL
, M_END_DATE DATE
, M_DELETED NUMBER(1) NOT NULL
, M_CHECK1 NUMBER(1) NOT NULL
, M_CHECK2 NUMBER(1) NOT NULL
, M_CHECK3 NUMBER(1)
, M_CREATION_DATE DATE
, M_CREATION_USER NUMBER(10)
, M_UPDATE_DATE DATE
, M_UPDATE_USER NUMBER(10)

, CONSTRAINT MY_TABLE_PK_1 PRIMARY KEY
  (
    M_ID
  )
  ENABLE
)
CREATE UNIQUE INDEX TMP_TABLE_PK_1 ON my_table (M_ID ASC)
CREATE INDEX TMP_TABLE_IX_1 ON my_table (M_UPDATE_DATE ASC, M_FK2 ASC)
CREATE INDEX TMP_TABLE_IX_2 ON my_table (M_FK1 ASC, M_FK2 ASC)

The tmp_table is a temporary table where i stored only the records and informations that will be updated in my_table. That means tmp_table.TROWID is the rowid of my_table row that should be merged.

Total merged records should be: 94M on a total anount of my_table of 540M.

The query:

MERGE /*+parallel*/ INTO my_table m
USING (SELECT /*+parallel*/ * FROM tmp_table) t
  ON (m.rowid = t.TROWID)
WHEN MATCHED THEN
  UPDATE SET m.M_FK1 = t.M_FK1 , m.M_FK2 = t.M_FK2 , m.M_UPDATE_DATE = trunc(sysdate)
  , m.M_UPDATE_USER = 0 , m.M_CREATION_USER = 0

The execution plan is:

             Operation            |   Table   |  Estimated Rows |
MERGE STATEMENT                   |           |                 |
- MERGE                           | my_table  |                 |
-- PX CORDINATOR                  |           |                 |
--- PX SENDER                     |           |                 |
---- PX SEND QC (RANDOM)          |           |             95M |
----- VIEW                        |           |                 |
------ HASH JOIN BUFFERED         |           |             95M |
------- PX RECEIVE                |           |             95M |
-------- PX SEND HASH             |           |             95M |
--------- PX BLOCK ITERATOR       |           |             95M |
---------- TABLE ACCESS FULL      | tmp_table |             95M |
------- PX RECEIVE                |           |            540M |
-------- PX SEND HASH             |           |            540M |
--------- PX BLOCK ITERATOR       |           |            540M |
---------- TABLE ACCESS FULL      | my_table  |            540M |

In the above plan the most expensive op is the HASH JOIN BUFFERED. For the two full scans I've seen that not require more of 5/6 minutes, instead for the hash join after 2h have reach 1% of the execution.

I've no idea how require that much time; any suggesitons?

EDIT

-----------------------------------------------------------------------------------------------------------
| Id  | Operation            | Name                       | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------------------
|   0 | MERGE STATEMENT      |                            |    94M|  9719M|       |  3027K  (2)| 10:05:29 |
|   1 |  MERGE               | my_table                   |       |       |       |            |          |
|   2 |   VIEW               |                            |       |       |       |            |          |
|*  3 |    HASH JOIN         |                            |    94M|  7109M|  3059M|  3027K  (2)| 10:05:29 |
|   4 |     TABLE ACCESS FULL| tmp_table                  |    94M|  1979M|       |   100K  (2)| 00:20:08 |
|   5 |     TABLE ACCESS FULL| my_table                   |   630M|    33G|       |   708K  (3)| 02:21:48 |
-----------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - access("tmp_table"."TROWID"="m".ROWID)

user_1864029 · Answer

You could do a number of things. Please check whether they are beneficial for your situation, as milage will vary.

1) Use only the columns of the target table you touch (by select or update):

MERGE 
 INTO (SELECT m_fk1, m_fk2, m_update_date, m_update_user, m_creation_user 
         FROM my_table) m

2) Use only the columns of the source table you need. In your case that's all columns, so there won't be any benefit:

MERGE 
 INTO (...) m
USING (SELECT trowid, new_fk1, new_fk2 FROM tmp_table) t

Both 1) and 2) will reduce the size of the storage needed for a hash join and will enable the optimizer to use an index over all the columns if available.

3) In your special case with ROWIDs, it seems to be very beneficial (at least in my tests) to sort the source table. If you sort the rowids, you will likely update rows in the same physical block together, which may be more performant:

MERGE 
 INTO (...) m
USING (SELECT ... FROM tmp_table ORDER BY trowid) t

4) As your source table is quite large, I guess that it's tablespace is distributed over a couple of datafiles. You can check this with the query

SELECT f, count(*) FROM (
  SELECT dbms_rowid.rowid_relative_fno(trowid) as f from tmp_table
) GROUP BY f ORDER BY f;

If your target table uses more than a handful of datafiles, you could try to partition your temporary table by datafile:

CREATE TABLE tmp_table (
  TROWID  ROWID NOT NULL
, NEW_FK1 NUMBER(10)
, NEW_FK2 NUMBER(10)
, FNO     NUMBER
) PARTITION BY RANGE(FNO) INTERVAL (1) (
  PARTITION p0 VALUES LESS THAN (0)
);

You can fill the column FNO with the following statement:

 dbms_rowid.rowid_relative_fno(rowid)

Now you can update datafile by datafile, reducing the required memory for the hash join. Get the list of file numbers with

SELECT DISTINCT fno FROM tmp_table;
14
15
16
17

and run the updates file by file:

MERGE 
 INTO (SELECT ... FROM my_table) m
USING (SELECT ... FROM tmp_table PARTITION FOR (14) ORDER BY trowid) t

and next PARTITION FOR (15) etc. The file numbers will obviously be different on your system.

5) Finally, try to use nested loops instead of a hash join. Usually the optimizer picks the better join plan, but I cannot resist trying it out:

MERGE /*+ USE_NL (m t) */
 INTO (SELECT ... FROM my_table) m
USING (SELECT ... FROM tmp_table partition for (14) ORDER BY trowid) t
  ON (m.rowid = t.TROWID)

Oracle - Hash Join Buffered Very Slow

1 Answers