INSERTING and RETURNING the ID of a new or existing record, need a better strategy in Postgres 12.5

Viewed 234

I'm trying to do something in Postgres 12.5 and am either missing a detail, or pursuing a flawed strategy entirely. I have a simple lookup table with some longer strings and extended detail. As data flows into the main table, I'd like to convert those strings to a simple int for the key. I figured that I could keep the lookups in a table, and then use INSERT....ON CONFLICT DO NOTHING...RETURNING to get these outcomes:

  • If the row does not exist in the lookup table, create it and return the new id. <-- This works.
  • If the row does exist, don't overwrite it and return the new id. <-- The id is not returned.

The second case is by far the more common. Chances are, the system will run for days or week between new values hitting the lookup table. I prefer to do the INSERT assertively to avoid a concurrency issue or unique conflict. However, RETURNING doesn't work the way I'd hoped. I checked the docs, and the behavior I'm seeing is clearly documented. And, thinking about it, makes a lot of sense. So, moving forward, does it make more sense to write a function where I can test the outcome and issue a SELECT id FROM pgconfig_target WHERE... to recover the existing id, or is some other strategy more straightforward?

For background, the main table is a push_audit_log summarizing insert calls from our clients with timing, record counts, and other details. There are going to be lots of these records. That's why I'd rather shift the strings over to another table.

Here's a short version of the lookup table's setup:

--------------------------------------
-- Define table
------------------------------------
BEGIN;

DROP TABLE IF EXISTS dba.pgconfig_target CASCADE;

CREATE TABLE IF NOT EXISTS dba.pgconfig_target (
    id                int2         GENERATED ALWAYS AS IDENTITY,
    schema_name       citext       NOT NULL DEFAULT NULL,
    target_name       citext       NOT NULL DEFAULT NULL,
    target_path       citext       NOT NULL DEFAULT NULL
);

ALTER TABLE dba.pgconfig_target
    OWNER TO user_change_structure;

COMMIT;

------------------------------------
-- Build indexes
------------------------------------
CREATE UNIQUE INDEX  pgconfig_target_unique_ix_btree
     ON dba.pgconfig_target (schema_name, target_name, target_path);

And here's the INSERT I'm experimenting with:

INSERT INTO pgconfig_target (schema_name, target_name, target_path) 
    VALUES ('hello','world','passthrough.hello.world')
    ON CONFLICT (schema_name,target_name,target_path) do nothing
    RETURNING id;

Thanks for suggestions.

#Later# I'd like to use a suggestion like the (ever helpful) GMB listed, but can't seem to get it to work on the first run. I'm curious why, as there's something about order or execution that I'm not understanding.

I tried out a little scratch function, and it works, but a pure-SQL statement solution seems a bit nicer, if possible. Here's the function:

CREATE OR REPLACE FUNCTION ascendco.pgconfig_target_add_if_missing (
     s_name   citext,
     t_name   citext,
     t_path   citext
)

RETURNS integer AS

$BODY$

DECLARE
   id_out integer = 0;

BEGIN

-- Insert the value, if it's missing.
INSERT INTO pgconfig_target  (schema_name, target_name, target_path)
                      VALUES (s_name, t_name, t_path)
                 ON CONFLICT DO NOTHING;

-- The ID should be there, either historically or becase it was just added.            
     SELECT id 
       FROM pgconfig_target 
      WHERE schema_name = s_name 
        AND target_name = t_name
        AND target_path = t_path
       INTO id_out;

 return id_out;
 
END

$BODY$
  LANGUAGE plpgsql;

A sample call such as this works:

   select * from pgconfig_target_add_if_missing ('hello', 'world', 'checking.it.out')

I'm using select * as I didn't sort out the syntax to make id an unambiguous result.

1 Answers

That just how on conflict works. When no row is inserted, nothing is returned.

A workaround is to rephrase the logic with a CTE: first insert (or do nothing), then select again from the table, using the input data for filtering.

with 
    data as (
        select * 
        from (values 
            ('hello','world','passthrough.hello.world')
        ) v(schema_name, target_name, target_path) 
    ),
    ins as (
        insert into pgconfig_target (schema_name, target_name, target_path) 
        select * 
        from data
        on conflict (schema_name, target_name, target_path) do nothing
    )
select c.id 
from pgconfig_target c
inner join data d using (schema_name, target_name, target_path)
Related