execute_values from psycopg2.extras is throwing "IndexError: list index out of range" when I try to load data

user_2909052 0
Viewed 404

I'm pulling in a collection from mongodb and trying to load it to postgresql making use of psycopg2.extras.execute_values

query = '''INSERT INTO table_name
(eventID,companyName,eventSlug,eventType,
eventTerm,eventYear,eventName,sessionId,_id,videoSrc,createdAt) 
VALUES %s'''

The values list contains ~95k elements like:

[1,
 u'foo',
 u'bar',
 u'baz',
 u'T3',
 u'2020',
 u'qux',
 u'quux',
 u'quuz',
 u'corge',
 datetime.datetime(2020, 9, 18)]

So I run:

execute_values(cur, query, values)

And for one of my 3 collections I'm loading, I get

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-291-64222b53f241> in <module>()
----> 1 execute_values(cur, query, values[:5])

/Users/<me>/.virtualenvs/ipython/lib/python2.7/site-packages/psycopg2/extras.pyc in execute_values(cur, sql, argslist, template, page_size, fetch)
   1276         parts = pre[:]
   1277         for args in page:
-> 1278             parts.append(cur.mogrify(template, args))
   1279             parts.append(b',')
   1280         parts[-1:] = post

IndexError: list index out of range

I've tried making the page_size argument to be greater than the length of my values list but that didn't work, and I've tried only running the function for a small subset of the values list values[:5]

The odd thing is that for 2/3 collections I load in, there are no problems. This one gives me IndexError no matter what I try. Am I missing something about the execute_values function?

psycopg2 postgresql python
1 Answers

User Error:

Because these records were coming from Mongo, there was no guarantee that each record would have all fields. When psycopg2 tried to expand the values in each record to fit the INSERT query, these records with missing fields would cause the error.

user_29090520
Related