Salesforce API totalSize does not always match to the number of rows returned

user_2700344 0
Viewed 178

I am downloading data from salesforce API. The table is big and I am using query_more to get full result. The problem is totalSize returned in result does not always match to the actual number of records returned. It can be few records less, sometimes it matches exactly.

Is totalSize an estimation only or it should match exactly with the number of records returned? Please help to clarify this issue. Code of method returning query result:

sf_conn = self.get_conn()
result = sf_conn.query(self.soql)

totalSize = result['totalSize'] # This does not always match !!!
logging.info('Total result set size: %s', totalSize )

while True:
    yield result
    if result['done']:
        break
    result = sf_conn.query_more(result['nextRecordsUrl'], identifier_is_url=True)
airflow salesforce python
1 Answers

Figured it out finally. I hope this answer will be useful for those who do not know yet how records get deleted in Salesforce.

When record gets deleted in Salesforce, it is being put in the Recycle bin. Such records have isDeleted=true. After 15 days these records will disappear completely.

sf_conn.query and sf_conn.query_more methods have parameter include_deleted, default False.

By default these methods do not return records having isDeleted=True. If the job running long time, some records get deleted during execution and are not fetched. If you doing data quality checks to check that processed records count equal to fetched records count from API, better do not rely on TotalSize field and count the number of records actually fetched. See also documentation about how records get deleted in Salesforce.

user_27003440
Related