I am trying to automate a daily download of MailChimp data. I have written a script that downloads successfully using API v3.0 through both regular/synchronous and batch/asynchronous requests. Unfortunately, the process is so slow that I can't download the dataset in just a day. Is there a faster way to get data from MailChimp?
If I only download records added or changed in the last 24 hours it's typically not an issue. Unfortunately, we sometimes do batch updates of our subscriber records in anticipation of a large mailing, like adding a unique coupon code to each user. This results in a large number of records being updated in a short period of time, and it exceeds the number of records that can be downloaded in 24 hours.
I am running my code in PowerShell but I do not believe the code is the fault - even the asynchronous calls take too long, and that processing is all on MailChimp's end.
Using synchronous calls it takes 200 milliseconds per user to download data. As the number of requests increases this seems to slow down, indicating throttling may be happening. Without throttling that would still make 37 hours to download one of our larger lists.
Milliseconds Per Member Retrieval Times (Synchronous Method)
| Member Records | 245 | 684 | 28,336 |
|---|---|---|---|
| Member List Count | 34.2 | 36.3 | 41.1 |
| Member List Details | 165.3 | 148.5 | 439.1 |
Less than 30,000 records took 2 hours and 49 minutes using synchronous calls.
I rewrote my code to use batch / asynchronous calls and it was so much slower:
Retrieval Times (Asynchronous Method)
| Records | 28,370 |
|---|---|
| Preprocessing Phase | 69 minutes |
| Started Phase | 921 minutes |
| Finalizing Phase | 69 minutes |
| Total Time | 1,059 minutes, 17 hours 39 minutes |
For those that may think I'm mad and it can't possibly be this slow, here's their own output (note I ran this back in November, but I'm thinking of banging my head against the wall again):
| status | finished |
| total_operations | 28370 |
| finished_operations | 28370 |
| errored_operations | 0 |
| submitted_at | 2021-11-24T17:06:23+00:00 |
| completed_at | 2021-11-25T10:49:24+00:00 |
So, how does one download hundreds of thousands of records from MailChimp in less than 24 hours? I know you can do a data export through the interface, but I don't think they intend that to be a regular occurence and I don't want to have to automate accessing their web UI, triggering events, clicking downloads, etc.
Note: I'm not open to discussions about whether we need the data in our reporting DB and should instead pull needed records real-time. I am open to discussions about leaving MailChimp but not in this venue.