Ways to do bulk import of data in Netsuite efficiently?

user_3475468 0
Viewed 581

I am exploring ways to perform bulk import customer data to Netsuite efficiently. I figured out multiple ways like CSV import, soap APIs, rest web services, suite script, etc.

I tried all these methods and got a throughput of 1.3 requests per second by a single thread, when using a concurrency level of 4 they are giving a throughput of 4.7 requests per second.

As per this throughput, if we have to migrate millions of records say 10M then it will take around 24 days to complete this migration activity.

According to my analysis, I am able to conclude that whatever bulk import option I choose, Netsuite will give me a throughput of 1 request per second per thread (standard account concurrency limit is 5). If a customer needs to increase throughput customer have to increase premium/plus account counts.

I found the concurrency governance policy from Netsuite which says I can increase the concurrency limits based on account type and premium supports.

Netsuite also says suite script 2.x supports map-reduce but I didn't find that also relevant in this case to increase throughput without increasing concurrency limits. Though CSV Import using map-reduce may give better results but not sure about that. But CSV Import has the limitation of 25000 lines or a 50 MB size limit per file, and CSV Import efficiency also does not get beyond the concurrency support for that account.

So my questions are :

  1. CSV Import can go beyond the concurrency level of that particular account?
  2. Is there any way to complete 10M records migrations in 2-5 days without worrying about the concurrency limit of the account.
  3. Does the Concurrency limit is the controlling factor here?
  4. Am I missing something in my analysis?
netsuite bulk import csv migration
3 Answers

I can not give exact answers to your questions, but I really think that the solution is using Map/Reduce and not use CSV import within (not sure though what did you mean by: Though CSV Import using map-reduce may give better results but not sure about that.).

By design, it is easy to make a Map/Reduce script which will read CSV file line by line and then create the records using Scripting logic. The strength of Map/Red is that there is no limit to the number of Data you can treat, and also it is, by Netsuite Design, the best tool to treat massive updates.

Also Map/Red is designed to better handle concurrent process; though it is still limited concurrency, by default, and maybe not enough for you to get the 2-5 days goal unless you by more "processors".

user_60538790

CSV Import are likely the way to go. Whether you can squeeze out the performance you need is another matter.

The most performance can be had if you are importing 1 record per line. For multi-file imports you can still get better peformance follwing skipping #1 below (I actually haven't tried skipping #1 for multi-line imports but you should experiment with that if you are importing transactions. )

When you set up your import templates do the following after on the Import Options page:

  1. Check 'Use Multi-Threading'
  2. Select a queue. You can select one queue for each of your Suite Cloud plus licenses. Utlimately achieving your goals may depend on purchasing enough Suite Cloud Plus licenses to meet your performance needs.
  3. Uncheck 'Run Server Scripts and Trigger Workflows' Which means any logic for setting up your records needs to be done at the CSV prep level.

Triggering your CSV imports may be done via scripts. For instance I have a RESTlet that receives files and then runs a configured script. That script receives the file id as a parameter and then processes the file. In your case the script could iterate the lines of the received file see N/file.lines.iterator() into CSV files of 25k lines or less and submit them via N/task.CsvImportTask using a round robin of the CSV import scripts you set up above.

user_38060170

For bulk import I strongly recommend the soap api approach to loading, but you are right that it will require you to have multiple suitecloud plus licenses for the added concurrency:

To answer your questions:

  1. CSV Import can go beyond the concurrency level of that particular account? No.
  2. Is there any way to complete 10M records migrations in 2-5 days without worrying about the concurrency limit of the account. No
  3. Does the Concurrency limit is the controlling factor here? Concurrency, plus how you implement the integration (using Boomi, or custom, or MuleSoft).
  4. Am I missing something in my analysis? no
user_60431700
Related