SSIS Runs perfectly on a remote server(Greenplum) Datalake but takes 8+ hours

user_14094599 0
Viewed 34

SSIS Package performs the ETL on a remote server (Greenplum envt). It runs fine but takes 8+ hours to complete. Data on the Remote server's interaction tables are massive (~ 1 Billion rows each). Is there a way or any option available on SSIS specifically for the massive amount of data?

Remote Server: Data Lake (Greeplum) PS: I cant Schedule my query on Data lake itself due to the company policy. but if run the same script on Data lake manually it takes approx 1hr 20 mins to complete the Job.

Thank you!

greenplum data-lake ssis
1 Answers

How does SSIS perform ETL? does it run insert into .. values ...? if so, the performance is expected to be bad because the insert overhead is high. there are several parameters that could help(reference https://greenplum.org/oltp-workload-performance-improvement-in-greenplum-6/):

  • gp_enable_global_deadlock_detector
  • checkpoint_segments

However, the suggested why to do ETL is through gpload/gpfdist (or gpss)

user_110037870
Related