How to add 18000 product variations fast?

Alapaska 0
Viewed 56

I need to add 18k product variations to around 5k variable products less than in 20 minutes. Now my script looks like this:

product__batch_create_variations = []
product__batch_update_variations = []

###VARIABLES###

d_variables = pandas.read_csv("toimport3.csv")
d_variables['C'] = np.arange(len(d_variables))
d_variables.reset_index()
d_variables = d_variables.to_json(orient ='index')
variables_loaded = json.loads(d_variables)

###VARIATIONS###

d_variations = pandas.read_csv("toimport_variations.csv")
d_variations['C'] = np.arange(len(d_variations))
d_variations.head(10)
d_variations.reset_index()
d_variations = d_variations.to_json(orient ='index')
v_loaded = json.loads(d_variations)


def slugify(text):
    text = unidecode.unidecode(text).lower()
    return re.sub(r'[\W_]+', '-', text)

for variable_iterator in variables_loaded:
    variable_sku = str(variables_loaded[variable_iterator]["1"])
    brands_array = str(variables_loaded[variable_iterator]["34"]).split(",")
    cats = str(variables_loaded[variable_iterator]["33"]).split(",")
    
    short_desc = []
    images = []
    desc = []
    print("start for: " + variable_sku)
    for v_iterator in v_loaded:
#         try:
#             sku = str(variables_loaded[variable_iterator]["17"])
#             productlist_variable=wcapi.get("products", params={'sku':variable_sku}).json()
#             productid=productlist_variable[0].get("id")
#             productlist=wcapi.get('products'+'/'+str(productid)+"/"+"variations", params={'sku':sku}).json()
            
#             short_desc_ = productlist[0].get("short_description")
#             images_ = productlist[0].get("images")
#             desc_ = productlist[0].get("description")
#             print(sku)
        
#         except:
#             short_desc_ = []
#             images_ = []
#             desc_ = str(v_loaded[v_iterator]["10"]).replace("None","")
            
        if str(variable_sku) == str(v_loaded[v_iterator]["17"]):
            product_variation_data = {
                "name": str(v_loaded[v_iterator]["2"]),
                'sku': str(v_loaded[v_iterator]["saku"]),
                "regular_price": str(v_loaded[v_iterator]["14"]).replace(',','.'),
                'sale_price': str(v_loaded[v_iterator]["15"]).replace(',','.'),
                "stock_quantity": v_loaded[v_iterator]["8"],
                'weight': str(v_loaded[v_iterator]["11"]).replace('.',','),
                'dimensions': {
                 'length': str(v_loaded[v_iterator]["12"]).replace('.',','),
                 'width': str(v_loaded[v_iterator]["12"]).replace('.',','),
                 'height': str(v_loaded[v_iterator]["13"]).replace('.',',')
                },
                'short_description': short_desc,
                'images': images,
                'description': desc,
                'attributes': [
                    {'id': 9, 'name': 'Brändi', 'position': 0, 'visible': True, 'variation': False, 'options': [str(v_loaded[v_iterator]["26"]).replace("None","NULL")]},
                    {'id': 8, 'name': 'Väri', 'position': 1, 'visible': True, 'variation': True, 'option':str(v_loaded[v_iterator]["19"]).replace("None","")},
                    {'id': 6, 'name': 'Koko', 'position': 2, 'visible': True, 'variation': True, 'option':str(v_loaded[v_iterator]["28"]).replace("None","")},
                ],
                "meta_data": [
                   { 
                    "key": "_pointss",
                    "value": v_loaded[v_iterator]["31"]
                   },
                   { 
                    "key": "_transitory_sku",
                    "value": v_loaded[v_iterator]["saku"]
                   },
                   {
                    "key": "_text_field",
                    "value": v_loaded[v_iterator]["3"]
                   }
                ],
            }
            
            product__batch_create_variations.append(product_variation_data)

            
            product__batch_data = { 
                "create": product__batch_create_variations, 
                "update": product__batch_update_variations
            }
            
            productlist_variable=wcapi.get("products", params={'sku':variable_sku}).json()
            productid_variable=productlist_variable[0].get("id")
            response = wcapi.post('products/'+str(productid_variable)+"/variations/batch", product__batch_data)
            print(response)

So it reads csv with variables + csv with variations for variables and then adds variations to one variable product at a time in very long-lasting loop. I don't have 48 hours for this method, I need my product variations added in much faster pace, perhaps in one big batch.

How can I do that? Please help me.

v2.0: Here are some examples from csv. Paste this to your csv file, comma is the separator: (variations) saku,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36, 328105000442/Black/Green/Gum,variation,328105000442/990_10.5,"TM-2 Jones 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,449,381.65,"Thirty Two,Lumilautakengät",328105000442,Väri,Neutraali,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,10.5,0,1,0,19,"7315,7599,7321","7318,7319,7320",TM-2 Jones 21,Black/Green/Gum, 328105000442/Black/Green/Gum,variation,328105000442/990_11,"TM-2 Jones 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,449,381.65,"Thirty Two,Lumilautakengät",328105000442,Väri,Neutraali,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,11,0,1,0,19,"7315,7599,7321","7318,7319,7320",TM-2 Jones 21,Black/Green/Gum, 328105000442/Black/Green/Gum,variation,328105000442/990_12,"TM-2 Jones 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,449,341.24,"Thirty Two,Lumilautakengät",328105000442,Väri,Neutraali,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,12,0,1,0,19,"7315,7599,7321","7318,7319,7320",TM-2 Jones 21,Black/Green/Gum, 328105000442/Black/Green/Gum,variation,328105000442/990_6,"TM-2 Jones 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,449,381.65,"Thirty Two,Lumilautakengät",328105000442,Väri,Neutraali,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,6,0,1,0,19,"7315,7599,7321","7318,7319,7320",TM-2 Jones 21,Black/Green/Gum, 328105000442/Black/Green/Gum,variation,328105000442/990_8.5,"TM-2 Jones 21, Black/Green/Gum",1,7320,visible,taxable,1,1.2000,0,0,0.01,0.0000,0.0000,449,341.24,"Thirty Two,Lumilautakengät",328105000442,Väri,Musta,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,8.5,0,1,0,18,"7315,7599,7321","7318,7319,7320",TM-2 Jones 21,Black/Green/Gum, 328105000442/Black/Green/Gum,variation,328105000442/990_9,"TM-2 Jones 21, Black/Green/Gum",1,7320,visible,taxable,1,1.2000,0,0,0.01,0.0000,0.0000,449,341.24,"Thirty Two,Lumilautakengät",328105000442,Väri,Musta,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,9,0,1,0,18,"7315,7599,7321","7318,7319,7320",TM-2 Jones 21,Black/Green/Gum, 328105000449/Black/Green/Gum,variation,328105000449/990_10,"Jones MTB 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,569,443.82,"Thirty Two,Lumilautakengät",328105000449,Väri,Neutraali,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,10,0,1,0,19,"7315,7599,7321","7318,7319,7320",Jones MTB 21,Black/Green/Gum, 328105000449/Black/Green/Gum,variation,328105000449/990_10.5,"Jones MTB 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,569,443.82,"Thirty Two,Lumilautakengät",328105000449,Väri,Neutraali,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,10.5,0,1,0,19,"7315,7599,7321","7318,7319,7320",Jones MTB 21,Black/Green/Gum, 328105000449/Black/Green/Gum,variation,328105000449/990_11,"Jones MTB 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,569,443.82,"Thirty Two,Lumilautakengät",328105000449,Väri,Neutraali,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,11,0,1,0,19,"7315,7599,7321","7318,7319,7320",Jones MTB 21,Black/Green/Gum, 328105000449/Black/Green/Gum,variation,328105000449/990_8.5,"Jones MTB 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,569,443.82,"Thirty Two,Lumilautakengät",328105000449,Väri,Neutraali,0,1,Brändi,Thirty Two,0,1,Thirty Two,Koko,8.5,0,1,0,19,"7315,7599,7321","7318,7319,7320",Jones MTB 21,Black/Green/Gum, 328105000449/Black/Green/Gum,variation,328105000449/990_9,"Jones MTB 21, Black/Green/Gum",1,7320,visible,taxable,1,1.2000,0,0,0.01,0.0000,0.0000,569,443.82,"Thirty Two,Lumilautakengät",328105000449,Väri,Musta,0,1,Brändi,Thirty Two

(variables) ,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34 8,variable,328105000442,"TM-2 Jones 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,449,341.24,"Thirty Two,Lumilautakengät",,Väri,,1,1,Brändi,Thirty Two,1,1,Thirty Two,Koko,,1,1,0,19,"7315,7599,7321","7318,7319,7320" 14,variable,328105000449,"Jones MTB 21, Black/Green/Gum",1,7320,visible,taxable,1,0.0000,0,0,0.01,0.0000,0.0000,569,443.82,"Thirty Two,Lumilautakengät",,Väri,,1,1,Brändi,Thirty Two,1,1,Thirty Two,Koko,,1,1,0,19,"7315,7599,7321","7318,7319,7320"

woocommerce-rest-api woocommerce python
1 Answers

The longest call in your code is probably:

productlist_variable=wcapi.get("products", params={'sku':variable_sku}).json()

It depends only on variable_sku, which does not change across iterations over v_loaded. Therefore you can do that request before the loop, instead of doing it at every iteration.

The same applies to

productlist=wcapi.get('products'+'/'+str(productid)+"/"+"variations", params={'sku':sku}).json()

You can also get slight performance gains by avoiding repeating this look-up: variables_loaded[variable_iterator]. Instead, keep the result aside, and use it everywhere needed:

current_variables = variables_loaded[variable_iterator]
variable_sku = str(current_variables["1"])
brands_array = str(current_variables["34"]).split(",")
# etc...
Victor Paléologue0
Related