I have a Python scraper that's meant to scrape monthly trade data using this handy dandy API.
import json
import numpy as np
import pandas as pd
import requests, time
def Comtrade_Scraper (ps: int,
type: str= 'C',
freq: str= 'M',
px : str= 'S2',
r : str= 'all',
p : int= 156,
rg : int= 2,
cc : str= 'AG2'):
"""
Wrapper for creating URLs to access the Comtrade API
ARGUMENTS
*********
Required
ps = year
"""
base = 'https://comtrade.un.org/api/get?max=100000'
url = f'{base}&type={type}&freq={freq}&px={px}&ps={ps}&r={r}&p={p}&rg={rg}&cc={cc}'
result = requests.get(url).json()
if 'dataset' in result:
df = pd.DataFrame(result['dataset'])
df = df.replace({None: np.nan})
df.columns= [i[:32] for i in df.columns]
df.to_csv(f'i_X_China_{ps}.csv')
time.sleep(6)
for i in range(2010,2022): Comtrade_Scraper(i)
When I set the loop at the very bottom of the code block to be from 2000 to 2022, the function Comtrade_Scraper only scrapes data from 2000 to 2009. Naturally, I want all monthly data to be scraped from 2000 to 2022, but instead, the CSV files that get created are empty and don't even have the data specified in the API. Why might this be, and what could the solution be?