Scraping data parallel + batch processing

Question

I am doing a task that requires scraping. I have a dataset with ids and for each id i need to scrape some new information. This dataset has around 4 million rows. Here is my code:

import pandas as pd
import numpy as np
import semanticscholar as sch
import time

# dataset with ids
df = pd.read_csv('paperIds-1975-2005-2015-2.tsv', sep='	', names=["id"])

# columns that will be produced
cols = ['id', 'abstract', 'arxivId', 'authors', 
        'citationVelocity', 'citations', 
        'corpusId', 'doi', 'fieldsOfStudy', 
        'influentialCitationCount', 'is_open_access', 
        'is_publisher_licensed', 'paperId', 
        'references', 'title', 'topics', 
        'url', 'venue', 'year']

# a new dataframe that we will append the scraped results
new_df = pd.DataFrame(columns=cols)

# a counter so we know when every 100000 papers are scraped
c = 0      
i = 0
while i < df.shape[0]:
    try:
        paper = sch.paper(df.id[i], timeout=10) # scrape the paper
        new_df = new_df.append([df.id[i]]+paper, ignore_index=True) # append to the new dataframe
        new_df.to_csv('abstracts_impact.csv', index=False) # save it 
        if i % 100000 == 0: # to check how much we did
            print(c)
            c += 1
        i += 1
    except:
        time.sleep(60)

The problem is that the dataset is pretty big and this approach is not working. I left it working for 2 days and it scraped around 100000 ids, and then suddenly just froze and all the data that was saved was just empty rows. I was thinking that the best solution would be to parallelize and batch processing. I never have done this before and I am not familiar with these concepts. Any help would be appreciated. Thank you!

user_6623589 · Answer

Okay, so first of all there is no data :( so I am just taking a sample ID from semanticscholar documents. Looking at your code, there I can see plenty of mistakes:

Don't always stick to pd.DataFrame for your work! Dataframe are great, but are also slow! You just need to get the ID from the 'paperIds-1975-2005-2015-2.tsv' so you can either read the file using file.readline() or you can save the data into a list:

data = pd.read_csv('paperIds-1975-2005-2015-2.tsv', sep='	', names=["id"]).id.values

From the code flow, what I understand is you want to save the scraped data into a single CSV file, right? So, why are you appending the data and saving the file again and again? This makes the code like 100000s time slower!
I really don't understand the purpose of time.sleep(60) you have added. If there is some error, you should print and move on - why wait?
For checking the progress, you can use the tqdm library which shows a nice progress bar for your code!

Taking these into consideration, I have modified your code as follows:

import pandas as pd
import semanticscholar as sch
from tqdm import tqdm as TQ # for progree-bar

data = ['10.1093/mind/lix.236.433', '10.1093/mind/lix.236.433'] # using list or np.ndarray looks more logical!
print(data)
>> ['10.1093/mind/lix.236.433', '10.1093/mind/lix.236.433']

Once you have done this, you can now go and scrape the data. Okay, before that pandas DataFrame is basically a dictionary with advanced features. So, for our purpose, we will first add all the information to the dictionary and then create the dataframe. I personally prefer this process - gives me more control, if there are any changes need to be done.

cols = ['id', 'abstract', 'arxivId', 'authors', 'citationVelocity', 'citations',
    'corpusId', 'doi', 'fieldsOfStudy', 'influentialCitationCount', 'is_open_access',
    'is_publisher_licensed', 'paperId', 'references', 'title', 'topics', 'url', 'venue', 'year']

outputData = dict((k, []) for k in cols)

print(outputData)
{'id': [],
 'abstract': [],
 'arxivId': [],
 'authors': [],
 'citationVelocity': [],
 'citations': [],
 'corpusId': [],
 'doi': [],
 'fieldsOfStudy': [],
 'influentialCitationCount': [],
 'is_open_access': [],
 'is_publisher_licensed': [],
 'paperId': [],
 'references': [],
 'title': [],
 'topics': [],
 'url': [],
 'venue': [],
 'year': []}

Now you can simply fetch the data and save it into your dataframe as below:

for _paperID in TQ(data):
    paper = sch.paper(_paperID, timeout = 10) # scrape the paper
    for key in cols:
        try:
            outputData[key].append(paper.get(key))
        except KeyError:
            outputData[key].append(None) # if there is no data, append none
            print(f"{key} not Found for {_paperID}")

pd.DataFrame(outputData).to_csv('output_file_name.csv', index = False)

This is the output that I have obtained:

Scraping data parallel + batch processing

1 Answers