Nested For Loops With Calculations Vs. Linear Process

user_9414906 0
Viewed 63

I'm iterating over M dataframes, each containing a column with N URLs. For each URL, I extract paragraph text, then conduct standard cleaning for textual analysis before calculating "sentiment" scores.

Is it more efficient for me to:

  1. Continue as it is (compute scores in the URL for-loop itself)

  2. Extract all of the text from URLs first, and then separately iterate over the list / column of text ?

Or does it not make any difference?

Currently running calculations within the loop itself. Each DF has about 15,000 - 20,000 URLs so it's taking an insane amount of time too!

# DFs are stored on a website
# I extract links to each .csv file and store it as a list in "df_links"

for link in df_links:
    cleaned_articles = []
    df = pd.read_csv(link, sep="\t", header=None)
    # Conduct df cleaning
    # URLs for articles to scrape are stored in 1 column, which I iterate over as...
    for url in df['article_url']:
        response = requests.get(url)
        soup = BeautifulSoup(response.text, 'html.parser')
        para_text = [text.get_text() for text in soup.findAll('p')]
        text = " ".join(para_text)
        words = text.split()

        if len(words) > 500:
            # Conduct Text Cleaning & Scores Computations
            # Cleaned text stored as a variable "clean_text"
            cleaned_articles.append(clean_text)

    df['article_text'] = cleaned_articles
    df.to_csv('file_name.csv')
for-loop python performance
2 Answers

To answer the question, it shouldn't make too much of a difference if you download the data and then apply analysis to it. You'd just be re arranging the order in which you do a set of tasks that would effectively take the same time.

The only difference may be if the text corpus' are rather large and then read write time to disk will start to play a part so could be a little faster running the analytics all in memory. But this still isn't going to really solve your problem.

May I be so bold as to reinterpret your question as: "My analysis is taking too long help me speed it up!"

This sounds like a perfect use case for multiprocessing! Since this sounds like a data science project you'll need to pip install multiprocess if you're using a ipython notebook (like Jupyter) or import multiprocessing if using a python script. This is because of the way python passes information between processes, don't worry though the API's for both multiprocess and multiprocessing are identical!

A basic and easy way to speed up your analysis is to indent you for loop and put it in a function. That function can then be passed to a multiprocessing map which can spawn multiple processes and do the analysis on several urls all at once:

from multiprocess import Pool
import numpy as np
import os
import pandas as pd
num_cpus = os.cpu_count()

def analytics_function(*args):
#Your full function including fetching data goes here and accepts a array of links
return something

df_links_split = np.array_split(df_links, num_cpus * 2) #I normally just use 2 as a rule of thumb
pool = Pool(num_cpus * 2) #Start a pool with num_cpus * 2 processes
list_of_returned = pool.map(analytics_function, df_links_split)

This will spin up a load of processes and utilise your full cpu. You'll not be able to do much else on your computer, and you'll need to have your resource monitor open to check you're not maxing our your memory and slowing down/crashing the process. But it should significantly speed up your analysis by roughly a factor of num_cpus * 2!!

user_93278480

Extracting all of the texts then processing all of it or extracting one text then processing it before extracting the next wont do any difference. Doing ABABAB takes as much time as doing AAABBB. You might however be interested in using threads or asynchronous requests to fetch all of the data in parallel.

user_40471850
Related