How to make bs4 code not look ugly as hell

Viewed 41

I'm making a scraping project with bs4 and I simply cannot see a way to make it less eye-bleeding. Everything looks like this:

try:
    max_tweet_date = max(
        [tweet_date_to_datetime(timeline_item.find('span', {'class': 'tweet-date'}).find('a')['title'])
         for timeline_item in tweets_or_replies_soup.find_all('div', {'class': 'timeline-item'})]
    )
except AttributeError:
    print('No tweets or replies')
    print()
    neo4j_database.set_user_updated_now(current_user)
except IndexError:
    print('Not enough tweets or no tweets')
    print()
    neo4j_database.set_user_updated_now(current_user)
except ValueError:
    print("Cannot recognise dates, skipping")
    print()
    neo4j_database.set_user_updated_now(current_user)

if is_tweet_too_old(max_tweet_date):
    print('User is inactive, deleting')
    print()
    neo4j_database.delete_user(current_user)

There are always tons of exceptions that need to be handled and the way it handles attributes is begging to be contained in a separate function. Are there any helpful patterns or maybe other parsing libraries that can help clean up this mess?

1 Answers

You can transform both find() and find_all() methods to standalone functions with meaningful names.

Related