Can't get rid of unwanted stuff while scraping email addresses

Viewed 499

I'm trying to capture email addresses from some site's landing pages using requests in combination with re module. This is the pattern [\w\.-]+@[\w\.-]+ that I've used within the script to capture them.

When I run the script, I do get email addresses. However, I also get some unwanted stuff that resemble email addresses but in reality they are not and for that reason I would like to get rid of them.

import re
import requests

links = (
    'http://www.acupuncturetx.com',
    'http://www.hcmed.org',
    'http://www.drmindyboxer.com',
    'http://wendyrobinweir.com',
)

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"}

for link in links:
    r = requests.get(link,headers=headers)
    emails = re.findall(r"[\w\.-]+@[\w\.-]+",r.text)
    print(emails)

Current output:

['react@16.5.2', 'react-dom@16.5.2', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com']
['hh-logo@2x.png', 'hh-logo@2x.png', 'hh-logo@2x.png', 'hh-logo@2x-300x47.png']
['leaflet@1.7.1']
['8b4e078a51d04e0e9efdf470027f0ec1@sentry.wixpress.com', 'requirejs-bolt@2.3.6', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wixstores-client-cart-icon@1.797.0', 'wixstores-client-gallery@1.1634.0']

Expected output:

['bai@acupuncturetx.com', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com']
[]
[]
['wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com']

How can I only capture email addresses and get rid of unwanted stuff using regex?

9 Answers

Parting from where you left, you can use a simply checker to verify if it's really a valid email.

So first we define the check function:

def check(email):
    regex = '^[a-z0-9]+[\._]?[a-z0-9]+[@]\w+[.]\w+$'
    if re.match(regex, email):
        return True
    else:
        return False

Then we use it to check your itens on your email list:

for link in links:
    r = requests.get(link, headers=headers)
    emails_list = re.findall(r"[\w\.-]+@[\w\.-]+", r.text)
    emails_list = [email for email in emails_list if check(email)]
    print(emails_list)

Outputs:

['bai@acupuncturetx.com', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com']
[]
[]
['wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com']

I happen to have a regex such as this that respects RFC 5321, which will help you weed out a lot of bogus (ie: non-local) email addresses, but not all. The code can easily be adapted to ignore more stuff should you want to...

For example, email 8b4e078a51d04e0e9efdf470027f0ec1@... does look bogus, but the "local name" part is technically correct as per the RFC... You could add checks on the local-name part (will be match.group(1) in my code snippet below)

Here's my code tidbit with the RFC-compliant regex in question:

# See https://www.rfc-editor.org/rfc/rfc5321
EMAIL_REGEX = re.compile(r"([\w.~%+-]{1,64})@([\w-]{1,64}\.){1,16}(\w{2,16})", re.IGNORECASE | re.UNICODE)


# Cache this (it doesn't change often), all official top-level domains
TLD_URL = "https://datahub.io/core/top-level-domain-names/r/top-level-domain-names.csv.json"
OFFICIAL_TLD = requests.get(TLD_URL).json()
OFFICIAL_TLD = [x["Domain"].lstrip(".") for x in OFFICIAL_TLD]


def extracted_emails(text):
    for match in EMAIL_REGEX.finditer(text):
        top_level = match.group(3)
        if top_level in OFFICIAL_TLD:
            email = match.group(0)
            # Additionally, max length of domain should be at most 255
            # You could also simplify this to simply: len(email) < 255
            if len(top_level) + len(match.group(2)) < 255:
                yield email


# ... 8< ... stripped unchanged code for brevity

for link in links:
    r = requests.get(link,headers=headers)
    emails = list(extracted_emails(r.text))
    print(emails)

This yields your expected results + the one bogus (but technically correct) 8b4e078a51d04e0e9efdf470027f0ec1@... email.

It uses a regex that strictly complies to RFC 5321, and double-checks the top-level domain against the official list for each substring that looks like a valid email.

The easiest fix for this is to realize that the top level domain is always alphabetic and that the subdomain components are always like "foo.".

EDIT More comprehensive look arounds.

/(?!<[\w.-])[\w.-]+@(?:[A-Za-z0-9-]+\.)+[A-Za-z]{2,}(?![A-Za-z0-9.-])/

You can always exclude certain string components with a negative look ahead. \.(?!gif|png|jpg)\w+

To exclude long hex sequences do this.

/(?!(?:[A-Fa-f0-9]{2})+@)/

# all together now
/(?!<[\w.-])(?!(?:[A-Fa-f0-9]{2})+@)[\w.-]+@(?:[A-Za-z0-9-]+\.)+[A-Za-z]{2,}(?![A-Za-z0-9.-])/

Incidentally, here is Perl's regex for an email address.

qr/(?^:(?:(?^:(?>(?^:(?^:(?>(?^:(?>(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*))|\.|\s*"(?^:(?^:[^\\"])|(?^:\\(?^:[^\x0A\x0D])))+"\s*))+))|(?>(?^:(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*))|(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*"(?^:(?^:[^\\"])|(?^:\\(?^:[^\x0A\x0D])))*"(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*)))+))?)(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*<(?^:(?^:(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*(?^:(?>[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+(?:\.[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+)*))(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*))|(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*"(?^:(?^:[^\\"])|(?^:\\(?^:[^\x0A\x0D])))*"(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*)))\@(?^:(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*(?^:(?>[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+(?:\.[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+)*))(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*))|(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*\[(?:\s*(?^:(?^:[^\[\]\\])|(?^:\\(?^:[^\x0A\x0D]))))*\s*\](?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*))))>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*)))|(?^:(?^:(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*(?^:(?>[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+(?:\.[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+)*))(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*))|(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*"(?^:(?^:[^\\"])|(?^:\\(?^:[^\x0A\x0D])))*"(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*)))\@(?^:(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*(?^:(?>[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+(?:\.[^\x00-\x1F\x7F()<>\[\]:;@\\,."\s]+)*))(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*))|(?^:(?>(?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*\[(?:\s*(?^:(?^:[^\[\]\\])|(?^:\\(?^:[^\x0A\x0D]))))*\s*\](?^:(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))|(?>\s+))*)))))(?>(?^:(?>\s*\((?:\s*(?^:(?^:(?>[^()\\]+))|(?^:\\(?^:[^\x0A\x0D]))|))*\s*\)\s*))*))/

HTH

Instead of only capturing e-mail addresses, you can test everything you capture with the package validate_email (pip install validate_email) and retain only valid ones. The code would be some version of the code below:

from validate_email import validate_email
emails = [x if validate_email(x) else '' for x in list_of_potential_emails]

This package checks with the corresponding server if the e-mail (or the server) exists.

Please find the code as below:

import re
import requests

links = (
    'http://www.acupuncturetx.com',
    'http://www.hcmed.org',
    'http://www.drmindyboxer.com',
    'http://wendyrobinweir.com',
)

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"}


def check(email):
    regex = '^[a-z0-9]+[\._]?[a-z0-9]+[@]\w+[.]\w{2,3}$'
    if (re.search(regex, email)):
        return email

Email_List = []

for link in links:
    r = requests.get(link,headers=headers)
    emails = re.findall(r"[\w\.-]+@[\w\.-]+",r.text)
    for email in emails:
        valid_email = check(email)
        if valid_email!= None:
            Email_List.append(valid_email)

    print(Email_List)
    Email_List.clear()

This regex should catch most of them:

print( re.findall(r"\b[\w.%+-]+@[\w.-]+\.[A-Z]{2,8}(?<!gif|jpg|png)\b", '''
['react@16.5.2', 'react-dom@16.5.2', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com', 'bai@acupuncturetx.com']
['hh-logo@2x.png', 'hh-logo@2x.png', 'hh-logo@2x.png', 'hh-logo@2x-300x47.png']
['leaflet@1.7.1']
['8b4e078a51d04e0e9efdf470027f0ec1@sentry.wixpress.com', 'requirejs-bolt@2.3.6', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wendyrobin16@gmail.com', 'wixstores-client-cart-icon@1.797.0', 'wixstores-client-gallery@1.1634.0']
''', re.IGNORECASE) )

There's an extra negative lookbehind at the end for special cases

I don't know if it's true for all the links in links, but at least for those in your question (and from what I can tell, in many cases), the common denominator seems to be that the spurious, email-like strings are all inside a <script> tag, while actual email addresses (if any) are found both inside and outside that tag. So the following will skip anything inside that tag (and, incidentally, remove a lot of duplication). If you are lucky and there are email addresses outside that, this should capture them:

from bs4 import BeautifulSoup as bs
for link in links:
    r = requests.get(link, headers=headers)
    for em in soup.find_all(text=re.compile(r"[\w\.-]+@[\w\.-]+")):
      if em.parent.name != "script":
        print(em)

With your 4 links, the output would be

bai@acupuncturetx.com

for the first, nothing for the 2nd and 3rd and

wendyrobin16@gmail.com
wendyrobin16@gmail.com

For the last one. Try it on your actual links and see if it gets you close enough.

  1. In your case, you can just make that the domain names end by a valid TLDs and sub TLDs. Here is a pretty comprehensive list of them: https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains

  2. Alternatively, if you are too lazy to do this, and if you don't mind extra delay in parsing email addresses, you may simply perform a DNS lookup for every found email address candidate to see if you can get an MX entry. Then treat it as a domain name only when it does have an MX entry.

There are two issues here:

  1. How to discard matches we don't want
  2. How to make it maintainable

1. How to discard matches we don't want

We can decide if we the match is good or not based on what's in the right hand side of the @ character without having to install/use more python modules.

The idea is to put a negative lookahead in your regex after the '@' character to match the stuff you don't want. If the content of the negative lookahead matches, you won't get that result.

Your regex will go from this:

[\w\.-]+@[\w\.-]+
         ^^^^^^^^---> We can decide based on what's here 

To this:

[\w\.-]+@(?! **STUFF_I_DO_NOT_WANT_TO_MATCH_AFTER_THIS_POINT** )[\w\.-]+
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           \-> Look ahead for the regex inside (?! ) and 
               continue if it does not match.

In order to discard the string, we need to look for:

  • strings with digits and dots: [\d\.]+
  • known file extensions at the end of the string: (png|other_extension)$

If we mach any of the conditions above, we can discard the string. We can consctruct the negative loookahead with an OR condition like this: (?![\d\.]+|(png|other_extension)$]).

Adding the neg. lookahead after the @ we get this regex:

[\w\.-]+@(?![\d\.]+|(png|other_extension)$)[\w\.-]+

2. How to make it easy to maintain?

The problem with regexes is they become unmantainable quickly.

You can read them without trouble while you're working in the problem; but after some time you might need to change the regex without frying your brain.

To mitigate this a biut you can define the regex in python this way:

email_regex = (r'[\w\.-]+@'    # Left hand side of the email - characters and dots.
               r'?![\d\.]+|(png|other_file_extension)$'  # Stuff we don't want at the end of the @ (not valid domains like digits and file extensions
               r'[\w\.-]'      # Email domain
              )

# Look for emails
emails = re.findall(email_regex,r.text)

This works because in python you can concatenate stings just separating them with withespace (just try "123asd" == "123" "asd")

Related