preprocessing the text and excluding form footnotes , extra spaces and

Viewed 103

I need to clean my corpus, it includes these problems

  • multiple spaces --> Tables .
  • footnote --> 10h 50m,1
  • unknown ” --> replace " instead of ” e.g

for instance, you see it here:

On 1580 November 12 at 10h 50m,1 they set Mars down at 8° 36’ 50” Gemini2 without mentioning the horizontal variations, by which term I wish the diurnal parallaxes and the refractions to be understood in what follows. Now this observation is distant and isolated. It was reduced to the moment of opposition using the diurnal motion from the Prutenic Tables  . 

I have done it using these functions

def fix4token(x):
    x=re.sub('”', '\"', x)
    if (x[0].isdigit()== False )| (bool(re.search('[a-zA-Z]', x))==True ):
        res=x.rstrip('0123456789')
        output = re.split(r"\b,\b",res, 1)[0]
        return output  
    else:
        return x
def removespaces(x):
    res=x.replace("  ", " ")
    return(res)

it works not bad for this but the result is so

On 1580 November 12 at 10h 50m, they set Mars down at 8° 36’ 50" Gemini without mentioning the horizontal variations,  by which term I wish the diurnal parallaxes and the refractions to be understood in what follows. Now this observation is distant and isolated. It was reduced to the moment of opposition using the diurnal motion from the Prutenic Tables.

but the problem is it damaged other paragraphs. it does not work ver well,

I guess because this break other things

x=re.sub('”', '\"', x)
    if (x[0].isdigit()== False )| (bool(re.search('[a-zA-Z]', x))==True ):
        res=x.rstrip('0123456789')
        output = re.split(r"\b,\b",res, 1)[0]

what is the safest way to do these?

1- remove footnotes like in these phrases

  • "10h 50m,1" or (extra foot note in text after comma)
  • "Gemini2" (zodic names of month + footnote)

without changing another part of the text (e.g my approach will break the "DC2" to "DC" which is not desired

2- remove multiple spaces before dot . like "Tables ." to no spaces or remove multiple before, like: ", by which term" to this 9only one space) ", by which term"

3-replace unknown ” -> replace " ...which is done

thank you

1 Answers

You can use

text = re.sub(r'\b(?:(?<=,)\d+|(Capricorn|Aquarius|Pisces|Aries|Taurus|Gemini|Cancer|Leo|Virgo|Libra|Scorpio|Ophiuchus|Sagittarius)\d+)\b|\s+(?=[.,])', r'\1', text, flags=re.I).replace('”', '"')
text = re.sub(r'\s{2,}', ' ', text)

Details:

  • \b - a word boundary
  • (?: - start of a non-capturing group:
    • (?<=,)\d+ - one or more digits that are preceded with a comma
    • | - or
    • (Capricorn|Aquarius|Pisces|Aries|Taurus|Gemini|Cancer|Leo|Virgo|Libra|Scorpio|Ophiuchus|Sagittarius)\d+ - one of the zodiac sign words (captured into Group 1, \1 in the replacement pattern refers to this value) and then one or more digits
  • ) - the end of the non-capturing group
  • \b
  • | - or
  • \s+ - one or more whitespaces
  • (?=[.,]) - that are immediately followed with . or ,.

The .replace('”', '"') replaces all ” with a " char.

The re.sub(r'\s{2,}', ' ', text) code replaces all chunks of two or more whitespaces with a single regular space.

Related