I have scraped a few thousand of different job postings for various companies.
For a specific company, there exists a variation of different boilerplate texts which describe the company as of whole. Text such as
About company X At X we create the technology to connect the world. Developing and delivering the industry's only end-to-end portfolio of network equipment, software, services and licensing that is available globally....
Then at some point the text which is regarding the actual position starts.
In some cases this boilerplate text is not present at all.
There is not any particular consistent format such that I can create a logic while scraping to only scrape sections not related to boilerplate text.
So the problem definition is
Is there a way, given a set of documents with various length of text, possible to identify which parts of the document has boilerplate text in it?
Valid assumption: If a document has boilerplate text in it, it is always at the beginning of the document.
My idea is to create a term frequency dictionary based on the entire set of documents. Then create a n words moving average timeseries of the word frequency for each document.
#Function that calculates the n word freq moving average
def moving_average(text,word_count,n=10):
timeseries = [word_count[x] for x in text]
numbers_series = pd.Series(timeseries)
windows = numbers_series.rolling(n)
moving_averages = windows.mean()
moving_averages_list = moving_averages.tolist()
without_nans = moving_averages_list[n - 1:]
return without_nans
df['Text Cleaned Tokenized'] = df['Text'].apply(lambda x: clean_and_tokenize(x)) #Generic tokenize
all_words = [a for b in data['Text Clean Tokenized'].tolist() for a in b] #Unroll the list of lists
word_count = dict(Counter(all_words)) #Count the frequency of each unique word in the document space
data['Timeseries'] = data['Text Clean Tokenized'].apply(lambda x: moving_average(x,n=10)) #1 Row per document
Plotting one realization of this timeseries
In the above picture we could imagine that the boilerplate text is up to around index ~30-40
So here I would create some additional function that determines a cutoff index value for what is considered to be the boilerplate text.
Is there any smarter way of doing this?