making new data frame from combining text pandas

user_9957961 0
Viewed 30

I have a pandas data frame that is structured something like this:

ID TEXT
1  Start of document
1  middle
1  end of document
2  start of document 2
2  middle
2  end of document 2

The raw data I got has repeating IDs which if you concatenate the text for each unique ID you get a resulting document. Some of these IDs repeat hundreds of times resulting in large quantities of text which I would like to boil down to one observation.

I'm not sure how to go about looping through and creating a new document. Also not sure if Pandas is the right data structure to store large quantities of text (these are transcribed call records--some of them 30 minute+ conversations). Would appreciate any pointers.

pandas dataframe hadoop text python
2 Answers

IIUC:

df.groupby('ID').TEXT.apply(' '.join)

ID
1        Start of document middle end of document
2    start of document 2 middle end of document 2
Name: TEXT, dtype: object
user_23366540

If without groupby 

(df.set_index('ID').TEXT+' ').sum(level=0).str[:-1]
Out[1066]: 
ID
1        Start of document middle end of document
2    start of document 2 middle end of document 2
Name: TEXT, dtype: object
user_79645270
Related