Storing output in a format that can be sorted in Python

user_12287234 0
Viewed 94

I am working on learning Python and am trying to take a list (comma separated) stored in a file and convert it to a data store that can be sorted with Python with each four strings filling a row. For example if I have the following in the file:

'apples are great'
,'neg': 0.485, 'neu': 0.392, 'pos': 0.123, 'compound': -0.812,
'crayons are waxy'
,'neg': 0.302, 'neu': 0.698, 'pos': 0.0, 'compound': -0.3818,
'a happy girl'

,'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0,
'a phone is alive' 
,'neg': 0.0, 'neu': 0.737, 'pos': 0.263, 'compound': 0.3612,..........

I would like a dataframe that has the following:

Subject           Neg    Neu    Pos    Compound 
apples are great  0.485  0.392  0.123  -0.812 
crayons are waxy  0.302  0.698  0.0    -0.3818 
a happy girl      0.0    1.0    0.0    0.0 
a phone is alive  0.0    0.737  0.263  0.3612

My goal is to sort by the compound column and also to look for frequency of words in the first column. I feel like this should be relatively easy, but have tried reading into a dataframe, but it becomes one row with each value in a column, and then also tried to make it a textblob with sentences and again get incorrect results.

Thank you in advance for the help.

Sample of what I have tried

test = open('Heros_toSort.txt')
test2=test.readlines()
df = pd.DataFrame(test2, columns = ['name'])
df.assign(name=df.name.str.split(','))
pandas python-3.x python
2 Answers

Python has a built-in library csv which can be used to easily read data. This could likely be performed with standard i/o tools as well, but that's the solution I've got.

In your csv, the first entry of each row is a name in quotes, and the second through to the fifth are values, all of which appear in the same order. The values have a number following a name and the pattern :. We can strip the quotes from around the name, and the number from after the colon, then use this to make a pandas dataframe.

import csv
import pandas as pd

data = []

with open('data.csv', 'r') as in_file:
  c_reader = csv.reader(in_file, delimiter=',')
  for row in c_reader:
    this_row = []

    #from each row, get the name first, stripping the leading and trailing ' single quotes.
    this_row.append(row[0].strip().lstrip('\'').rstrip('\''))

    #get the remaining values
    for i in range(1, len(row)):

      #all values appear after a ': ' pattern, in the same order. split on ': '
      #and get the second half of the split - it's the value we're looking for
      this_row.append(row[i].split(": ",1)[1])

    #add this to the array
    data.append(this_row)

#make a dataframe out of the csv file
df = pd.DataFrame(data, columns=['Name', 'neg', 'neu', 'pos', 'compound'])
print(df)

Demo

user_74318600 
import re
import collections
import pandas as pd

# The input is a single text line
stringinput = ("'apples are great','neg': 0.485, 'neu': 0.392, 'pos': 0.123,"
              + " 'compound': -0.812,'crayons are waxy','neg': 0.302, "
              + "'neu': 0.698, 'pos': 0.0, 'compound': -0.3818,'a happy girl',"
              + "'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0,"
              + "'a phone is alive' ,'neg': 0.0, 'neu': 0.737, 'pos': 0.263,"
              + " 'compound': 0.3612")

# clean up text file
remap = {
    ord('\'') : ''
}

cleaned_text = stringinput.translate(remap)
# remove tags of values
f_test = re.sub( "neg:|neu:|pos:|compound:*", '', cleaned_text )
# break text into list
string_to_list = f_test.split(',')
# create list of lists with
# list comprehension.
# Each inner list contains
# 5 elements, such as 
# 'Subject', 'Neg','Neu','Pos','Compound'
list_to_df = [ string_to_list[i : i + 5] 
              for i in range(0, len(string_to_list), 5) ]
# generate pandas dataframe
df = pd.DataFrame(
    list_to_df,
    columns = ['Subject', 'Neg','Neu','Pos','Compound']
    )
# sort dataframe based on Compound
df_sorted = df.sort_values(['Compound'],
                 ascending = False
                 )
# word frequency
freq = df_sorted['Subject'].to_list()

freq_dict = collections.defaultdict(int)

for text in freq:
  for word in text.split(' '):
    freq_dict[word] += 1

for word, freq in freq_dict.items():
  print(word, freq, sep = '\t')
user_44920080
Related