How do I count the occurencies of a word in csv file under conditions?

Viewed 94

I have a data.csv file as such (where data are words)

data
data1|data2|data3
data1|data2
data1|data4|data6
data2|data3
data4|data5|data6
data4
data5|data7

I want to read how many times data1 (for example) is present when data2 is present. So when and how many times in the rows two data coexist, as I want to form relationships between them. I am aware of the counter method but I am not interested in just counting instances of the data.

How do I solve this? Do I create a data frame with pd or do I use a list?

(edit:max number of columns in the data.csv is 6)

The desired output would be something like:

data1-data2: 2
data1-data3: 1
data1-data4: 1
data1-data5: 0 
etc

saved into a CSV file

3 Answers

Let's try with get_dummies and combinations:

from itertools import combinations

df = pd.read_csv('data.csv')
d = df['data'].str.get_dummies('|')

dct = {}
for x, y in combinations(g, r=2):
    dct[f'{x}:{y}'] = d[[x, y]].eq(1).all(1).sum()

# or with dict comprehension
# {'{}:{}'.format(*v): d[[*v]].eq(1).all(1).sum() for v in combinations(g, r=2)}

Details:

First read the csv file as pandas dataframe:

print(df)
                data
0  data1|data2|data3
1        data1|data2
2  data1|data4|data6
3        data2|data3
4  data4|data5|data6
5              data4
6        data5|data7

Encode the values in data as dummy/indicator columns:

print(d)
   data1  data2  data3  data4  data5  data6  data7
0      1      1      1      0      0      0      0
1      1      1      0      0      0      0      0
2      1      0      0      1      0      1      0
3      0      1      1      0      0      0      0
4      0      0      0      1      1      1      0
5      0      0      0      1      0      0      0
6      0      0      0      0      1      0      1

generate combinations of indicator columns and use dict to store their frequency of occurrence together:


print(dct)
{'data1:data2': 2,
 'data1:data3': 1,
 'data1:data4': 1,
 'data1:data5': 0,
 'data1:data6': 1,
 'data1:data7': 0,
 'data2:data3': 2,
  ...
  ...
 'data5:data7': 1,
 'data6:data7': 0}

Try the following:

import itertools

with open('yourfile.csv') as f:
    l=f.readlines()

l=[i[:-1] for i in l[:-1]] + [l[-1]] #remove '\n' at the end

res={}

for i in l:
    temp=i.split('|')
    pairs=[k for k in itertools.combinations(temp, 2)]
    for j in pairs:
        if j in res:
            res[j]+=1
        elif j[::-1] in res:
            res[j[::-1]]+=1
        else:
            res[j]=1

Output:

>>>print(res)

{('data1', 'data2'): 2, ('data1', 'data3'): 1, ('data2', 'data3'): 2, ('data1', 'data4'): 1, ('data1', 'data6'): 1, ('data4', 'data6'): 2, ('data4', 'data5'): 1, ('data5', 'data6'): 1, ('data5', 'data7'): 1}

If you want the pairs that have 0 occurences, you can do the following:

s={i for j in res.keys() for i in j}

allpairs=[i for i in itertools.combinations(s,2)]

for i in allpairs:
    if i not in res and (i[1], i[0]) not in res:
        res[i]=0

The question is very ambiguous, hence so many different answers. I assume here you mean to count the combinations of the first column and any of the following columns.

import csv
from collections import Counter

counter = Counter()
with open('data.csv') as csvfile:
  reader = csv.reader(csvfile, delimiter='|')
  for line in reader:
    if line[0]:
      for fragment in line[1:]:
        entry = line[0] + '-' + fragment
        counter.update({entry: 1})

print(counter)

Output:

Counter({'data1-data2': 2, 'data1-data3': 1, 'data1-data4': 1, 'data1-data6': 1, 'data2-data3': 1, 'data4-data5': 1, 'data4-data6': 1, 'data5-data7': 1})

EDIT 1:

Assuming you want any non-zero combinations of existing data fields:

import csv
from collections import Counter
from itertools import combinations

counter = Counter()
with open('data.csv') as csvfile:
  reader = csv.reader(csvfile, delimiter='|')
  for line in reader:
    counter.update(combinations(line, 2))

print(counter)

Output:

Counter({('data1', 'data2'): 2, ('data2', 'data3'): 2, ('data4', 'data6'): 2, ('data1', 'data3'): 1, ('data1', 'data4'): 1, ('data1', 'data6'): 1, ('data4', 'data5'): 1, ('data5', 'data6'): 1, ('data5', 'data7'): 1})

EDIT 2: Assuming you want each data cell combined with each other data cell including those relationships that do not show up anywhere on the same line:

import csv
from collections import Counter
from itertools import combinations

counter = Counter()
unique = set()
with open('data.csv') as csvfile:
  reader = csv.reader(csvfile, delimiter='|')
  for line in reader:
    unique.update(line)
    counter.update(combinations(line, 2))
counter.update({entry: 0 for entry in combinations(unique, 2)})
print(counter)

Output:

Counter({('data1', 'data2'): 2, ('data2', 'data3'): 2, ('data4', 'data6'): 2, ('data1', 'data3'): 1, ('data1', 'data4'): 1, ('data1', 'data6'): 1, ('data4', 'data5'): 1, ('data5', 'data6'): 1, ('data5', 'data7'): 1, ('data7', 'data5'): 0, ('data7', 'data2'): 0, ('data7', 'data4'): 0, ('data7', 'data6'): 0, ('data7', 'data1'): 0, ('data7', 'data'): 0, ('data7', 'data3'): 0, ('data5', 'data2'): 0, ('data5', 'data4'): 0, ('data5', 'data1'): 0, ('data5', 'data'): 0, ('data5', 'data3'): 0, ('data2', 'data4'): 0, ('data2', 'data6'): 0, ('data2', 'data1'): 0, ('data2', 'data'): 0, ('data4', 'data1'): 0, ('data4', 'data'): 0, ('data4', 'data3'): 0, ('data6', 'data1'): 0, ('data6', 'data'): 0, ('data6', 'data3'): 0, ('data1', 'data'): 0, ('data', 'data3'): 0})
Related