Detect duplicate elements python

user_9743555 0
Viewed 100

I get continuously data from a server and can receive the data via the following line of code:

id, type, value = getId(payload)

After that I would like to write them into a file via: 

out.write(str(id) + ";" + str(type) + ";" + str(value) + "\n")

Now the case is, that the same id can appear multiple times, but the value will be a different one. Therefore I would like to extend the out.write into the following way that the different values are added at the right side but still being referred to the same id: 

out.write(str(id) + ";" + str(type) + ";" + str(value) + ";" + str(value1) + ";" + str(value2) + "\n")

Does anyone has an idea how to do this in python?

python-3.x list arrays python
2 Answers

You want to keep a dictionary where the key in the dictionary is your id and the value is a set and you can add all the values

from collections import defaultdict
results = defaultdict(set)

id, type, value = getId(payload)
results[id].add(value)

This dictionary will then keep all the values per id.

It should be noted that using a set will remove the count of duplicates, so if you want to keep every value, including duplicates use a list instead

user_22364010

Using the hints that were already added as comments you can create something similar to this:

from collections import defaultdict

values = defaultdict(set)
types = dict()

for payload in input_stream:
    id, type, value = get(payload)
    values[id].add(value)
    types[id] = type

for id in types.keys():
    out.write(";".join(map(str, [id, types[id]] + list(values[id]))) + "\n")

If the values is more of a time series (order is important), then replace set with list.

user_1067460
Related