Lua - find/remove duplicate rows in a .csv file to create a pure/unique list

user_6676439 0
Viewed 150

I have a 3000+ row .csv file that I created by pulling multiple .csv sources together - example of the content is below.

NEC;    35;     -1;     10
RC5;    0;      -1;     12
Panasonic;      176;    0;      61
RC5;    21;     -1;     1
RC5;    21;     -1;     1
NEC;    0;      -1;     0
RECS80; 4;      -1;     30
NEC;    1;      249;    3
Denon;  1;      -1;     40
NEC;    132;    60;     33

Please could someone let me know how I can create a new .csv file from the above that only has unique rows ?

If possible please could someone go on to explain how I could do this for both my existing .csv file and also at the an Lua array/table creation phase - many thanks !..

FYI - my new Lua code uses a loop and the following to populate the .csv

local info = table.concat ({protocol, D, S, F}, '; ')
csv lua
1 Answers

The simplest way is to probably read the lines one by one and keep track of those lines you've already seen without parsing them:

local csv = assert(io.open("test.csv", "r"))
local lines = {}
for line in csv:lines() do
  if not lines[line] then
    print(line)
    lines[line] = true
  end
end

This should print all unique lines from text.csv to the standard output (in the same order as they appeared in the file).

You can also use uniq in combination with sort if they are available on your system.

If you need to apply the same logic to lines in a table, then something like this should work (assuming the lines are in table tt):

local lines = {}
for _, line in ipairs(tt) do
  if not lines[line] then
    print(line)
    lines[line] = true
  end
end
user_14429170
Related