How many count lines duplicates in text files

Fiínek Cahů 0
Viewed 41

please how can I get count of duplicate lines?

Source data: line e.g. user_id;name;surname;3400;44711;30.05.2022 7:00:00;30.05.2022 15:30:00;0;480;0;1;682;10000120;9

Private Sub remove_duplicite(sender As Object, e As EventArgs)
        Dim sFiles() As String
        sFiles = Directory.GetFiles(filesPath1, remove_dupl)

    Dim path As String = String.Join("", sFiles)
    'MessageBox.Show(path)
    Dim lines As New HashSet(Of String)()
    'Read to file
    Using sr As StreamReader = New StreamReader(path)
        Do While sr.Peek() >= 0
            lines.Add(sr.ReadLine())

        Loop
    End Using

    'Write to file
    Using sw As StreamWriter = New StreamWriter(path)
        For Each line As String In lines
            sw.WriteLine(line)
        Next
    End Using
    Close()
End Sub

I try some answers but no success.But I think that will be easy.

Thank you

vb.net
1 Answers
Dim sList As New List(of String)
sList.Add("1")
sList.Add("2")
sList.Add("2")
sList.Add("3")
Dim sListDistinct As List(Of String) = sList.Distinct().ToList()
Dim iCount as Integer = sList.Count - sListDistinct.Count

But depending on the size of your file, this isn't the best performance way. Maybe check in your HashSet with .Contains and count if entry already exists

0x300
Related