How to group lines separated by a blank line more pythonically

user_133298 0
Viewed 1337

I have a file that contains a list of duplicate, but uniquely named files.

For example:

<md5sum>  /var/www/one.png
<md5sum>  /var/www/one-1.png

<md5sum>  /var/www/two.png
<md5sum>  /var/www/two-1.png
<md5sum>  /var/www/two-2.png

The goal is to end up with the following:

[
    [
        '/var/www/one.png',
        '/var/www/one-1.png'
    ],
    [
        '/var/www/two.png',
        '/var/www/two-1.png',
        '/var/www/two-2.png'
    ]
]

This is output from a command I ran earlier. Now I need to process this output, and I came up with the following code for starters:

from pprint import pprint
DUPES_FILE = './dupes.txt'

def process_dupes(dupes_file):
    groups = [[]]
    index = 0
    for line in dupes_file:
        if line != '\n':
            path = line.split('  ')[1]
            groups[index].append(path)
        else:
            index += 1
            groups.append([])

    pprint(groups)

with open(DUPES_FILE, 'r') as dupes_file:
    process_dupes(dupes_file)

Is there a more concise way to write this?

python-3.x python
3 Answers

Read the entire file into a variable. Use split("\n\n") to separate it into the duplicate groups, then split that with split("\n") to get each line, and finally split each line with split(" ").

def process_dupes(dupes_file)
    contents = dupes_file.read()
    groups = [[line.split("  ")[1] for line in group.split("\n") if line != ""] for group in contents.split("\n\n")]
user_14918950

Slightly better version. Also handles the case when there are more than one new lines between groups

def get_groups(dupes_file):
    group = []
    for line in dupes_file:
        if line == "\n":
            if group:
                yield group
                group = []
        else:
            md5sum, path = line.split('  ')
            group.append(path.strip())
    if group:
        yield group

Output:

In [61]: with open(DUPES_FILE, 'r') as dupes_file:
    ...:     pprint(list(get_groups(dupes_file)))
    ...:     
    ...:     
[['/var/www/one.png\n', '/var/www/one-1.png\n'],
 ['/var/www/two.png\n', '/var/www/two-1.png\n', '/var/www/two-2.png\n']]

If this is confusing, one improvement on your version is ignore remove the index variable and use -1 as you always want to add to the last list.

def process_dupes(dupes_file):
    groups = [[]]
    for line in dupes_file:
        if line != '\n':
            path = line.split('  ')[1]
            groups[-1].append(path)
        else:
            groups.append([])

    pprint(groups)
user_4637580

The following will process the data in the file iteratively, instead of reading whole thing into memory first:

from itertools import groupby
from pprint import pprint

DUPES_FILE = './dupes.txt'

def process_dupes(dupes_file):
    groups = [
        [line.rstrip().split('  ')[1] for line in lines]
            for blank, lines in groupby(dupes_file, lambda line: line == '\n')
                if not blank
    ]
    pprint(groups)

with open(DUPES_FILE, 'r') as dupes_file:
    process_dupes(dupes_file)

Output:

[['/var/www/one.png', '/var/www/one-1.png'],
 ['/var/www/two.png', '/var/www/two-1.png', '/var/www/two-2.png']]
user_3552300
Related