Better regex implementation than for looping whole file?

user_8565438 0
Viewed 83

I have files looking like this:

#     BJD     K2SC-Flux EAPFlux   Err  Flag Spline
2457217.463564 5848.004 5846.670 6.764 0 0.998291
2457217.483996 6195.018 6193.685 6.781 1 0.998291
2457217.504428 6396.612 6395.278 6.790 0 0.998292
2457217.524861 6220.890 6219.556 6.782 0 0.998292
2457217.545293 5891.856 5890.523 6.766 1 0.998292
2457217.565725 5581.000 5579.667 6.749 1 0.998292
2457217.586158 5230.566 5229.232 6.733 1 0.998292
2457217.606590 4901.128 4899.795 6.718 0 0.998293
2457217.627023 4604.127 4602.793 6.700 0 0.998293

I need to find and count the lines with Flag = 1. (5th column.) This is how I have done it:

foundlines=[]
c=0
import re
with open('examplefile') as f:
    for index, line in enumerate(f):
        try:
            found = re.findall(r' 1 ', line)[0]
            foundlines.append(index)
            print(line)
            c+=1
        except:
            pass
print(c)

In Shell, I would just do grep " 1 " examplefile | wc -l, which is much shorter than the Python script above. The python script works, but I am interested in whether is there a shorter, more compact way to do the task than the script above? I prefer the shortness of Shell so I would like to have something similar in Python.

python-3.x shell text python regex
3 Answers

You have CSV data, you can use the csv module:

import csv

with open('your file', 'r', newline='', encoding='utf8') as fp:
    rows = csv.reader(fp, delimiter=' ')

    # generator comprehension
    errors = (row for row in rows if row[4] == '1')

for error in errors:
    print(error)
Tomalak_187710

You shell implementation can be made even shorter, grep has -c option to get you a count, no need for an anonymous pipe and wc:

grep -c " 1 " examplefile

You shell code simply gets you the line counts where the pattern 1 is found, but your Python code additionally keeps a list of indexes of lines where the pattern is matched.

Only to get the line counts, you can use sum and genexp/list comprehension, also no need for Regex; simple string __contains__ check would do as strings are iterable:

with open('examplefile') as f:
    count = sum(1 for line in f if ' 1 ' in line)
    print(count)

If you want to keep indexes as well, you can stick to your idea with only replacing re test with str test:

count = 0
indexes = []
with open('examplefile') as f:
    for idx, line in enumerate(f):
        if ' 1 ' in line:
            count += 1
            indexes.append(idx)

Additionally, doing a bare except is almost always a bad idea (at least you should use except Exception to leave out SystemExit, KeyboardInterrupt like exceptions), catch only the exceptions you know might be raised.

Also, while parsing structured data, you should use specific tool e.g. here csv.reader with space as the separator (line.split(' ') should do in this case as well) and checking against index-4 would be safest (see Tomalak's answer). With the ' 1 ' in line test, there would be misleading results if any other column contains 1.

Considering the above, here's the shell way using awk to match against the 5-th field:

awk '$5 == "1" {count+=1}; END{print count}' examplefile
user_37895500

Shortest code

This is a very short version under some specific preconditions:

  • You just want to count occurrences like your grep invocation
  • There is guaranteed to be only one " 1 " per line
  • That " 1 " can only occur in the desired column
  • Your file fits easily into memory

Note that if these preconditions are not met, this may cause issues with memory or return false positives.

print(open("examplefile").read().count(" 1 "))

Easy and versatile, slightly longer

Of course, if you're interested in actually doing something with these lines later on, I recommend Pandas:

df = pandas.read_table('test.txt', delimiter=" ",
                       comment="#",
                       names=['BJD', 'K2SC-Flux', 'EAPFlux', 'Err', 'Flag', 'Spline'])

To get all the rows where Flag is 1:

flaggedrows = df[df.Flag == 1]

returns:

            BJD  K2SC-Flux   EAPFlux    Err  Flag    Spline
1  2.457217e+06   6195.018  6193.685  6.781     1  0.998291
4  2.457218e+06   5891.856  5890.523  6.766     1  0.998292
5  2.457218e+06   5581.000  5579.667  6.749     1  0.998292
6  2.457218e+06   5230.566  5229.232  6.733     1  0.998292

To count them:

print(len(flaggedrows))

returns 4

user_16480330
Related