Python: While reading the file and counting the words in a line, I want to count words coming between " " or ' ' as a single word

If you only have this simple format (no nested quotes or escaped quotes), you could use a simple regex:

lines = '''TopLevel  
    DISPLAY "In TopLevel. Starting to run program"  
    PERFORM OneLevelDown  
    DISPLAY "Back in TopLevel."  
    STOP RUN.'''.split('\n')

import re
counts = [len(re.findall(r'\'.*?\'|".*?"|\S+', l))
          for l in lines]
# [1, 2, 2, 2, 2]

If not, you have to write a parser

If you are looking for a not regex solution, this is my method for you:

# A simple function that will simply count words in each line
def count_words(line):
    # Check the next function
    line = manage_quotes(line)
    words = line.strip()
    # In case of several spaces in a row, We need to filter empty words
    words = [word for word in words if len(word) > 0]
    return len(words)

# This method will manage the quotes
def manage_quotes(line):
    # We do not mind the escaped quotes, They are like a simple char
    # Also since the changes will be local we can replace words in line
    line = line.replace("\\\"", "Q").replace("\\\'", "q")

    # As all words between 2 quotes act as one word we can replace them with 1 simple word and we start with `"`
    # This loop will help to find all quotes in one line
    while True:
        i1 = line.find("\"")
        if (i1 == -1): # No `"` anymore
            break
        i2 = line[i1+1:].find("\"") # Search after the previous one
        if (i2 == -1): # What shall we do with not paired quotes???
            # raise Exception()
            break
        line = line[:i1-1] + "QUOTE" + line[i2:]
    # Now search for `'`
    while True:
        i1 = line.find("\'")
        if (i1 == -1): # No `'` anymore
            break
        i2 = line[i1+1:].find("\'") # Search after the previous one
        if (i2 == -1): # What shall we do with not paired quotes???
            # raise Exception()
            break
        line = line[:i1-1] + "quote" + line[i2:]
    
    return line

This is how this method works, For example, You have a line like this DISPLAY "Part One \'Test1\'" AND 'Part Two \"Test2\"'

At first, we remove escaped quotes: DISPLAY "Part One qTest1q" AND 'Part Two QTest2Q'

Then we replace double quotations: DISPLAY QUOTE AND 'Part Two QTest2Q'

Then the other one: DISPLAY QUOTE AND quote

And now we count this which is 4

You can solve this without regex if you keep some marker if you are inside a quoted area or not.

• str.split() - splitts at spaces, returns a list
• str.startswith()
• str.endswith() - takes a (tuple of) string(s) and returns True if it starts/ends with (any of) it

Code:

# create input file
name = "file.txt"
with open(name, "w") as f:
    f.write("""TopLevel  
    DISPLAY "In TopLevel. Starting to run program"  
    PERFORM OneLevelDown  
    DISPLAY "Back in TopLevel."  
    STOP RUN.""")

# for testing later
expected = [(1,1),(2,2),(3,2),(4,2),(5,2)]  # 1 base line/word count

# program that counts words
counted = []
with open(name) as f:
    for line_nr, content in enumerate(f,1): # 1 based line count
        splt = content.split()
        in_quotation = []
        line_count = 0
        for word in splt:
            if not in_quotation:
                line_count += 1  # only increments if list empty
            if word.startswith(("'",'"')):
                in_quotation.append(word[0])
            
            if word.endswith(("'","'")):
                in_quotation.pop()
        counted.append((line_nr, line_count))
            
print(expected)
print(counted)
print("Identical: ", all(a == expected[i] for i,a in enumerate(counted)))

Output:

[(1, 1), (2, 2), (3, 2), (4, 2), (5, 2)]
[(1, 1), (2, 2), (3, 2), (4, 2), (5, 2)]
Identical: True

You can tinker with the code - currently it does not well behave if you space out your " - it does not know if something ends or starts and both tests are True.

It seems that the code attached above doesn't care about ' or ". And here is the definition of str.split in Python here.

If sep is not specified or is None, a different splitting algorithm is applied: runs of consecutive whitespace are regarded as a single separator, and the result will contain no empty strings at the start or end if the string has leading or trailing whitespace. Consequently, splitting an empty string or a string consisting of just whitespace with a None separator returns [].

Code:

input_str = '''TopLevel  
    DISPLAY "In TopLevel. Starting to run program"  
    PERFORM OneLevelDown  
    DISPLAY "Back in TopLevel."  
    STOP RUN.'''

input_str_list = input_str.split('\n')
print(input_str_list)


def get_trick_word(s: str):
    num = 0
    quotation_list = []
    previous_char = ' '
    for c in s:
        need_set_previous = False
        if c == ' ' and len(quotation_list) == 0 and previous_char != ' ':
            num = num + 1
        else:
            has_quato = len(quotation_list)
            if c == '\'' or c == '"':
                if len(quotation_list) != 0 and quotation_list[-1] == c:
                    quotation_list.pop()
                else:
                    quotation_list.append(c)
            if has_quato and len(quotation_list) == 0:
                num = num + 1
                need_set_previous = True

        previous_char = c
        if need_set_previous:
            previous_char = ' '
    if previous_char != ' ' and len(quotation_list) == 0:
        num = num + 1
    return num


result = [get_trick_word(s) for s in input_str_list]
print(result)

And the result is:

# ['TopLevel  ', '    DISPLAY "In TopLevel. Starting to run program"  ', '    PERFORM OneLevelDown  ', '    DISPLAY "Back in TopLevel."  ', '    STOP RUN.']
# [1, 2, 2, 2, 2]