List comprehension guard being ignored

user_1612432 0
Viewed 103

Lets say I have a dict:

d = {'AA': 'BB', 'BB': None}

And this for comprehension:

[v for t in u'{}'.format(v.lower()) for k, v in d.items()]

Its obvious that its going to fail with 'NoneType' object has no attribute 'lower', so lets try this one:

[v for t in u'{}'.format(v.lower()) for k, v in d.items() if v is not None]

The same happens!, Why? If I add another guard:

[v for t in u'{}'.format(v.lower()) if v is not None for k, v in d.items() if v is not None]

Same thing.

Why is v.lower() being called even with the guards?

However, this works:

for k,v in d.items():
    if v is not None:
        [v for t in u'{}'.format(v.lower())]

Update

This is the actual code that is giving me the problem, The above code was to simplify the example, but given that the answer provided below, I think I will post the actual code:

x = {'A': 'This is a Line to Be tokenized'}
for k,v in x.items():
    if v is not None:
        pat = [{'LOWER': str(t)} for t in tokenizer(u'{}'.format(v.lower()))]

This generate patters for Spacy, in this format:

[{'LOWER': 'this'},
 {'LOWER': 'is'},
 {'LOWER': 'a'},
 {'LOWER': 'line'},
 {'LOWER': 'to'},
 {'LOWER': 'be'},
 {'LOWER': 'tokenized'}]

So, originally my for comprehension to generate that output is 

[{'LOWER': str(t)} for k, v in x.items() if v is not None for t in tokenizer(u'{}'.format(v))]

But, as mentioned above, when the value of the dictionary is None, it fails even when a guard is provided.

Update 2

Here are more examples:

x = {'A': 'This is a Line to Be tokenized', 'B': 'Hello'}
for k,v in x.items():
    if v is not None:
        pat = [{'LOWER': str(t)} for t in tokenizer(u'{}'.format(v.lower()))]
        print(pat)

# [{'LOWER': 'this'}, {'LOWER': 'is'}, {'LOWER': 'a'}, {'LOWER': 'line'}, {'LOWER': 'to'}, {'LOWER': 'be'}, {'LOWER': 'tokenized'}]
# [{'LOWER': 'hello'}]

So, basically I want to convert that loop into a comprehension.

spacy list-comprehension python
2 Answers

You need to change the order of the fors and ifs:

>>> [v for k, v in d.items() if v is not None for t in u'{}'.format(v.lower())]
['BB', 'BB']
>>>

  • The if statement has to be before the location where it's gonna cause an error.

  • The second for loop has to be after the for loop which contains the iterators that the second for loop uses.

user_87083640

It can be written as follows:

x = {'A': 'This is a Line to Be tokenized', 'B':None}
for k,v in x.items():
    if v is not None:
        pat = [{'LOWER': str(t)} for t in tokenizer(u'{}'.format(v.lower()))]


res = [[{'LOWER': str(t)} for t in tokenizer(u'{}'.format(v.lower()))] for k, v in x.items() if v is not None]

Output:

[[{'LOWER': 'this'},
  {'LOWER': 'is'},
  {'LOWER': 'a'},
  {'LOWER': 'line'},
  {'LOWER': 'to'},
  {'LOWER': 'be'},
  {'LOWER': 'tokenized'}]]

However, at this point, you do not gain a lot from writing it in a single line, but lose a lot in terms of readability. I would encourage you not to write this in a single line.

user_106185400
Related