Undefined variable in lambda function

user_5013752 0
Viewed 1159

I have the code :

from functools import reduce

public_ids = [1,2,3,4,5]
filepath = '/path/to/file/'

rdd = sc.textFile(
    filepath
)

new_rdd = reduce(
    lambda a, b: a.filter(
        lambda x: b not in x
    ),
    public_ids,
    rdd
)

This code is suppose to filter lines in a rdd according to a list of ids. The rdd is created from files located in filepath using spark context sc's textFile method.

This code works fine, but pylint raises the error :

E: Undefined variable 'b' (undefined-variable)

I believe the way I coded it is not the proper way. How can I change it so pylint does not raise the error again ? Or is it just a structure that pylint does not recognize properly ?

pyspark pylint python
1 Answers

Most likely it's a bug in pylint.

Here is a similar bug report from 2 years ago

foo = lambda x: lambda: x + 1 print(foo(1)())

correctly prints 2 when run, but pylint incorrectly reports

E: 1,24: Undefined variable 'x' (undefined-variable)

This is a regression from pylint 1.4.x.

And here is a recent issue reporting the same issue on 11/14/2018

The issue has been reported at #760 and fixed by #2274. However, the fix is merged only into pylint 2.x which supports only python >= 3.4 leaving us with the bug unresolved in the pylint 1.x series for python 2.

EDIT

Looks like your false positive might be slightly different than the issue above, nevertheless, I would still consider this a bug.

I would try creating an issue on their repo and see what happens (if you decide to do so, please post link in comments so we can follow it.)

user_44111960
Related