Pylint: same pylint and pandas version on 2 machines, 1 fails

user_264745 0
Viewed 93

I have 2 places running the same linting job:

Machine 1: Ubuntu over SSH

  • pandas==1.2.3
  • pylint==2.7.4
  • python 3.8.10

Machine 2: Gitlab CI Docker image, python:3.8.12-buster

  • pandas==1.2.3
  • pylint==2.7.4
  • Python 3.8.12

The Ubuntu machine is able to lint all the code fine, and it has for many months. Same for the CI job, except it had been running Python 3.7.8. Now that I upgraded the Docker image to Python 3.8.12, it throws several no-member linting errors on some Pandas objects. I've tried clearing CI caches etc.

I wish I could provide something more reproducible. But, to check my understanding of what a linter is doing, is it theoretically possible that a small version difference in python messes up pylint like this? For something like a no-member error on Pandas objects, I would think the dominant factor is the pandas version, but those are equal, so I'm confused!

Update:

I've looked at the Pandas code for pd.read_sql_query, which is what's causing the no-member error. It says:

def read_sql_query(
    sql,
    con,
    index_col=None,
    coerce_float=True,
    params=None,
    parse_dates=None,
    chunksize: Optional[int] = None,
) -> Union[DataFrame, Iterator[DataFrame]]:

In Docker, I get E1101: Generator 'generator' has no 'query' member (no-member) (because I'm running .query on the returned dataframe). So it seems Pylint thinks that this function returns a generator. But it does not make this assumption in my other setup. (I've also verified the SHA sum of pandas/io/sql.py matches). This seems similar to this issue, but I am still baffled by the discrepancy in environments.

pandas pylint
1 Answers

A fix that worked was to bump a limit like:

init-hook = "import astroid; astroid.context.InferenceContext.max_inferred = 500"

in my .pylintrc file, as explained here.

I'm unsure why/if this is connected to my change in Python version, but I'm happy to use this and move on for now. It's probably complex.

(Another hack was to write a function that returns the passed arg if the passed arg is a dataframe, and returns 1 dataframe if the passed arg is an iterable of dataframes. So the ambiguous-type object could be passed through this wrapper to clarify things for Pylint. While this was more intrusive on our codebase, we had dozens of calls to pd.read_csv and pd.real_sql_query, and only about 3 calls caused confusion for Pylint, so we almost used this solution)

user_2647450
Related