flake8 linting for databricks python code in github using workflows

user_496289 0
Viewed 1629

I have my databricks python code in github. I setup a basic workflow to lint the python code using flake8. This fails because the names that are implicitly available to my script (like spark, sc, dbutils, getArgument etc) when it runs on databricks are not available when flake8 lints it outside databricks (in github ubuntu vm).

How can I lint databricks notebooks in github using flake8?

E.g. errors I get:

test.py:1:1: F821 undefined name 'dbutils'
test.py:3:11: F821 undefined name 'getArgument'
test.py:5:1: F821 undefined name 'dbutils'
test.py:7:11: F821 undefined name 'spark'

my notebook in github:

dbutils.widgets.text("my_jdbcurl", "default my_jdbcurl")

jdbcurl = getArgument("my_jdbcurl")

dbutils.fs.ls(".")

df_node = spark.read.format("jdbc")\
  .option("driver", "org.mariadb.jdbc.Driver")\
  .option("url", jdbcurl)\
  .option("dbtable", "my_table")\
  .option("user", "my_username")\
  .option("password", "my_pswd")\
  .load()

my .github/workflows/lint.yml

on:
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v1
      with:
        python-version: 3.8
    - run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Lint with flake8
      run: |
        pip install flake8
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
flake8 databricks apache-spark github python
2 Answers

One thing you can do is this:

from pyspark.sql import Spark Session


spark = SparkSession.builder.getOrCreate()

This will work with or without Databricks, in normal Python or in the pyspark client.

To detect if you are in a file or in a Databricks notebook, you can run:

try:
    __file__
    print("We are in a file, like in our IDE or being tested by flake8.")
except NameError:
    print("We are in a Databricks notebook. Act accordingly.")

You could then conditionally initialize or create dummy variables for display() and other tools.

This is only a partial solution. I am working on a better solution, I will keep this answer updated.

user_139690

This is my opinion, all linters dont work for all use cases, this is what I do. I am using a pre-commit hook and ignoring rule F821.

# Flake rules: https://lintlyci.github.io/Flake8Rules/
- repo: https://gitlab.com/pycqa/flake8
  rev: 3.8.4
  hooks:
    - id: flake8
      exclude: (^docs/)
      additional_dependencies: [flake8-typing-imports==1.7.0]
      # F821 undefined name
      args:
        [
          "--max-line-length=127",
          "--config=setup.cfg",
          "--ignore=F821",
        ]

To match your syntax, add the --ignore flag:

flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --ignore=C901,F821 --statistics
user_66798670
Related