How to use "sqlContext" in different notebooks when using one of them as a module (Pyspark)

Viewed 3806

I have a notebook a.pynb which has the function for read statement of a parquet file.

I am using a.pynb in another notebook b.pynb and in this new notebook, i am calling a function of a.pynb to read this parquet file and create a sqltable. But it always fails with

      Error: global name sqlContext is not defined, 

When it is defined in both the notebooks.

The exact code :

a.pynb ( Utils)

   sc = SparkContext.getOrCreate()
   sqlContext = SQLContext(sc)

    def parquet_read(file_name):
        df = sqlContext.read.parquet(file_name+"*.parquet")
        return df

In b.pynb I have used this function

    import nbimporter
    import a as commonUtils
    reload(commonUtils)
    sc = SparkContext.getOrCreate()
    sqlContext = SQLContext(sc)

    df2 = commonUtils.parquet_read("abc")

It always fails with

    Error: global name sqlContext is not defined, 

When it is defined in both the notebooks.

1 Answers
Related