I've been plodding along making my way through some Data Science courses on Coursera. I've picked up some familiarity with Python, Numpy, Pandas, Tensorflow, Keras etc... Now I need to set up some dev tooling environment to get more productive and efficient as I proceed further.
I'm doing almost all my work in Jupyter Notebooks, whether mostly on Coursera notebooks or VS Code notebooks (but have fooled around with Spyder and DataSpell).
But I've used these enough now to have developed the sinking feeling that I am not using these tools correctly.
I find myself with a bunch of code cells that are mostly commands that I am trying out while learning the syntax of a function, trying to look at the output and moving to the next statement, while hoping that the code cells I keep will give me back the same memory state when I run the notebook from start to finish later. I am constantly adding a short code cell just to inspect a variable to see if it's in the expected state.
So my notebooks become a messy tangle of, for instance with Pandas, of code cells littered around containing:
df.head()
df.describe()
len(df)
df.dtypes
aggdf = df.groupby(['col1', 'col2']).agg(min,max,median)
len(aggdf)
aggdf.describe()
aggdf.reset_index()
aggdf.head()
....
I'm hoping I'm giving at least a passing idea of how I'm using notebooks, and illustrating a useful example from which to discuss further...
Now, even though I feel I'm not using them correctly, I do believe there is something intrinsically exciting and very powerful about Jupyter Notebooks. I need the cipherkey to unlock it's potential.
I would suspect that Variable Explorers and such do provide some assistance, but I'm looking for more. Whether in Spyder, VS.Code or DataSpell/Datalore/PyCharm I have this available. I also can fire up an Ipython console side by side with my Notebook.
But why is it, that regardless of which IDE I'm using, the notebook and console are running in different kernels, thus making it not possible to inspect notebook variables or experiment with commands or call functions from the notebook?
I assume that some greater minds than mine were at work when creating these tools, and many more greater minds are successfully making use of them.
So, I'm hoping that you Jupyter Python ninjas can course correct me and get me on the right track towards improving my overall productivity.
Any Comments or Suggestions or dialogue forming follow up questions are appreciated, and even some playful teasing about my incompetence, but only if it's funny... ;)
Or even a response summarizing the when, where and how you use Jupyter Notebooks would be fantastic too!