Databricks: How to switch from R Dataframe to Pandas Dataframe (R to python in the same notebook)

user_12886251 0
Viewed 3392

I am writing R code in a Databricks notebook that performs several operations in R. Once the dataframe is cleaned up, I would like to invoke it in a python cell using '%python' and therefore use python code to continue operations on the dataframe.

I would thus like to transform, within the python block, my R Dataframe into a Pandas dataframe. Does anybody know how to do this? Thanks!

pandas databricks dataframe r python
2 Answers

I think the namespace between different kernels is separate on Databricks. So even in the same notebook, you will not see an R variable in Python or vice versa.

My understanding is that there are two methods to share data between kernels: 1) using the filesystem (csv, etc) and 2) temporary Databricks tables. I believe the latter is the more typical route[1].

  1. Filesystem:
%r
write.csv(df, "/FileStore/tmp.csv")

%python
import pandas as pd
df = pd.read_csv("/FileStore/tmp.csv")
  1. Temporary databricks table:
%r
library(SparkR)
sparkR.session()
df <- read.df("path/to/original_file.csv", source="csv")
registerTempTable(df, "tmp_df")

%python
df = spark.sql("select * from tmp_df").toPandas()

[1] https://forums.databricks.com/questions/16039/use-python-and-r-variable-in-the-same-notebook-amo.html

user_6971120

Note: Since rpy2 release 3.3.0 explicit conversion is done as follows

import rpy2.robjects as ro

dt = pd.DataFrame()

To R DataFrame

r_dt = ro.conversion.py2rpy(dt)

To pandas DataFrame

pd_dt = ro.conversion.rpy2py(r_dt)
user_102052200
Related