Apache Livy: query Spark SQL via REST: possible?

user_2793858 0
Viewed 5057

The apache Livy documentation is sparse: is it possible to return Spark SQL query resultsets as REST calls using Apache Livy? The calling application is mobile and it cannot use odbc/jdbc to connect. So the Spark thriftserver is not an option.

apache-spark
2 Answers

If you're using Livy 0.7.0 or up, you don't really need magic strings.

  1. Create a session:
curl --location --request POST 'http://<host>:<port>/sessions' \
--header 'Content-Type: application/json' \
--data-raw '{
    "kind": "sql",
    "proxyUser": "cooL_user"
}'
  1. Run a query:
curl --location --request POST 'http://<host>:<port>/sessions/<sessionid>/statements' \
--header 'Content-Type: application/json' \
--data-raw '{
    "code": "select 1"
}'
  1. Periodically poll on the result:
curl --location --request GET http://<host>:<port>/sessions/<sessionid>/statements/0 | jq '.output.data.application/json.data'

That's it. Now you can use this approach with any language beside Bash. There's already a Python Library called PyLivy which roughly follows this approach under the hood.

user_37282330
Related