Why is Apache Livy session showing Application id NULL?

Viewed 2301

I've implemented a fully functional Spark 2.1.1 Standalone cluster, where I POST job batches via the curl command using Apache Livy 0.4. When consulting the Spark WEB UI I see my job along with its application id (something like this: app-20170803115145-0100), with application name, cores, time, state, etc. But when consulting the Livy WEB UI (http://localhost:8998 by default), I see the following structure:

| Batch Id | Application Id | State   |
| -------- | -------------- | ------- |
| 219      | null           | success |
| 220      | null           | running |

And if I GET the state of all the batches I obtain the following result:

{
  "from": 0,
  "total": 17,
  "sessions": [
    {
      "id": 219,
      "state": "success",
      "appId": null,
      "appInfo": {
        "driverLogUrl": null,
        "sparkUiUrl": null
      },
      "log": ["* APPLICATION OUTPUT"]
    },
    {
      "id": 220,
      "state": "running",
      "appId": null,
      "appInfo": {
        "driverLogUrl": null,
        "sparkUiUrl": null
      },
      "log": ["* APPLICATION OUTPUT"]
    },       
  ]
}

which is obviously correct, but I always see the null value in the appId field, also the driverLogUrl and the sparkUiUrl.

Is this field suppose to show the same application id that I see in the Spark WEB UI ? If so, how can I configure this or it has to be automatic ? Don't know if I'm missing some configuration line on the livy.conf or livy-env.sh file, because I can't find any example nor documentation regarding this.

This is my livy.conf file:

livy.server.host = IP_ADDRESS
livy.server.port = 8998
livy.spark.master = spark://MASTER_IP_ADDRESS:PORT
livy.spark.deploy-mode = cluster
livy.ui.enabled = true

This is the livy-env.sh file:

export JAVA_HOME=/opt/java8/jdk1.8.0_91
export SPARK_HOME=~/sparkFiles/spark-2.1.1-bin-hadoop2.7
export SPARK_CONF_DIR=${SPARK_HOME}/conf
export LIVY_LOG_DIR=~/LivyRestServer/logs

Please let me know if you need some more information.


UPDATE For those who has had the same issue. Using the Standalone cluster manager unfortunately I couldn't fixed, but then I had the necessity to change it to YARN to better manage pools and queues, and that magically fix the issue, I was able to see all that information. Don't know why the standalone manager can't push the applicationId to Livy, but YARN does, so, it just fixed by itself, I didn't change anything in the Livy conf. files either.

1 Answers

livy.conf

livy.spark.master default is local mode,need set yarn mode;

Maybe it’s a livy bug,From a design perspective, it should also be obtained in local mode. It may also be livy in the future only for the yarn mode.

Related