Download Livy packages from here.
To run the Livy server, you will also need an Apache Spark installation. You can get Spark releases at https://spark.apache.org/downloads.html. Livy requires at least Spark 1.6 and supports both Scala 2.10 and 2.11 builds of Spark. To run Livy with local sessions, first export these variables:
Then start the server with:
Livy uses the Spark configuration under
SPARK_HOME by default. You can override the Spark configuration by setting the
SPARK_CONF_DIR environment variable before starting Livy.
It is strongly recommended to configure Spark to submit applications in YARN cluster mode. That makes sure that user sessions have their resources properly accounted for in the YARN cluster, and that the host running the Livy server doesn’t become overloaded when multiple user sessions are running.
Livy uses a few configuration files under configuration the directory, which by default is the conf directory under the
Livy installation. An alternative configuration directory can be provided by setting the
variable when starting Livy.
The configuration files used by Livy are:
Once the Livy server is running, you can connect to it on port 8998 (this can be changed with the
livy.server.port config option). Some examples to get started are provided here,
or you can check out the API documentation: