org.apache.livy

Interface LivyClient

public interface LivyClient

A client for submitting Spark-based jobs to a Livy backend.

Method Summary

Modifier and Type	Method and Description
Future<?>	addFile(URI uri) Adds a file to the running remote context.
Future<?>	addJar(URI uri) Adds a jar file to the running remote context.
<T> Future<T>	run(Job<T> job) Asks the remote context to run a job immediately.
void	stop(boolean shutdownContext) Stops the remote context.
<T> JobHandle<T>	submit(Job<T> job) Submits a job for asynchronous execution.
Future<?>	uploadFile(File file) Upload a file to be passed to the Spark application
Future<?>	uploadJar(File jar) Upload a jar to be added to the Spark application classpath

Method Detail

submit

<T> JobHandle<T> submit(Job<T> job)

Submits a job for asynchronous execution.

Type Parameters:

T - The return type of the job

Parameters:

job - The job to execute.

Returns:

A handle that be used to monitor the job.

run

<T> Future<T> run(Job<T> job)

Asks the remote context to run a job immediately.

Normally, the remote context will queue jobs and execute them based on how many worker threads have been configured. This method will run the submitted job in the same thread processing the RPC message, so that queueing does not apply.

It's recommended that this method only be used to run code that finishes quickly. This avoids interfering with the normal operation of the context.

Type Parameters:

T - The return type of the job

Parameters:

job - The job to execute.

Returns:

A future to monitor the result of the job.

stop

void stop(boolean shutdownContext)

Stops the remote context. Any pending jobs will be cancelled, and the remote context will be torn down.

Parameters:

shutdownContext - Whether to shutdown the underlying Spark context. If false, the context will keep running and it's still possible to send commands to it, if the backend being used supports it.

uploadJar

Future<?> uploadJar(File jar)

Upload a jar to be added to the Spark application classpath

Parameters:

jar - The local file to be uploaded

Returns:

A future that can be used to monitor this operation

addJar

Future<?> addJar(URI uri)

Adds a jar file to the running remote context.

Note that the URL should be reachable by the Spark driver process. If running the driver in cluster mode, it may reside on a different host, meaning "file:" URLs have to exist on that node (and not on the client machine).

If the provided URI has no scheme, it's considered to be relative to the default file system configured in the Livy server.

Parameters:

uri - The location of the jar file.

Returns:

A future that can be used to monitor the operation.

uploadFile

Future<?> uploadFile(File file)

Upload a file to be passed to the Spark application

Parameters:

file - The local file to be uploaded

Returns:

A future that can be used to monitor this operation

addFile

Future<?> addFile(URI uri)

Adds a file to the running remote context.

Note that the URL should be reachable by the Spark driver process. If running the driver in cluster mode, it may reside on a different host, meaning "file:" URLs have to exist on that node (and not on the client machine).

If the provided URI has no scheme, it's considered to be relative to the default file system configured in the Livy server.

Parameters:

uri - The location of the file.

Returns:

A future that can be used to monitor the operation.