org.apache.livy

Interface JobContext

public interface JobContext

Holds runtime information about the job execution context. An instance of this class is kept on the node hosting a remote Spark context and is made available to jobs being executed via RemoteSparkContext#submit().

Method Summary

Modifier and Type	Method and Description
void	createStreamingContext(long batchDuration) Creates the SparkStreaming context.
File	getLocalTmpDir()
<E> E	getSharedObject(String name) Get shared object
org.apache.spark.sql.hive.HiveContext	hivectx()
<E> E	removeSharedObject(String name) Remove shared object from cache
org.apache.spark.api.java.JavaSparkContext	sc()
<E> void	setSharedObject(String name, E object) Set shared object, it will replace the old one if already existed
<E> E	sparkSession()
org.apache.spark.sql.SQLContext	sqlctx()
void	stopStreamingCtx() Stops the SparkStreaming context.
org.apache.spark.streaming.api.java.JavaStreamingContext	streamingctx()

Method Detail

sc

org.apache.spark.api.java.JavaSparkContext sc()

Returns:

The shared SparkContext instance.

sqlctx

org.apache.spark.sql.SQLContext sqlctx()

Returns:

The shared SQLContext instance.

hivectx

org.apache.spark.sql.hive.HiveContext hivectx()

Returns:

The shared HiveContext instance.

streamingctx

org.apache.spark.streaming.api.java.JavaStreamingContext streamingctx()

Returns:

The JavaStreamingContext which has already been created.

getSharedObject

<E> E getSharedObject(String name)
               throws NoSuchElementException

Get shared object

Type Parameters:

E - The type of the shared object

Parameters:

name - Name of the shared object to return

Returns:

The shared object matching name

Throws:

NoSuchElementException

setSharedObject

<E> void setSharedObject(String name,
                         E object)

Set shared object, it will replace the old one if already existed

Type Parameters:

E - The type of the shared object

Parameters:

name - Name of the shared object to be set

object - The object to be set

removeSharedObject

<E> E removeSharedObject(String name)

Remove shared object from cache

Type Parameters:

E - The type of the shared object

Parameters:

name - Name of the shared object to be removed

Returns:

The object that was removed

createStreamingContext

void createStreamingContext(long batchDuration)

Creates the SparkStreaming context.

Parameters:

batchDuration - Time interval at which streaming data will be divided into batches, in milliseconds.

stopStreamingCtx

void stopStreamingCtx()

Stops the SparkStreaming context.

getLocalTmpDir

File getLocalTmpDir()

Returns:

A local tmp dir specific to the context

sparkSession

<E> E sparkSession()
            throws Exception

Type Parameters:

E - The type of the sparksession object

Returns:

The SparkSession if it exists

Throws:

Exception - If SparkSession does not exist