pyspark.streaming.StreamingContext.getActiveOrCreate

classmethod StreamingContext.getActiveOrCreate(checkpointPath: str, setupFunc: Callable[], StreamingContext]) → pyspark.streaming.context.StreamingContext

Either return the active StreamingContext (i.e. currently started but not stopped), or recreate a StreamingContext from checkpoint data or create a new StreamingContext using the provided setupFunc function. If the checkpointPath is None or does not contain valid checkpoint data, then setupFunc will be called to create a new context and setup DStreams.

Parameters

checkpointPathstr

Checkpoint directory used in an earlier streaming program. Can be None if the intention is to always create a new context when there is no active context.

setupFuncfunction

Function to create a new JavaStreamingContext and setup DStreams

pyspark.streaming.StreamingContext.getActive pyspark.streaming.StreamingContext.getOrCreate