Class SparkContext

Create a new SparkContext. At least the master and app name should be set, either through the named parameters here or through conf.

Parameters:

• master - Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
• appName - A name for your job, to display on the cluster web UI.
• sparkHome - Location where Spark is installed on cluster nodes.
• pyFiles - Collection of .zip or .py files to send to the cluster and add to PYTHONPATH. These can be paths on the local file system or HDFS, HTTP, HTTPS, or FTP URLs.
• environment - A dictionary of environment variables to set on worker nodes.
• batchSize - The number of Python objects represented as a single Java object. Set 1 to disable batching or -1 to use an unlimited batch size.
• serializer - The serializer for RDDs.
• conf - A SparkConf object setting Spark properties.

  >>> from pyspark.context import SparkContext
  >>> sc = SparkContext('local', 'test')

  >>> sc2 = SparkContext('local', 'test2') # doctest: +IGNORE_EXCEPTION_DETAIL
  Traceback (most recent call last):
      ...
  ValueError:...

Overrides: object.__init__

Set a Java system property, such as spark.executor.memory. This must must be invoked before instantiating SparkContext.

Default level of parallelism to use when not given by user (e.g. for reduce tasks)

Decorators:

• @property

Distribute a local Python collection to form an RDD.

>>> sc.parallelize(range(5), 5).glom().collect()
[[0], [1], [2], [3], [4]]

Build the union of a list of RDDs.

This supports unions() of RDDs with different serialized formats, although this forces them to be reserialized using the default serializer:

>>> path = os.path.join(tempdir, "union-text.txt")
>>> with open(path, "w") as testFile:
...    testFile.write("Hello")
>>> textFile = sc.textFile(path)
>>> textFile.collect()
[u'Hello']
>>> parallelized = sc.parallelize(["World!"])
>>> sorted(sc.union([textFile, parallelized]).collect())
[u'Hello', 'World!']

Broadcast a read-only variable to the cluster, returning a Broadcast object for reading it in distributed functions. The variable will be sent to each cluster only once.

Create an Accumulator with the given initial value, using a given AccumulatorParam helper object to define how to add values of the data type if provided. Default AccumulatorParams are used for integers and floating-point numbers if you do not provide one. For other types, a custom AccumulatorParam can be used.

Add a file to be downloaded with this Spark job on every node. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI.

To access the file in Spark jobs, use SparkFiles.get(path) to find its download location.

>>> from pyspark import SparkFiles
>>> path = os.path.join(tempdir, "test.txt")
>>> with open(path, "w") as testFile:
...    testFile.write("100")
>>> sc.addFile(path)
>>> def func(iterator):
...    with open(SparkFiles.get("test.txt")) as testFile:
...        fileVal = int(testFile.readline())
...        return [x * 100 for x in iterator]
>>> sc.parallelize([1, 2, 3, 4]).mapPartitions(func).collect()
[100, 200, 300, 400]

Add a .py or .zip dependency for all tasks to be executed on this SparkContext in the future. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI.

Set the directory under which RDDs are going to be checkpointed. The directory must be a HDFS path if running on a cluster.