pyspark.pandas.read_excel#

pyspark.pandas.read_excel(io, sheet_name=0, header=0, names=None, index_col=None, usecols=None, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skiprows=None, nrows=None, na_values=None, keep_default_na=True, verbose=False, parse_dates=False, date_parser=None, thousands=None, comment=None, skipfooter=0, **kwds)[source]#

Read an Excel file into a pandas-on-Spark DataFrame or Series.

Support both xls and xlsx file extensions from a local filesystem or URL. Support an option to read a single sheet or a list of sheets.

Parameters
iostr, file descriptor, pathlib.Path, ExcelFile or xlrd.Book

The string could be a URL. The value URL must be available in Spark’s DataFrameReader.

Note

If the underlying Spark is below 3.0, the parameter as a string is not supported. You can use ps.from_pandas(pd.read_excel(…)) as a workaround.

sheet_namestr, int, list, or None, default 0

Strings are used for sheet names. Integers are used in zero-indexed sheet positions. Lists of strings/integers are used to request multiple sheets. Specify None to get all sheets.

Available cases:

  • Defaults to 0: 1st sheet as a DataFrame

  • 1: 2nd sheet as a DataFrame

  • "Sheet1": Load sheet with name “Sheet1”

  • [0, 1, "Sheet5"]: Load first, second and sheet named “Sheet5” as a dict of DataFrame

  • None: All sheets.

headerint, list of int, default 0

Row (0-indexed) to use for the column labels of the parsed DataFrame. If a list of integers is passed those row positions will be combined into a MultiIndex. Use None if there is no header.

namesarray-like, default None

List of column names to use. If file contains no header row, then you should explicitly pass header=None.

index_colint, list of int, default None

Column (0-indexed) to use as the row labels of the DataFrame. Pass None if there is no such column. If a list is passed, those columns will be combined into a MultiIndex. If a subset of data is selected with usecols, index_col is based on the subset.

usecolsint, str, list-like, or callable default None

Return a subset of the columns.

  • If None, then parse all columns.

  • If str, then indicates comma separated list of Excel column letters and column ranges (e.g. “A:E” or “A,C,E:F”). Ranges are inclusive of both sides.

  • If list of int, then indicates list of column numbers to be parsed.

  • If list of string, then indicates list of column names to be parsed.

  • If callable, then evaluate each column name against it and parse the column if the callable returns True.

dtypeType name or dict of column -> type, default None

Data type for data or columns. E.g. {‘a’: np.float64, ‘b’: np.int32} Use object to preserve data as stored in Excel and not interpret dtype. If converters are specified, they will be applied INSTEAD of dtype conversion.

enginestr, default None

If io is not a buffer or path, this must be set to identify io. Acceptable values are None or xlrd.

convertersdict, default None

Dict of functions for converting values in certain columns. Keys can either be integers or column labels, values are functions that take one input argument, the Excel cell content, and return the transformed content.

true_valueslist, default None

Values to consider as True.

false_valueslist, default None

Values to consider as False.

skiprowslist-like

Rows to skip at the beginning (0-indexed).

nrowsint, default None

Number of rows to parse.

na_valuesscalar, str, list-like, or dict, default None

Additional strings to recognize as NA/NaN. If dict passed, specific per-column NA values. By default the following values are interpreted as NaN.

keep_default_nabool, default True

If na_values are specified and keep_default_na is False the default NaN values are overridden, otherwise they’re appended to.

verbosebool, default False

Indicate number of NA values placed in non-numeric columns.

parse_datesbool, list-like, or dict, default False

The behavior is as follows:

  • bool. If True -> try parsing the index.

  • list of int or names. e.g. If [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column.

  • list of lists. e.g. If [[1, 3]] -> combine columns 1 and 3 and parse as a single date column.

  • dict, e.g. {{‘foo’ : [1, 3]}} -> parse columns 1, 3 as date and call result ‘foo’

If a column or index contains an unparseable date, the entire column or index will be returned unaltered as an object data type. For non-standard datetime parsing, use pd.to_datetime after pd.read_csv

Note: A fast-path exists for iso8601-formatted dates.

date_parserfunction, optional

Function to use for converting a sequence of string columns to an array of datetime instances. The default uses dateutil.parser.parser to do the conversion. pandas-on-Spark will try to call date_parser in three different ways, advancing to the next if an exception occurs: 1) Pass one or more arrays (as defined by parse_dates) as arguments; 2) concatenate (row-wise) the string values from the columns defined by parse_dates into a single array and pass that; and 3) call date_parser once for each row using one or more strings (corresponding to the columns defined by parse_dates) as arguments.

thousandsstr, default None

Thousands separator for parsing string columns to numeric. Note that this parameter is only necessary for columns stored as TEXT in Excel, any numeric columns will automatically be parsed, regardless of display format.

commentstr, default None

Comments out remainder of line. Pass a character or characters to this argument to indicate comments in the input file. Any data between the comment string and the end of the current line is ignored.

skipfooterint, default 0

Rows at the end to skip (0-indexed).

**kwdsoptional

Optional keyword arguments can be passed to TextFileReader.

Returns
DataFrame or dict of DataFrames

DataFrame from the passed in Excel file. See notes in sheet_name argument for more information on when a dict of DataFrames is returned.

See also

DataFrame.to_excel

Write DataFrame to an Excel file.

DataFrame.to_csv

Write DataFrame to a comma-separated values (csv) file.

read_csv

Read a comma-separated values (csv) file into DataFrame.

Examples

The file can be read using the file name as string or an open file object:

>>> ps.read_excel('tmp.xlsx', index_col=0)  
       Name  Value
0   string1      1
1   string2      2
2  #Comment      3

>>> ps.read_excel(open('tmp.xlsx', 'rb'),
...               sheet_name='Sheet3')  
   Unnamed: 0      Name  Value
0           0   string1      1
1           1   string2      2
2           2  #Comment      3

Index and header can be specified via the index_col and header arguments

>>> ps.read_excel('tmp.xlsx', index_col=None, header=None)  
     0         1      2
0  NaN      Name  Value
1  0.0   string1      1
2  1.0   string2      2
3  2.0  #Comment      3

Column types are inferred but can be explicitly specified

>>> ps.read_excel('tmp.xlsx', index_col=0,
...               dtype={'Name': str, 'Value': float})  
       Name  Value
0   string1    1.0
1   string2    2.0
2  #Comment    3.0

True, False, and NA values, and thousands separators have defaults, but can be explicitly specified, too. Supply the values you would like as strings or lists of strings!

>>> ps.read_excel('tmp.xlsx', index_col=0,
...               na_values=['string1', 'string2'])  
       Name  Value
0      None      1
1      None      2
2  #Comment      3

Comment lines in the excel input file can be skipped using the comment kwarg

>>> ps.read_excel('tmp.xlsx', index_col=0, comment='#')  
      Name  Value
0  string1    1.0
1  string2    2.0
2     None    NaN