sedona.spark.geoarrow package

Submodules

sedona.spark.geoarrow.geoarrow module

sedona.spark.geoarrow.geoarrow.create_spatial_dataframe(spark: SparkSession, gdf: gpd.GeoDataFrame) → DataFrame[source]

Create a Spark DataFrame from a GeoPandas GeoDataFrame, transferring the data with Arrow.

This is faster than spark.createDataFrame(gdf) for large GeoDataFrames. Geometry columns become Sedona geometry columns, and the other columns get the Spark types of their Arrow types.

Parameters:

  • spark – The Spark session.

  • gdf – The GeoPandas GeoDataFrame to convert. It must have at least one geometry column.

Returns:

A Spark DataFrame with the columns of gdf.

Raises:

ValueError – If gdf has no geometry column.

sedona.spark.geoarrow.geoarrow.crs_to_json(crs)[source]
sedona.spark.geoarrow.geoarrow.dataframe_to_arrow(df, crs=None)[source]

Collect a DataFrame as a PyArrow Table

In the output Table, geometry will be encoded as a GeoArrow extension type. The resulting output is compatible with lonboard.viz(), geopandas.GeoDataFrame.from_arrow(), or any library compatible with GeoArrow extension types.

Parameters:

  • df – A Spark DataFrame

  • crs – A CRS-like object (e.g., pyproj.CRS or string interpretable by pyproj.CRS). If provided, this will override any CRS present in the output geometries. If omitted, the CRS will be inferred from the values present in the output if exactly one CRS is present in the output.

Returns:

A pyarrow.Table.

sedona.spark.geoarrow.geoarrow.dataframe_to_arrow_raw(df)[source]

Backport of toArrow() (available in Spark 4.0)

sedona.spark.geoarrow.geoarrow.infer_schema(gdf: gpd.GeoDataFrame) → StructType[source]
sedona.spark.geoarrow.geoarrow.try_register_extension_types()[source]

Try to register extension types using geoarrow-types

Do this defensively, because it can fail if the extension type was registered in some other way (notably: old versions of geoarrow-pyarrow, which is a dependency of Kepler).

sedona.spark.geoarrow.geoarrow.unique_srid_from_ewkb(obj)[source]
sedona.spark.geoarrow.geoarrow.wrap_geoarrow_extension(col, spec, crs)[source]
sedona.spark.geoarrow.geoarrow.wrap_geoarrow_field(field, col, crs)[source]

Module contents