Skip to content
πŸŽ‰ Apache Sedona 1.9.1 is out now! πŸ—ΊοΈ Geography SQL functions, Box2D & Box3D types, raster Python UDFs & more. Read the release notes β†’

Set up your Apache Spark cluster

Download a Spark distribution from Spark download page.

Preliminary

  1. Set up a password-less SSH on your cluster. Each master-worker pair should have bidirectional password-less SSH.
  2. Make sure you have installed JRE 1.8 or later.
  3. Add the list of your workers' IP address in ./conf/slaves
  4. Besides the necessary Spark settings, you may need to add the following lines in the Spark configuration files to avoid Sedona memory errors:

In ./conf/spark-defaults.conf

spark.driver.memory 10g
spark.network.timeout 1000s
spark.driver.maxResultSize 5g
  • spark.driver.memory tells Spark to allocate enough memory for the driver program because Sedona needs to build global grid files (global index) on the driver program. If you have a large amount of data (normally, over 100 GB), set this parameter to 2~5 GB will be good. Otherwise, you may observe "out of memory" error.
  • spark.network.timeout is the default timeout for all network interactions. Sometimes, spatial join query takes longer time to shuffle data. This will ensure Spark has enough patience to wait for the result.
  • spark.driver.maxResultSize is the limit of total size of serialized results of all partitions for each Spark action. Sometimes, the result size of spatial queries is large. The "Collect" operation may throw errors.

For more details of Spark parameters, please visit Spark Website.

Start your cluster

Go the root folder of the uncompressed Apache Spark folder. Start your Spark cluster via a terminal

./sbin/start-all.sh

Start a Spark Connect server

Spark Connect runs SQL functions in the remote server process. The Sedona jars and SQL extension must therefore be configured when the server starts. Installing Sedona only on the Python client, or calling SedonaContext.create() after connecting, does not register Sedona SQL functions on the server.

From the root of your Spark distribution, start the server with packages that match your Spark and Scala versions:

./sbin/start-connect-server.sh \
  --packages org.apache.spark:spark-connect_<SCALA_VERSION>:<SPARK_VERSION>,org.apache.sedona:sedona-spark-shaded-<SPARK_MAJOR_MINOR>_<SCALA_VERSION>:1.9.1,org.datasyslab:geotools-wrapper:1.9.1-33.5 \
  --repositories https://artifacts.unidata.ucar.edu/repository/unidata-all \
  --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
  --conf spark.kryo.registrator=org.apache.sedona.core.serde.SedonaKryoRegistrator \
  --conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions

Replace <SPARK_VERSION>, <SPARK_MAJOR_MINOR>, and <SCALA_VERSION> with the versions from your Spark distribution. For example, Spark 3.5 built with Scala 2.12 uses spark-connect_2.12:<SPARK_VERSION> and sedona-spark-shaded-3.5_2.12:1.9.1. See Sedona Maven coordinates for the supported combinations. Spark 4.0 and later include Spark Connect, so omit the org.apache.spark:spark-connect_... package from --packages for those versions.

After the server starts, connect from Python using the server URL. The default Spark Connect port is 15002:

from pyspark.sql import SparkSession
from sedona.spark import SedonaContext

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
sedona = SedonaContext.create(spark)

sedona.sql("SELECT ST_GeomFromWKT('POINT (1 2)') AS geom").show()

The Sedona Python package on the client should use the same Sedona version as the server package. See Install Sedona Python for client installation details.