Connect pyspark in client mode to DSE spark cassandra cluster — by Steve Lacerda

Pyspark does not support cluster for the deploy mode. What does that mean, it means that the driver must be run locally, in client mode, rather than on the DSE spark cluster, cluster mode.

To be able to connect to DSE remotely in client mode from your laptop, or other local system, you should follow these steps:

1. Download and install DSE on your local system. You will not need to start the services, but you need the dse tools in order to connect to the dse spark cluster.
2. Set the following config in the spark-defaults.conf on the local system:

spark.driver.port 38000
spark.blockManager.port 45000
spark.port.maxRetries 16

The above forces the ports, otherwise you will get random ports, which could cause problems through a firewall. To get through the firewall, you will have to open the following ports:

38000–38015, 45000–45015, 7077, 7080, 5598, 5599.

The reason the range is from 38000–38015, is because of the number you supplied to maxRetries above. What maxRetries does is attempts to connect on port 38000, if that’s in use or unavailable, then it tries, 38001, and so on for 16 attempts, then fails. The same holds for port 45000.

By the way, the ports I chose are totally arbitrary, you can choose whatever ports you want for driver.port and blockManager.port.

3. You should now be able to run spark in client mode, using something similar to the following command:

dse pyspark --master dse://10.101.33.200 --conf spark.cassandra.connection.host=10.101.32.110 

Instead of using spark-defaults.conf, you could also supply these parameters directly:

dse pyspark --master dse://10.101.33.200 --conf spark.cassandra.connection.host=10.101.32.110 --conf spark.driver.port=38000 --conf spark.blockManager.port=45000 --conf spark.port.maxRetries=16