BYOS (bring your own spark) connecting to Cassandra with TLS/SSL

In order to connect byos Spark to Cassandra using TLS/SSL your shell conf should look something like the following:

pyspark --deploy-mode client \
--jars /home/hadoop/extrajars/dse-byos_2.11-5.1.10.jar \
--properties-file /home/hadoop/extrajars/byos.properties \
--conf spark.cassandra.auth.username=cassandra \
--conf spark.cassandra.auth.password=foo \
--conf spark.cassandra.connection.ssl.enabled=true \
--conf spark.cassandra.connection.ssl.keyStore.password=foo \
--conf spark.cassandra.connection.ssl.keyStore.path=/home/ec2-user/spark_config/keystore.jks \
--conf spark.cassandra.connection.ssl.trustStore.password=foo \
--conf spark.cassandra.connection.ssl.trustStore.path=/home/ec2-user/spark_config/truststore.jks

The dse-byos_2.11–5.1.10.jar is specific to the DSE Cassandra version. The nomenclature is 2.11 scala with 5.1.10 DSE. That jar file contains the spark Cassandra connector, so you don’t need to add a package for the spark Cassandra connector and in fact you shouldn’t because it may cause you problems down the line.

The keystore/truststore was moved to my local where I’m running spark, and it’s the same as what I have on the dse cluster nodes.

That should allow you to connect to Cassandra with TLS/SSL.