Mongodb Spark Connector Py4j.protocol.py4jjavaerror: An Error Occurred While Calling O50.load

I have been able to load this MongoDB database before, but am now receiving an error I haven't been able to figure out. Here is how I start my Spark session: spark = SparkSession.

Solution 1:

I figured out the answer to my question. This was a compatibility issue with the Mongo-Spark connector and the version of Spark that I upgraded to. Specifically, the findTightestCommonTypeOfTwo value was renamed in the PR:

https://github.com/apache/spark/pull/16786/files

For Spark 2.2.0 the compatible Mongo-Spark connector is also 2.2.0, thus in my example, the package would be loaded like this:

--packages org.mongodb.spark:mongo-spark-connector_2.11:2.2.0\

This could change in the future so when using the connector, you should check for compatibility with the version of Spark being used.