Loading Dependencies

When your code requires external libraries for Spark, %spark.dep helps load them from a maven repository. The %spark.dep interpreter leverages the Scala environment and enables you can write Scala expressions to call dependency load APIs.

// load Apache Commons RNG libraries from maven repository

Note that %spark.dep should be the first interpreter run in the notebook before %spark, %spark.pyspark or %spark.sql. Otherwise, %spark.dep will print several error messages and you'll need to shutdown and restart the container for the notebook again.