Unable to build Spark application with multiple main classes for Databricks job
Questions : Unable to build Spark application with multiple main classes for Databricks job
I have a spark application that contains in4codes_databricks multiple spark jobs to be run on Azure data in4codes_databricks bricks. I want to build and package the in4codes_databricks application into a fat jar. The application in4codes_databricks is able to compile successfully. While I am in4codes_databricks trying to package (command: sbt package) the in4codes_databricks application, it gives an error “[warn] in4codes_databricks multiple main classes detected: run ‘show in4codes_databricks discoveredMainClasses’ to see the list”.
How to build the application jar (Without in4codes_databricks specifying any main class) so that I can in4codes_databricks upload it to Databricks job and specify the in4codes_databricks main classpath over there?
Answers 1 : of Unable to build Spark application with multiple main classes for Databricks job
This message is just a warning (see in4codes_apache-spark [warn] in it), it doesn’t prevent in4codes_apache-spark generation of the jar files (normal or in4codes_apache-spark fat). Then you can upload resulting jar in4codes_apache-spark to DBFS (or ADLS for newer Databricks in4codes_apache-spark Runtime versions), and create Databricks in4codes_apache-spark job either as Jar task, or Spark Submit in4codes_apache-spark task.
If sbt fails, and doesn’t produce jars, in4codes_apache-spark then you have some plugin that forces in4codes_apache-spark error on the warnings.
Also notice that sbt package doesn’t in4codes_apache-spark produce fat jar – it produce jar only in4codes_apache-spark for classes in your project. You will in4codes_apache-spark need to use sbt assembly (install in4codes_apache-spark sbt-assembly plugin for that) to in4codes_apache-spark generate fat jar, but make sure that you in4codes_apache-spark marked Spark & Delta dependencies as in4codes_apache-spark provided.