Spark Submit Command

Spark submit is a command line tool to run spark applications, it packages the spark code and runs on cluster.

The spark cluster can be standalone,local,K8s or YARN.

Master can run on yarn,local or k8s

deploy-mode -> specifies where driver runs

--class -> not required for python, just scala or java

--jars -> my sql connector jar files

spark.dynamicAllocation.enabled -> free's up some memory if we are not using it

We provide two arguments to main.py file.

We can provide syntax to generate log file.

The local system computer from where we run the command is called edge node.