# Setup local Spark Cluster on Mac and explore UI for Master, Workers, and Jobs(Python using Jupyter notebook) ## What is covered - In this blog, I would describe my experiments about setting up a local (standalone) Spark cluster on Mac M1 machine. - Would start jupyter notebook using anaconda navigator - Submit jobs using Python code executing in the notebook with jobs submitted to the master created above. - Explore the UI for: - Master - Worker - Jobs (transformation and action operations, Directed Acyclic Graph) ## Why local (standalone) server - local server is a simple deployment model and is possible to run the daemons in a single node - Jobs submitted using pyspark are explored using the same UI in previous [blog](https://sriram.uscconsulting.net/setup-local-spark-cluster-on-mac-and-explore-ui-for-master-workers-and-jobspyspark) - Jobs submitted using python programs are explored in this blog ## Start the master and worker - After installation of pyspark (using homebrew), scripts for starting the master and worker were available at the following location: ``` cd /opt/homebrew/Cellar/apache-spark/3.2.1/libexec/sbin ``` - start the server (note that would be specific to the machine) ``` % ./start-master.sh starting org.apache.spark.deploy.master.Master, logging to /opt/homebrew/Cellar/apache-spark/3.2.1/libexec/logs/.out ``` - check the log file for the spark master server details ( is obtained from the previous step> ``` % tail -10 /opt/homebrew/Cellar/apache-spark/3.2.1/libexec/logs/.out ... 22/05/18 07:41:57 INFO Utils: Successfully started service 'sparkMaster' on port 7077. 22/05/18 07:41:57 INFO Master: Starting Spark master at spark://:7077 22/05/18 07:41:57 INFO Master: Running Spark version 3.2.1 22/05/18 07:41:57 INFO Utils: Successfully started service 'MasterUI' on port 8080. ... ``` - check the master UI - notice that workers are zero, since, we have not yet started any ![Screen Shot 2022-05-18 at 7.44.38 AM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1652928152348/AT3DpaTFB.png align="left") - start the worker by selecting the number of cores and memory based on your system configuration. - please check logfile in the previous step and replace with the appropriate value for ``` % ./start-worker.sh --cores 2 --memory 2G spark://:7077 starting org.apache.spark.deploy.worker.Worker, logging to /opt/homebrew/Cellar/apache-spark/3.2.1/libexec/logs/.out ``` - check the master UI for worker information (now it shows 1 worker) ![Screen Shot 2022-05-18 at 7.48.16 AM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1652929733598/8oJu7OnuF.png align="left") - also, check the worker UI for the cores and memory allocated ![Screen Shot 2022-05-18 at 7.50.01 AM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1652929776378/dDrPz_k_4.png align="left") ## submit jobs using python in jupyter notebook and observe jobs UI - started jupyter notebook using anaconda navigator - write python code to submit jobs to the master setup above - operations are performed on RDDs (Resilient Distributed Dataset) - spark has two kinds of operations: - Transformation → operations such as map, filter, join or union that are performed on an RDD that yields a new RDD containing the result - Action → operations such as reduce, first, count that return a value after running a computation on an RDD - Actions could be displayed as jobs in the UI and transformations could be observed while exploring the DAG (Directed Acyclyic Graph) output - started jupyter notebook from anaconda navigator ![Screen Shot 2022-05-27 at 10.51.46 PM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1653711266488/8tYul4U4j.png align="left") - create Spark Application in notebook (replace with your-master) ``` import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder.master("") \ .appName('SparkTestPython') \ .getOrCreate() print("First SparkContext:"); print("APP Name :"+spark.sparkContext.appName); print("Master :"+spark.sparkContext.master); ``` ![Screen Shot 2022-05-27 at 11.50.44 PM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1653715011121/P_8VNOxdz.png align="left") - Now observe the master and worker that the Application is listed ![Screen Shot 2022-05-27 at 11.51.11 PM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1653714991092/OVPFDlRWs.png align="left") ![Screen Shot 2022-05-27 at 11.55.46 PM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1653715259407/6ESG1QCZa.png align="left") - we will write a simple code in the notebook to do one transformation (parallelize) and one action (count) operation ``` a=("hello","world","jupyter","local","mac","m1") b=spark.sparkContext.parallelize(a) b.count() ``` ![Screen Shot 2022-05-27 at 11.54.09 PM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1653715206100/QjwxebdGV.png align="left") - From the master, click on the application to check the application UI: - In the application UI, click on "Application Detail UI" to get details on the jobs ![Screen Shot 2022-05-27 at 11.54.59 PM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1653715351272/7sKdMIYs7.png align="left") - to see DAG (Directed Acyclic Graph) visualization click on the job (under the Description column) - this is a simple DAG with only one transformation and one operation ![Screen Shot 2022-05-27 at 11.56.52 PM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1653715375168/PhEAOd0ru.png align="left") - we will write another simple code in the notebook to do two transformations (read from file and map) and one action (collect) operation ``` x=spark.sparkContext.textFile('/tmp/sparktest.txt') y=x.map(lambda n: n.upper()) y.collect() ``` ![Screen Shot 2022-05-28 at 12.04.02 AM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1653715597406/YxzKtZMWy.png align="left") - Check the jobs UI ![Screen Shot 2022-05-28 at 12.06.04 AM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1653715658555/9dwVLZ83w.png align="left") ## Next steps - try out additional spark operations, and actions, and check additional details like storage, etc. in the UI - using data frames ## References - [Spark Standalone Server](https://spark.apache.org/docs/latest/spark-standalone.html) - [Anaconda Distribution](https://www.anaconda.com/products/distribution)