Spark Installation And Configuration On Macos Importerror: No Module Named Pyspark

I'm trying to configure apache-spark on MacOS. All the online guides ask to either download the spark tar and set up some env variables or to use brew install apache-spark and then

Solution 1:

pyspark module is not include in your python

Try this instead

import os
import sys

os.environ['SPARK_HOME'] = "/usr/local/Cellar/apache-spark/2.1.0/libexec/"

sys.path.append("/usr/local/Cellar/apache-spark/2.1.0/libexec/python")
sys.path.append("/usr/local/Cellar/apache-spark/2.1.0/libexec/python/lib/py4j-0.10.4-src.zip")

try:
    from pyspark import SparkContext
    from pyspark import SparkConf

except ImportError as e:
    print ("error importing spark modules", e)
    sys.exit(1)

sc = SparkContext('local[*]','PySpark')

---

if you don't want that, include them into your system PATH. And don't forget to include the python path.

export SPARK_HOME=/usr/local/Cellar/apache-spark/2.1.0/libexec/
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$PYTHONPATHexport PATH=$SPARK_HOME/python:$PATH

Solution 2:

sorry I dont use MAC , but there is another way in linux beside above answer:

sudo ln -s $SPARK_HOME/python/pyspark /usr/local/lib/python2.7/site-packages

Python will read module from /path/to/your/python/site-packages at last