How To Classify Images Using Spark And Caffe

I am using Caffe to do image classification, can I am using MAC OS X, Pyhton. Right now I know how to classify a list of images using Caffe with Spark python, but if I want to make

Solution 1:

When you work with complex, non-native objects initialization has to moved directly to the workers for example with singleton module:

net_builder.py:

import cafe 

net = Nonedefbuild_net(*args, **kwargs):
     ...  # Initialize net herereturn net       

defget_net(*args, **kwargs):
    global net
    if net isNone:
        net = build_net(*args, **kwargs)
    return net

main.py:

import net_builder

sc.addPyFile("net_builder.py")

def classify_image(image_path, transformer, *args, **kwargs):
    net = net_builder.get_net(*args, **kwargs)

It means you'll have to distribute all required files as well. It can be done either manually or using SparkFiles mechanism.

On a side note you should take a look at the SparkNet package.