Retrieving S3 Path From Payload Inside Aws Glue Pythonshell Job

I have a pythonshell job inside AWS glue that needs to download a file from a s3 path. This s3 path location is a variable so will come to the glue job as a payload in start_run_jo

Solution 1:

Check the docimentation. All you need is here.

You can use the getResolvedOptions as follows:

import sys
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv,
                          ['JOB_NAME',
                           'day_partition_key',
                           'hour_partition_key',
                           'day_partition_value',
                           'hour_partition_value'])
print"The day partition key is: ", args['day_partition_key']
print"and the day partition value is: ", args['day_partition_value']