Python Developer

I get the following error when I try to call I use python client for the spark. lines = sc.textFil… Read more Spark Java.lang.verifyerror

What is the best way to create/write/update a file in remote HDFS from local python script? I am a… Read more Python Write To Hdfs File

I'm trying to do a Spark tutorial that comes with the Cloudera Virtual Machine. But even though… Read more Exceptions When Reading Tutorial Csv File In The Cloudera Vm

I'm currently using Python 3 and would like to load a pickle file out of HDFS. from pywebhdfs.… Read more Unpickle File From Hdfs

I am running a Hadoop program and have the following as my input file, input.txt: 1 2 mapper.py: i… Read more Hadoop: Output File Has Double Output

I have installed, pip install thrift pip install PyHive pip install thrift-sasl and since pip ins… Read more Pyhive, Sqlalchemy Can Not Connect To Hadoop Sandbox

I'm on a W8 machine, where I use Python (Anaconda distribution) to connect to Impala in our Had… Read more Connect To Impala Using Impyla Client With Kerberos Auth

My map task completes successfully and I can see the application logs, but reducer stays in pending… Read more Hadoop-streaming : Reduce Task In Pending State Says "no Room For Reduce Task."

We have a hive warehouse, and wanted to use spark for various tasks (mainly classification). At tim… Read more Reading And Writing From Hive Tables With Spark After Aggregation

My question is similar to : hadoop streaming: how to see application logs? (The link in the answer … Read more Hadoop Streaming: Where Are Application Logs?

I wrote a python code to read and write to a hadoop file system with IP hdfs_ip. It takes 3 argumen… Read more Why Am I Getting These Strange Connection Errors When Reading Or Writing To Hadoop File System With A Python Script?

I have an existing Python program that has a sequence of operations that goes something like this: … Read more Using Hadoop To Run A Jar File - Python

I would like to populate a database of Postgresql by using a mapper with MrJob and Hadoop 2.7.1. I … Read more How To Populate A Postgresql Database With Mrjob And Hadoop

I have a hadoop cluster running on centos 6.5. I am currently using python 2.6. For unrelated reaso… Read more List All Files In Hdfs Python Without Pydoop

I'm trying to setup Hadoop on my EC2 instance using this tutorial. I'm trying to setup the … Read more Ambari Server Setup: Oserror: [errno 2] No Such File Or Directory

I followed this article to send some data to AWS ES, and I used the jar elasticsearch-hadoop. Here … Read more Write Data From Pyspark To Elasticsearch

Below is my spark data frame a b c 1 3 4 2 0 0 4 1 0 2 2 0 My output should be as below a b c 1 3 … Read more Pyspark Dynamic Column Computation

i am performing a hive query and storing the output in a tsv file in the local FS. I am running a f… Read more Remove Empty Line Printed From Hive Query Output Using Python