Hadoop Adding multiple inputs with a different mapper using python

0 votes
I want to add multiple inputs with a different mapper for each using python script. I have used Java for this but I dont know how to do it in python. Can someone tell me how to do this?
Nov 19, 2018 in Big Data Hadoop by digger
• 26,740 points
2,552 views

1 answer to this question.

0 votes
hadoop jar hadoop-multiple-streaming.jar \  
  -input    myInputDirs \  
  -multiple "outputDir1|mypackage.Mapper1|mypackage.Reducer1" \  
  -multiple "outputDir2|mapper2.sh|reducer2.sh" \  
  -multiple "outputDir3|mapper3.py|reducer3.py" \  
  -multiple "outputDir4|/bin/cat|/bin/wc" \  
  -libjars  "libDir/mypackage.jar" \
  -file     "libDir/mapper2.sh" \  
  -file     "libDir/mapper3.py" \  
  -file     "libDir/reducer2.sh" \  
  -file     "libDir/reducer3.py"

Source: https://github.com/hyonaldo/hadoop-multiple-streaming

answered Nov 19, 2018 by Omkar
• 69,220 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Is there any way to include a python package with Hadoop streaming job?

See I don't know the answer to ...READ MORE

answered May 10, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
2,188 views
0 votes
1 answer

Error running hadoop mapreduce in Python using Hadoop Streaming

Hi As you write mapper and reducer program  ...READ MORE

answered Jan 21, 2020 in Big Data Hadoop by anonymous
2,539 views
0 votes
1 answer

How to get started with Hadoop and do some development using Eclipse IDE?

Alright, there are couple of things that ...READ MORE

answered Apr 4, 2018 in Big Data Hadoop by Ashish
• 2,650 points
2,116 views
0 votes
1 answer

Different ports in a Hadoop cluster environment?

Below image will help you in understanding ...READ MORE

answered Apr 6, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
1,866 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
11,015 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,528 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
108,740 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,350 points
4,605 views
0 votes
1 answer

Hadoop Hive: How to split a single row into multiple rows?

Try this SELECT ID1, Sub FROM tableName lateral view ...READ MORE

answered Nov 14, 2018 in Big Data Hadoop by Omkar
• 69,220 points
9,340 views
0 votes
1 answer

Hadoop with Python: How to use python to stop Hadoop tasktracker?

Here you go os.system('sudo -u hadoop bin/hadoop-daemon.sh stop ...READ MORE

answered Nov 16, 2018 in Big Data Hadoop by Omkar
• 69,220 points
615 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP