It sounds like a simple job, but with MapReduce it doesn't seem that straight-forward.
I have N files in which there is only one line of text for each file. I'd like the Mapper to output key value pairs like < filename, score >, in which 'score' is an integer calculated from the line of text. As a sidenote I am using the below snippet to do so (hope it's correct).
FileSplit fileSplit = (FileSplit)reporter.getInputSplit();
String fileName = fileSplit.getPath().getName();
Assuming the mapper does its job correctly, it should output N key value pairs.
Now the problem is how should I program the Reducer to output the one key value pair with the maximum 'score'?
From what I know Reducer only works with key value pairs that share the same key. Since the output in this scenario all have different keys, I am guessing something should be done before the Reduce step. Or perhaps should the Reduce step be omitted altogether?
Lets assume that
File1 has 10,123,23,233
File2 has 1,3,56,1234
File3 has 6,1,3435,678
Here is the approach for finding the maximum number from all the input files.
Lets first do some random sampling (like say every N records). From File1 123 and 10, from File2 56 and 1, from File3 1 and 678.
Pick the maximum number from the random sampling, which is 678.
Pass the maximum number from the random sampling to the mapper and ignore the input numbers less the maximum number found in the random sampling and emit the others in the mappers. Mappers will ignore anything less than 678 and emit 678, 1234 and 3435.
Configure the job to use 1 reducer and find the max of all the numbers sent to the reducer. In this scenario reducer will receive 678, 1234 and 3435. And will calculate the maximum number to be 3435.
Some observations of the above approach
The data has to be passed twice.
The data transferred between the mappers and reducers is decreased.
The data processed by the reducers also decreases.
Better the input sampling, faster the Job completes.
Combiner with similar functionality as the Reducer will further improve the Job time.
You can use the setup() and cleanup() methods (configure() and close() methods in old API).
Declare a global variable in reduce class, which determines the maximum score. For each call to reduce, you would compare the input value (score) with the global variable.
Setup() is called once before all reduce invocations in the same reduce task. Cleanup() is called after last reduce invocation in the same reduce task. So, if you have multiple reducers, Setup() and cleanup() methods would be called separately on each reduce task.
You can return the the filename and the score as the value and just return any constant as the key from your mapper
Refer slide 32 & 33 of http://www.slideshare.net/josem.alvarez/map-reduceintro
I used the same approach and got the result. Only concern is when you have multiple fields, you need to create fieldnamemin and fieldnamemax individually.
Omit the Reducer !!
Use the Configuration to set the global variable as score and key and then access it in the mapper to do a simple selection of max score by using the global variable as the memory of max score and key
It should be simple. I guess.
Related
I am relatively new to the hadoop world. I have been following examples I could find to understand how the record splitting step works for mapreduce jobs. I noticed that TextInputFormat splits file into records with key as the byte offset and value as a string. In this case, we could have two different records in a mapper having same offset from different input files.
Does it affect the mapper in any way? I think the uniqueness of the key to mapper is irrelevant if we do not process it (e.g. wordcount). But if we have to process it in mapper, the key may have to be unique. Can anyone elaborate on this ?
Thanks in advance.
Input to mapper is a file (or hdfs block) and not a key-value pair. In other words, mapper itself creates key-value pairs and does not get impacted by duplicate keys.
The "final" output generated by a Mapper is a multivalued hashmap.
< Key, <List of Values>>
This output becomes input to Reducer. All values of a key are processed by same reducer. It is ok for mappers to create more than one value for a key. Infact some solutions depend on this behavior.
Actually answer to your question totally depends on the scenario.
If you are not utilizing key(i.e. byte offset in case of textinputformat which is least utilized, but probably if you are using keyvalusepairInputformat you may be utilizing it.) then it never Impact, but if your map() function logic is such that you are doing some calculations on the basis of key then it will definitely impact.
So it totally depends on the scenario.
There is a misunderstanding. Actuallty,
For every input split of the file one Mapper will be assigned. All the
records from a single input split will be processed by only one Mapper
for a given job.
It is not required to bother about records with duplicate keys are arriving for mapper since mapper's execution scope is always at one key/value pair at any point of time.
The output from mapper task which is n number of key/value pairs are eventually merged, sorted and partitioned based on the keys.
The reducer will collect the required outputs from all mappers according to the partition and brings to the memory of reducer where it handles to arrange the key/value pairs as <key , Iterable <value> > .
I already installed hadoop mapreduce in one node and I have a top ten problem.
Let's say I have a 10k pair data (key,value) and search 10 data with the best value.
Actually, I create a simple project to iterate whole data and I need just a couple minute to got the answer.
then, I create mapreduce application with top ten design pattern to solve same problem, and I need more than 4 hour to get the answer. (obviously, I use the same machine and same algorithm to sort)
I think, that probably happens because mapreduce need more service to run, need more network activity, need more effort to read and write to hdfs.
Any other's factor to prove that mapreduce (in that condition) is slower than not using mapreduce?
mapreduce is slower on a single node setup because only one mapper and one reducer can work on it at any given time. mapper has to iterate through the each one of the splits and the reducer works on two mapper outputs simultaneously and then on two such reducer out puts ans so on..
so In terms of complexity:
for normal project :t(n) = n => O(n)
for mapreduce:t(n) = (n/x)*t(n/2x) => O((n/x)log(n/x)) where x is the number of nodes
which do you think is bigger? for single node and multinode..
explanation for mapreduce complexity:
time for one iteration: n
number of simultaneous map function: x since only one can work on each node
then time required for mapping complete data: n/x since n is the time 1 mapper takes for complete data
for reduce job half of the time is required as compared to the previous map since it works on two mapper outputs simultaneously therefore: time = n/2x for x reducers on x nodes
hence the equation that every next step will take half the time than the previous one.
t(n) = (n/x)*t(n/2x)
solving this recursion we get, O((n/x)log(n/x)).
this is not supposed to be exact but an approximation
What I need to do and am having some trouble doing is to have two values output for one key as the output to my reduce function. The reduce function receives data in the form of an Id and a list of integers associated with that Id. It needs to output that Id, the average of the integers in the list and the length of the list.
However, the implementation of the reduce function is supposed to have OutputCollector <Text, IntWritable> as an argument which clearly limits the number of outputs associated with each key to 1.
Any help in this regard would be greatly appreciated. Thanks in advance.
Hadoop version: 2.0.0
You have to use MultipleOutputs. In the Job:
MultipleOutputs.addMultiNamedOutput(job,
"Name",
SequenceFileOutputFormat.class, Text.class,
Writable.class);
In the reducer :
multipleOutputs.getCollector
Here are a few answers to your vague question.
You can call collect() as many times as you want for the same key if you don't mind each of the data (length and mean) appearing on its own record in a blended output. This could be accomplished by writing the key differently to distinguish the different records types, as follows:
oc.collect( new Text( k.toString() + " mean", mean );
oc.collect( new Text( k.toString() + " length", length );
OR
You should choose a different value type V3 instead of IntWritable. Either create a PairOfIntWritable or use an ArrayWritable to shove whatever you want into a single call to collect. Then the length and mean can be "fields" of the value for a single record for each key.
OR
If you absolutely have to use IntWritable, use an invertible pairing function to combine two integers into one. You'll need to insure that you can't exceed the maximum value of an IntWritable with any possible pair you could generate from your input data.
OR
Use MultipleOutputs to send one record each to a different file distinguished by name, so the part-r-nnnnn contain means and length-r-nnnnn contain lengths, for example. The JavaDoc on MultipleOutputs explains its use.
I am able to do few preprocessing steps in datamining using Hadoop MapReduce.
One such is normalization.
say
100,1:2:3
101,2:3:4
into
100 1
100 2
100 3
101 2
101 3
101 4
Like wise am i able to do binning for a numerical data say iris.csv.
I worked out the maths behind it
Iris DataSet: http://archive.ics.uci.edu/ml/datasets/Iris
find out the minimum and maximum values of each attribute
in the data set.
Sepal Length |Sepal Width| Petal Length| Petal Width
Min | 4.3| 2.0 | 1.0| 0.1
Max | 7.9 | 4.4 |6.9 | 2.5
Then, we should divide the data values of each attributes into ānā buckets .
Say, n=5.
Bucket Width= (Max - Min) /n
Sepal Length= (7.9-4.3)/5= 0.72
So, the intervals will be as follows :
4.3 - 5.02
5.02 - 5.74
Likewise,
5.74 -6.46
6.46 - 7.18
7.18- 7.9
continue for all attributes
Are we able to do the same in Mapreduce .
Please Suggest.
I am not sure if I understood your question, but what you want to do is to get the maximum and minimum for each of the attributes of that dataset, to then divide them, all in the same job, right? Ok, in order to divide the attributes, you need to feed the reducer with the max and min values instead of relying on the reducer to do the work for you. And I am guessing this is where your trouble starts.
However there is one thing you could do, a MapReduce design pattern called in-mapper combiner. When each mapper has finished its job, it calls a method called cleanup. You can implement the cleanup method so that it gets the max and min values of each of the attributes for each of the map nodes. This way, you give the reducer (only one reducer) only a collection with X values, being X the number of mappers in your cluster.
Then, the reducer gets the max and min values for each of the attributes, since it will be a very short collection so there won't be any problems. Finally, you divide each of the attributes into the 'n' buckets.
There is plenty of information about this pattern on the web, an example could be this tutorial. Hope it helps.
EDIT: you need to create an instance variable in the mapper where you will store each of the values in the map method, so that they will be available in the cleanup method, since it's only called once. A HashMap for example will do. You need to remember that you cannot save the values in the context variable in the map method, you need to do this in the cleanup method, after iterating through the HashMap and finding out the max and min value for each column. Then, as for the key, I don't think it really matters in this case, so yes, you could use the csv header, and as for the value you are correct, you need to store the whole column.
Once the reducer receives the output from the mappers, you can't calculate the buckets just yet. Bear in mind that you will receive one "column" for each mapper, so if you have 20 mappers, you will receive 20 max values and 20 min values for each attribute. Therefore you need to calculate the max and min again, just like you did in the cleanup method of the mappers, and once this is done, then you can finally calculate the buckets.
You may be wondering "if I still need to find the max and min values in the reducer, then I could omit the cleanup method and do everything in the reducer, after all the code would be more or less the same". However, to do what you are asking, you can only work with one reducer, so if you omit the cleanup method and leave all the work to the reducer, the throughput would be the same as if working in one machine without Hadoop.
I am reading about MapReduce and the following thing is confusing me.
Suppose we have a file with 1 million entries(integers) and we want to sort them using MapReduce. The way i understood to go about it is as follows:
Write a mapper function that sorts integers. So the framework will divide the input file into multiple chunks and would give them to different mappers. Each mapper will sort their chunk of data independent of each other. Once all the mappers are done, we will pass each of their results to Reducer and it will combine the result and give me the final output.
My doubt is, if we have one reducer, then how does it leverage the distributed framework, if, eventually, we have to combine the result at one place?. The problem drills down to merging 1 million entries at one place. Is that so or am i missing something?
Thanks,
Chander
Check out merge-sort.
It turns out that sorting partially sorted lists is much more efficient in terms of operations and memory consumption than sorting the complete list.
If the reducer gets 4 sorted lists it only needs to look for the smallest element of the 4 lists and pick that one. If the number of lists is constant this reducing is an O(N) operation.
Also typically the reducers are also "distributed" in something like a tree, so the work can be parrallelized too.
As others have mentioned, merging is much simpler than sorting, so there's a big win there.
However, doing an O(N) serial operation on a giant dataset can be prohibitive, too. As you correctly point out, it's better to find a way to do the merge in parallel, as well.
One way to do this is to replace the partitioning function from the random partitioner (which is what's normally used) to something a bit smarter. What Pig does for this, for example, is sample your dataset to come up with a rough approximation of the distribution of your values, and then assign ranges of values to different reducers. Reducer 0 gets all elements < 1000, reducer 1 gets all elements >= 1000 and < 5000, and so on. Then you can do the merge in parallel, and the end result is sorted as you know the number of each reducer task.
So the simplest way to sort using map-reduce (though the not the most efficient one) is to do the following
During the Map Phase
(Input_Key, Input_Value) emit out (Input_Value,Input Key)
Reducer is an Identity Reduceer
So for example if our data is a student, age database then your mapper input would be
('A', 1) ('B',2) ('C', 10) ... and the output would be
(1, A) (2, B) (10, C)
Haven't tried this logic out but it is step in a homework problem I am working on. Will put an update source code/ logic link.
Sorry for being late but for future readers, yes, Chander, you are missing something.
Logic is that Reducer can handle shuffled and then sorted data of its node only on which it is running. I mean reducer that run at one node can't look at other node's data, it applies the reduce algorithm on its data only. So merging procedure of merge sort can't be applied.
So for big data we use TeraSort, which is nothing but identity mapper and reducer with custom partitioner. You can read more about it here Hadoop's implementation for TeraSort. It states:
"TeraSort is a standard map/reduce sort, except for a custom partitioner that uses a sorted list of N ā 1 sampled keys that define the key range for each reduce. In particular, all keys such that sample[i ā 1] <= key < sample[i] are sent to reduce i. This guarantees that the output of reduce i are all less than the output of reduce i+1."
I think, combining multiple sorted items is efficient than combining multiple unsorted items. So mappers do the task of sorting chunks and reducer merges them. Had mappers not done sorting, reducer will have tough time doing sorting.
Sorting can be efficiently implemented using MapReduce. But you seem to be thinking about implementing merge-sort using mapreduce to achieve this purpose. It may not be the ideal candidate.
Like you alluded to, the mergesort (with map-reduce) would involve following steps:
Partition the elements into small groups and assign each group to the mappers in round robin manner
Each mapper will sort the subset and return {K, {subset}}, where K is same for all the mappers
Since same K is used across all mappers, only one reduce and hence only one reducer. The reducer can merge the data and return the sorted result
The problem here is that, like you mentioned, there can be only one reducer which precludes the parallelism during reduction phase. Like it was mentioned in other replies, mapreduce specific implementations like terasort can be considered for this purpose.
Found the explanation at http://www.chinacloud.cn/upload/2014-01/14010410467139.pdf
Coming back to merge-sort, this would be feasible if the hadoop (or equivalent) tool provides hierarchy of reducers where output of one level of reducers goes to the next level of reducers or loop it back to the same set of reducers