Creating a custom class of type WritableComparator

Question

Creating a custom class of type WritableComparator

Navigation

#1 by (1 votes)
#2 by (0 votes)

0

I'm creating an application to process video in a distributed way in hadoop using the map-reduce paradigm. I have several videos stored in the cluster and I wish from them to create a single video file.

To do this, I'm extracting the frames of each video in the map phase and sending them to the phase stage. Each video is numbered from 1 to 9, and the beginning of the key of each frame corresponds to the video that originated it, for example the key 11788847 corresponds to a frame of the video 1, on the other hand the key 57845 corresponds to the video of number 5. Note that even the 11788847 key being larger than the 57845 key, it must come before the frames are grouped according to the video.

For this, I've implemented the SortingCustomComparator class that inherits from WiritableComparator . Here is the code below:

public class SortingCustomComparator extends WritableComparator{

protected SortingCustomComparator(){
    super(Text.class, true);
}
@Override
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2){
    // Converte a string em um inteiro
    String key1 = null;
    String key2 = null;
    try {
        key1 = Text.decode(b1, s1, l1);
        key2 = Text.decode(b2, s2, l2);
    } catch (CharacterCodingException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }


    int keyInt = Integer.valueOf(key1.trim());
    int key2Int = Integer.valueOf(key2.trim());
    return (keyInt >key2Int) ? 1 : -1;

}  }

The problem is that the above code does not have the expected behavior. The created video contains some frames that are placed in a random position. I would like the keys to be grouped by blocks. The keys to the first video must be sorted in ascending order and must belong to the set of the first video as well as the second belonging to the second and so on. Example: {11, 12, 13, 14, ..., 111, 112}, {2, 21, 22, ...}. At the end these key blocks should be ordered {11, 12, 13, 14, ..., 2, 21, 22}. So the frames will be grouped in the correct order, the video will not be buggy. Would anyone know how to implement this correctly?

Thank you in advance.

java hadoop

asked by anonymous 07.08.2017 / 22:13

2 answers

1

Try changing this:

int keyInt = Integer.valueOf(key1.trim());
int key2Int = Integer.valueOf(key2.trim());
return (keyInt >key2Int) ? 1 : -1;

So:

return key1.compareTo(key2);

07.08.2017 / 23:11

size an image inside a button Angular Service using third party packages

score 0 · Accepted Answer

I solved the problem as follows:

1 - Each video starts with a number that matches your source video; 2 - The source number is concatenated with the frame number that is generated in the application; 3 - I created a custom class that compares according to the code below:

public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2){
    /*
     * Converte a string em um inteiro
     */ 
    String key1 = null;
    String key2 = null;
    try {
        key1 = Text.decode(b1, s1, l1);
        key2 = Text.decode(b2, s2, l2);
    } catch (CharacterCodingException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }


    int keyInt = Integer.valueOf(key1.trim());
    int key2Int = Integer.valueOf(key2.trim());

    String strKey1 = Integer.toString(keyInt);
    String strKey2 = Integer.toString(key2Int);

    int keyFirstP1 = Integer.parseInt(strKey1.substring(0, 1));
    int key2FirstP1 = Integer.parseInt(strKey2.substring(0, 1));

    strKey1 = modifieString(strKey1);
    strKey2 = modifieString(strKey2);


    //return cmp.compare(strKey1, strKey2);

    if(keyFirstP1 == key2FirstP1){
            return (keyInt > key2Int) ? 1 : -1; 
    }

    if(keyFirstP1 != key2FirstP1){
         return (keyFirstP1 > key2FirstP1) ? 1 : -1;
    }
    return 0;

First we compare the first element of each of the keys. If they are equal the key is compared in full, otherwise we only return the comparison between the first element of each key, thus ensuring the correct order of the frames.