Use of Uniq

Navigation
1

I have been researching without much success, something that I believe in theory to be very simple, but I did not find the right command.

I have a LOG file with several information, but certain information is repeated, but only in a certain column, and everything that is repeated in this column, I wanted it to be deleted, with only one remaining. Example: 

  6; Mar 21 03:18; 182.69.170.145;  unknown;  <[email protected]>;  Get much more positive aspects out of your work out; HIGH
  3; Mar 21 03:20; 182.69.170.145;  unknown;  <[email protected]>;  Eating healthful is not assisting you lose weight; HIGH
  2; Mar 21 03:18; 182.69.170.145;  unknown;  <[email protected]>;  Asian infused diet program pill makes it was West; MEDIUM
  2; Mar 21 13:50; 201.53.117.127;  unknown;  <[email protected]>;  want to see me?; MEDIUM
  3; Mar 21 12:28; 179.208.77.183;  unknown;  <[email protected]>;  how do you like it here?; HIGH
  3; Mar 21 13:49; 201.53.117.127;  unknown;  <[email protected]>;  Good Evening How are things? I m Yana; HIGH

Is there any command, or even these (SORT and UNIQ) with some specific parameter that does this?

Thank you.

    
asked by anonymous 24.03.2017 / 23:27

2 answers

1

You can use this command: 

sort -u -t ';' -k5,5 nome-do-arquivo

  • -u (unique) which makes equal values grouped.

  • -t ';' Set the column separator (which in the case of your file is ; ).

  • -k5,5 Set the number of the column you want to work on (in your case 5, which is email, and only it).

You can read more about the command sort here .

    
27.03.2017 / 20:37
1

However, if it is important to maintain the original file order, You can: 

awk -F';' '++n[$5] == 1' nome
  • -F ';' - defines the FieldSeparator (field separator)
  • n[$5] - counts the number of occurrences of each value in field 5 (email); The n vector has string type indexes (associative array)
  • ++n[$5] - increments the value corresponding to the specific email
  • ++n[$5] == 1 - first occurrence of this email (action by default: print )
27.03.2017 / 23:06
Reading CSV with Python Field 'Hash' does not have a default value