Uniq command Linux Filter adjacent matching lines from INPUT

Uniq Command Linux is used to report or omit repeated lines. Uniq is useful and handy tool to remove duplicate data from N number of lines of file. Its not only used to remove duplicate data it also helps in shell scripting to make reports.

Options Uniq Command Linux

  • -c,  Option will count number of occurrences
  • -d,  Only print duplicate lines, one for each group
  • -D, Print all duplicate lines groups can be delimited with an empty line METHOD={none(default),prepend,separate}
  • -f,  Avoid comparing the first N fields
  • -i, Ignore differences in case when comparing
  • -s, Avoid comparing the first N characters
  • -u, Only print unique lines
  • -z, End lines with 0 byte, not newline
  • -w, Compare no more than N characters in lines

Normal Uniq Command without options

[root@localhost Linux_Commands]# uniq testingdata
aaaaa
bbbbb
ccccc
ddddd
eeeee eeeee
Test File data used for this article explanation
[root@localhost Linux_Commands]# cat testingdata
aaaaa
aaaaa
aaaaa
aaaaa
aaaaa
aaaaa
bbbbb
ccccc
ccccc
ccccc
ccccc
ddddd
eeeee eeeee
eeeee eeeee
eeeee eeeee
eeeee eeeee

Uniq command without any options will print uniq lines

Count Recurrences using Uniq

If you have an data with multiple duplicates, you can even count the recurrences using Uniq command below is the best example

[root@localhost Linux_Commands]# uniq -c testingdata
6 aaaaa
1 bbbbb
4 ccccc
1 ddddd
4 eeeee eeeee
[root@localhost Linux_Commands]# uniq -c testingdata |sort -nr
6 aaaaa
4 eeeee eeeee
4 ccccc
1 ddddd
1 bbbbb

Print Only each line which has duplicates

Only print duplicate lines, one for each group

[root@localhost Linux_Commands]# uniq -d testingdata
aaaaa
ccccc
eeeee eeeee

Duplicate data group

Using Uniq command print group of duplicate lines using -D

[root@localhost Linux_Commands]# uniq --all-repeated=separate testingdata
aaaaa
aaaaa
aaaaa
aaaaa
aaaaa
aaaaa
ccccc
ccccc
ccccc
ccccc
eeeee eeeee
eeeee eeeee
eeeee eeeee
eeeee eeeee
[root@localhost Linux_Commands]# uniq -D testingdata
aaaaa
aaaaa
aaaaa
aaaaa
aaaaa
aaaaa
ccccc
ccccc
ccccc
ccccc
eeeee eeeee
eeeee eeeee
eeeee eeeee
eeeee eeeee

Uniq Lines Printing out of all

Simple method to separate Uniq lines (which lines does not have any duplicates) using -u option

[root@localhost Linux_Commands]# uniq -u testingdata
bbbbb
ddddd

N Characters in lines

Compare no more than N characters specified using -w option

[root@localhost Linux_Commands]# uniq -c -w 2 testingdata
6 aaaaa
1 bbbbb
4 ccccc
1 ddddd
4 eeeee eeeee

Avoid Comparing N Number of fields

-f option eliminate comparing N number of fileds

[root@localhost Linux_Commands]# uniq -f1 testingdata
aaaaa
eeeee eeeee
[root@localhost Linux_Commands]# uniq -f2 testingdata
aaaaa

Uniq Command Linux Conclusion : 

Linux Command is used to find Uniq values out of number of duplicate lines. Count duplicate occurrences and print only files by specified numer.

Related Articles

25 Linux basic commands

sed command 20 practical examples

bc command in linux examples

netstat command Linux

Thanks for your wonderful Support and Encouragement

ARK

My Name is ARK. Expert in grasping any new technology, Interested in Sharing the knowledge. Learn more & Earn More

You may also like...

Leave a Reply