Ask Your Question
1

Problem with uniq command

asked 2018-03-09 00:57:43 -0600

updated 2018-03-09 00:58:15 -0600

Say I have a file name "reading.txt" with lots of duplicates:

book

book

paper

paper

article

article

magazine

magazine

I want to remove the duplicates, so I typed this

"uniq reading.txt"

The output should come like this

book

paper

article

magazine

But it shows all elements of reading.txt file like this

[z@localhost Documents]$ cat reading.txt book

book

paper

paper

article

article

magazine

magazine [z@localhost Documents]$ uniq reading.txt book

book

paper

paper

article

article

magazine

magazine [z@localhost Documents]$

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
2

answered 2018-03-09 01:52:47 -0600

sideburns gravatar image

That's interesting, and I was able to duplicate it. Checking the man page, I found this:

Note: 'uniq' does not detect repeated lines unless they are adjacent.

I edited the text file to remove the blank lines, and found that the command worked as expected. I'm not sure how much help this will be, but at least it explains what's been happening.

edit flag offensive delete link more

Comments

Try sort -u

villykruse gravatar imagevillykruse ( 2018-03-09 01:57:42 -0600 )edit
1

You should have made this an answer. You have to sort your input before using uniq in most cases.

ssieb gravatar imagessieb ( 2018-03-09 02:13:54 -0600 )edit

sort -u reading.txt | uniq should deal with empty lines (unless they've been only added here, because of messed up askbot's behaviour - ignoring singular new lines).

ozeszty gravatar imageozeszty ( 2018-03-09 02:14:42 -0600 )edit

I see "uniq" command works fine without any blank lines. Thank you all.

montasimmamun gravatar imagemontasimmamun ( 2018-03-09 02:49:14 -0600 )edit
1

answered 2018-03-09 04:19:13 -0600

Infoweb gravatar image

uniq removes only duplicated lines that follow each other. You can use the sort -u command, it will remove duplicates, but it will sort the strings in alphabetical order. If you need to preserve the original order of strings, try this command:

awk '!($0 in dup) { dup[$0] = ""; print $0 }' reading.txt
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2018-03-09 00:57:43 -0600

Seen: 79 times

Last updated: Mar 09 '18