Questions tagged [uniq]
The uniq tag has no summary.
178 questions
0
votes
2
answers
80
views
pipe to uniq from a variable not showing the desired output
I have a pipeline using array jobs and need to change the number of inputs for some steps. I thought about testing uniq since the only part changing in my folders are the last four characters (the hap ...
- 283
1
vote
1
answer
115
views
How can I find duplicate lines among files?
I have a software module which contains some files with same pattern.
<whitespaces>private static final long serialVersionUID = \dL;
How can I find files with the same value?
$ grep -R ...
- 564
1
vote
1
answer
191
views
Why is sorted uniq -c command showing duplicates
I am trying to count how many times I use a certain version of a library on my computer.
For some reason, uniq -c is outputing duplicates, despite sorting it, and despite the sort order seeming in ...
- 171
-1
votes
2
answers
197
views
how to de-duplicate block (timestamp+command) from bash history?
I'm working with bash_history file containing blocks with the following format: #unixtimestamp\ncommand\n
here's sample of the bash_history file:
#1713308636
cat > ./initramfs/init << "...
- 33
0
votes
3
answers
89
views
Does a command exist that lists all the directories where a word appears in a file or directory name?
When I don't remember where a file or a folder is, I sometime use the locate command
(that finds more occurrences, allow more candidates than a find, in my mind. But maybe I'm mistaking).
But then ...
- 2,385
11
votes
1
answer
1k
views
Use uniq to filter adjacent lines in pipeline
I'm trying to monitor theme changes using this command:
dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" | grep -o "uint32 ."
Output ...
- 235
1
vote
5
answers
502
views
Find and delete partially duplicate lines
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/...
0
votes
0
answers
26
views
numeric sort with unique option does not show 0! [duplicate]
I have a file with many redundant numbers in each row. Imagine something like the below:
echo "10
9
5
6
4
cell
3
2
0
7
0
1" > test
When I use sort -un test I get the following output:
...
- 131
0
votes
2
answers
333
views
tar processing files multiple times with find -newer
I'm trying to use tar(1) to create an archive of files newer than a specific file (fileA). However, when I use find(1) to obtain the list of files to pass to tar, some files are listed multiple times:
...
- 1,639
4
votes
1
answer
11k
views
Difference between sort -u and uniq -u
I always have been using sort -u to get rid of duplicates until now.
But I am having a real doubt about a list generated by a software tool.
The question is: is the output of sort -u |wc the same as ...
- 248
0
votes
3
answers
97
views
de-duplicate list but group parts of it
I am compiling some access rules from failed logins and after some piping I arrived at this:
cat <<INPUT | sort -k 3,3 --unique
Deny from 13.42.98.142 # demo
Deny from 13.42.98.142 # test
Deny ...
- 513
14
votes
2
answers
4k
views
Who killed my sort? or How to efficient count distinct values from a csv column
I'm doing some processing trying to get how many different lines in a file containing 160,353,104 lines. Here is my pipeline and stderr output.
$ tail -n+2 2022_place_canvas_history.csv | cut -d, -f2 |...
- 213
3
votes
5
answers
885
views
find duplicate 1st field and concat its values in single line
I have a file that has entries in key: value format like the below:
cat data.txt
name: 'tom'
tom_age: '31'
status_tom_mar: 'yes'
school: 'anne'
fd_year_anne: '1987'
name: 'hmz'
hmz_age: '21'
...
- 527
-1
votes
1
answer
124
views
Skip line from console if equal than line before, adding count (in realtime)
Using uniq it is possible to filter out sequential duplicate lines.
while (true) do echo 1; echo 2; echo 2; echo 1; sleep 1; done | uniq
becomes:
1
2
1
Is there a way to have duplicated sequential ...
- 101
3
votes
2
answers
864
views
Output lines with the same md5 sum
I have a script like this
find path -type f -exec md5sum {} +'
It has this conclusion
/tmp
❯ find $pwd -type f -exec md5sum {} +
\a7c8252355166214d1f6cd47db917226 ./guess.bash
...
- 51