ls --color | grep doesn't work

Question

I'm trying to list all hidden dirs with the following command

ls -lhAF1 | grep -E '^d.*[0-9]{2}:[0-9]{2} \.'

which works perfectly fine

Explanation for Regex: I'm trying to get all rows that have the following Format: d, then some text, then the timestamp, then a space, then the dot and then more text

However when I try to color the ls output with this command:

ls --color -lhAF1 | grep -E '^d.*[0-9]{2}:[0-9]{2} \.'

It gives zero results, the output without --color is:

drwxr-xr-x 1 User Group 4096 Feb 1 08:48 .invisible

Why does ls / grep behave this way?

Answer 1

--color adds escape sequences for the color. You can see this if you redirect the output (of ls --color) to a file.

This is what it looks like:

 drwxr-xr-x  6 root root 4.0K Jan  9 08:23 ^[[01;34m.cabal^[[0m/

To account for this, try this instead:

 ls -lhAF1 --color | grep -E '^d.*[0-9]{2}:[0-9]{2} .*\.'

Answer 2

Parsing ls is often a bad idea. Often, but not always. Here's another suggestion for you, which collects the required directories together before passing the set to ls.

find .* -maxdepth 0 -type d \( -name '.[^.]' -o -name '.??*' \) -exec ls -ld --color=always {} +

It's been pointed out that the original code actually limits the list of directories to those modified in the last six months. This can be handled with the following alternative solution.

find .* -maxdepth 0 -type d -mtime -180 -mtime +0 \( -name '.[^.]' -o -name '.??*' \) -exec ls -ld --color=always {} +

_{As ever, if your find doesn't understand the trailing + replace it with \; at the cost of efficiency.}

Answer 3

The reason is that ls always colorizes its output even if it is connected to a terminal. From man ls:

   --color[=WHEN]
          colorize the output.   WHEN  defaults  to  'always'  or  can  be
          'never' or 'auto'.  More info below

Many other tools such as grep do not retain colors when standard output is terminal but for some reasons ls was designed to act differently. Colorizing output is achieved by using ANSI escape codes that are interpreted by your terminal. Redirect ls output to a file and open it in editor:

$ ls --color  -lhaF1  > /tmp/RESULT
$ less /tmp/RESULT
total 12K
drwxr-xr-x  3 ja   users 4.0K Feb  2 09:47 ESC[0mESC[01;34m.ESC[0m/
drwxrwxrwt 12 root root  4.0K Feb  2 09:51 ESC[30;42m..ESC[0m/
drwxr-xr-x  2 ja   users 4.0K Feb  2 09:47 ESC[01;34m.invisibleESC[0m/
-rw-r--r--  1 ja   users    0 Feb  2 08:35 a|a

If you have $LESS variable set you probably need to unset it before running less in order to see raw escape codes instead of color. So, to sum up, when using --color what grep gets is not what you see but a bunch of escape codes together with it. To fix it either don't use --colors at all or set it to --auto:

$ ls --color=auto -lhaF1 | grep  -E '^d.*[0-9]{2}:[0-9]{2} \.'

Answer 4

Ok I finally figured out how to list all hidden dirs while preserving the Colors and not including dirs like "hello.world":

 ls -lhAF1 --color | grep -E "^d[rwx-]{9}.*[0-9]{2}:[0-9]{2} "$'\x1b'"\[([0-9]{1,2}m?)(;[0-9]{1,2}m?)?\."

Answer 5

Rather than applying grep to the output of ls which is not post-processable reliably anyway, I'd rather filter on the list of files passes to ls, like:

ls -lFhd --color -- .*(/)

In zsh. Or as your own approach suggests you only want the directories last modified between 6 months ago and now and not the ones with special permissions:

ls -lFhd --color -- .*(/mM-6mM+0f-7000)