Rename file sets with multiple (incl. compounded) extensions in shell

Question

I have a list of file sets with a lot of extensions, but unique names.

filename-1.foo.001
...
filename-1.foo.020
filename-1.foo.baz
filename-1.foo.bar1-2.baz
...
filename-1.foo.bar7-8.baz

another_filename.foo.001
another_filename.foo.002
...
another_filename.foo.009
another_filename.foo.baz
another_filename.foo.bar1-2.baz
another_filename.foo.bar3-4.baz
another_filename.foo.bar4-5.baz
another_filename.foo.bar7-8.baz

yet.a.different.file.name.foo.001
yet.a.different.file.name.foo.002
...
yet.a.different.file.name.foo.287
yet.a.different.file.name.foo.baz
yet.a.different.file.name.foo.bar1-2.baz
yet.a.different.file.name.foo.bar3-4.baz
yet.a.different.file.name.foo.bar4-5.baz
yet.a.different.file.name.foo.bar7-8.baz

moreFileNaming.foo.001
...
moreFileNaming.foo.009
moreFileNaming.foo.baz
moreFileNaming.foo.bar1-2.baz
moreFileNaming.foo.bar3-4.baz
moreFileNaming.foo.bar4-5.baz
moreFileNaming.foo.bar7-8.baz

I would like to rename these using the output of openssl rand -hex 8 to get a random filename for each set like this:

9874f7187c914ea9.foo.001
...
9874f7187c914ea9.foo.020
9874f7187c914ea9.foo.baz
9874f7187c914ea9.foo.bar1-2.baz
...
9874f7187c914ea9.foo.bar7-8.baz

2f54a0b6528e3927.foo.001
2f54a0b6528e3927.foo.002
...
2f54a0b6528e3927.foo.009
2f54a0b6528e3927.foo.baz
2f54a0b6528e3927.foo.bar1-2.baz
2f54a0b6528e3927.foo.bar3-4.baz
2f54a0b6528e3927.foo.bar4-5.baz
2f54a0b6528e3927.foo.bar7-8.baz

71ad0aa90148b2f5.foo.001
71ad0aa90148b2f5.foo.002
...
71ad0aa90148b2f5.foo.287
71ad0aa90148b2f5.foo.baz
71ad0aa90148b2f5.foo.bar1-2.baz
71ad0aa90148b2f5.foo.bar3-4.baz
71ad0aa90148b2f5.foo.bar4-5.baz
71ad0aa90148b2f5.foo.bar7-8.baz

3721323156e921b5.foo.001
...
3721323156e921b5.foo.009
3721323156e921b5.foo.baz
3721323156e921b5.foo.bar1-2.baz
3721323156e921b5.foo.bar3-4.baz
3721323156e921b5.foo.bar4-5.baz
3721323156e921b5.foo.bar7-8.baz

I have tried for name (*.(<->|baz|bar<->.baz) result=$(openssl rand -hex 16) && mv $name $result (this may be nonfunctioning as it was several iterations ago) but when it worked it would give every single file a random name, I just want each set to remain with the same name, just random and with the same size. Sha1sum or any other method would be fine too.

How do I accomplish this? Particularly for the files .foo.bar-*.baz ?

If we drop foo

3721323156e921b5.001
...
3721323156e921b5.009
3721323156e921b5.baz
3721323156e921b5.bar1-2.baz
3721323156e921b5.bar3-4.baz
3721323156e921b5.bar4-5.baz
3721323156e921b5.bar7-8.baz

Would also be ok. Some more questions:

1. how do I target from beginning of file until .foo?
2. how do I loop the created variable e.g. result=$(openssl rand -hex 8) in order to use it in the renames, and only when a set is done, to assign it again to loop it through until the next set is done, etc.?

Thanks!

Answer 1

There are several parts to this problem:

1. Decompose each file name into a base part and extensions.
2. Apply a consistent transformation to the base part of each name.
3. Rename the files according to the chosen transformation of the base part, preserving the extensions.

1. Decomposing file names

It's not completely clear from your example names what you consider to be the base part of a file name. The separator is evidently a dot, but in an example like yet.a.different.file.name.foo.bar1-2.baz, which dot? You mention an attempt using *.(<->|baz|bar<->.baz), which would not treat foo or bar1-2 as an extension. A tweak that allows them as an extension is .(foo|<->|baz|bar<->(|-<->).baz). Then you can break a file name $f as follows:

setopt extended_glob
base=${f%%(.(<->|baz|bar<->(|-<->).baz))#}; extensions=${f#$base}

Alternatively, if it's acceptable to define the base as everything up to and excluding the first .foo., the decomposition is simpler:

base=${f%*.foo.*}; extensions=${f#$base}

2. Applying a consistent transformation

If you want to apply a deterministic transformation, you can just recalculating each time. For example, you can obtain a pseudorandom result by taking a MAC of the name with a secret key, using the same secret key each time.

secret=$(openssl rand -hex 32)
for … # Loop over the files as per (3.), set $base and $extensions as per (1.)
  new_base=${"$(openssl dgst -sha256 -hmac $secret <<<$base)"[-16,-1]}

(Note: the secret key will be visible to other users if they run ps while openssl is running. I assume this is not a problem in your case, but future readers should beware of this.)

If you want to apply a randomized transformation, you need to remember what each base maps to. There are two ways to do that:

• You can group files by their base, and then process one base at a time.
• You can process the files one by one, but remember what each base maps to, and only generate a new mapping if the base hasn't been seen yet.

The second method is easier and the first method has no particular advantage so I'll only show the second method.

Build an associative array mapping bases to new bases.

typeset -A mapping
mapping=()
for … # Loop over the files as per (3.), set $base and $extensions as per (1.)
  if ((!$+mapping[$base])); then
    mapping[$base]=$(openssl rand -hex 8)
  fi
  new_base=$mapping[$base]

3. Renaming the files

Zsh comes with a very useful tool to rename files: zmv. The transformation you want to do is sufficiently complex that zmv doesn't make it trivial: both the file name decomposition and the transformation require additional work. There are some minor advantages to zmv even in your case. In particular, zmv will error out if there's a clash (extremely unlikely due to random factors unless you use shorter lengths). However, due to the difficulty in the name transformation, using zmv is awkward, and a simple loop is easier to write.

Here's a complete snippet using random names.

setopt extended_glob
typeset -A mapping
mapping=()
for f in *.(foo|<->|baz|bar<->(|-<->).baz); do
  base=${f%%(.(foo|<->|baz|bar<->(|-<->).baz))#}; extensions=${f#$base}
  if ((!$+mapping[$base])); then
    mapping[$base]=$(openssl rand -hex 8)
  fi
  new_base=$mapping[$base]
  mv -i -- $f $new_base$extensions
done

Here's a complete snippet using deterministic names for a given value of $secret.

setopt extended_glob
secret=$(openssl rand -hex 32)
for f in *.(foo|<->|baz|bar<->(|-<->).baz); do
  base=${f%%(.(foo|<->|baz|bar<->(|-<->).baz))#}; extensions=${f#$base}
  new_base=${"$(openssl dgst -sha256 -hmac $secret <<<$base)"[-16,-1]}
  mv -i -- $f $new_base$extensions
done

And here's a one-liner using zmv for the deterministic case, using the first .foo. to mark the end of the base. The -w flag helps with the breakdown.

autoload zmv
secret=$(openssl rand -hex 32)
zmv -w '*.foo.*' '${"$(openssl dgst -sha256 -hmac $secret <<<$1)"[-16,-1]}.foo.$2'

Using zmv in the randomized case is trickier because we need to preserve information from one transformation step to the next. We can just pack some code into a command substitution zmv … '$(…; if …; then mapping[$base]=…; …)' because the assignment to mapping would be inside the command substitution subshell and would therefore only have an effect inside the subshell. However, we can use a conditional parameter assignment ${name=word}, to set mapping[$base] only if it's unset.

typeset -A mapping; mapping=()
zmv -w '*.foo.*' '${mapping[${1}]=$(openssl rand -hex 16)}.foo.$2'

Using zmv with a decomposition that doesn't take advantage of .foo, like the more complex example in (1.) above, results in much more complex code. Just for the sake of the example, here's a zmv invocation for the deterministic case, using base as a temporary variable to store the base name. It uses ${name::=word} to assign to a variable during parameter expansion, and ${…}[0] to suppress that part from the expansion ([0] takes the substring from the 0th character, which doesn't exist since zsh starts numbering array elements and string characters at 1; something like [2,1] would also work).

zmv  '*.(<->|baz|bar<->.baz)' '${${base::=${f%%(.(<->|baz|bar<->(|-<->).baz))#}}[0]}${"$(openssl dgst -sha256 -hmac $secret <<<$base)"[-16,-1]}.${f#$base}'

Answer 2

You could do something like:

autoload -Uz zmv # best in ~/.zshrc
typeset -A rand
zmv '(*).foo(.*)' '${rand[$1]=$(openssl rand -hex 8)}$2'

or '(*)(.foo.*)' to not drop .foo.

To test first, add the -n option (dry-run) to zmv.

zmv is a batch renaming tool implemented as an autoloadable function.

The first argument is an extended glob pattern, and second argument a string that undergoes word expansions that determines how the files are removed with $1, $2... referring to the corresponding (...)s in the pattern.

${rand[$1]=$(cmd)} above sets the member of the associative array for the key being what's left of the rightmost .foo. to the output of cmd if was previously unset which makes sure you always get the same value of a given key.