Skip to main content
replaced http://stackoverflow.com/ with https://stackoverflow.com/
Source Link

This is happening because cut is outputting NULL characters in the output. You can't pass a program arguments which contain a null character (see thisthis).

In bash this works because bash can't handle NULL characters in strings, and it strips them out. Zsh is a bit more powerful, and it can handle NULL characters. However when it comes time to pass the string to the program, it still contains the null, which signals the end of the argument.

Let's look at this in detail.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3 | xxd
0000000: 456e 676c 6973 6800 4155 0a              English.AU.

Here we simulated one of your files, passing the path through cut. Notice the xxd output which has a NULL character between English and AU.

Now lets run through and simulate the rest of the script.

$ l=$(echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3)
$ dest=/stuff2/${l}.utf8.txt
$ echo "$dest" | xxd
0000000: 2f73 7475 6666 322f 456e 676c 6973 6800  /stuff2/English.
0000010: 4155 2e75 7466 382e 7478 740a            AU.utf8.txt.

Notice the NULL after the English. The echo displays it properly because echo is a shell built-in. If we use an external echo, it also exhibits the issue.

$ /bin/echo "$dest" | xxd 
0000000: 2f73 7475 6666 322f 456e 676c 6973 680a  /stuff2/English.

P.S. You really should be quoting too :-)


The solution is to not use cut, use awk instead.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | awk -F/ '{ print $1$3 }' | xxd
0000000: 456e 676c 6973 6841 550a                 EnglishAU.

This is happening because cut is outputting NULL characters in the output. You can't pass a program arguments which contain a null character (see this).

In bash this works because bash can't handle NULL characters in strings, and it strips them out. Zsh is a bit more powerful, and it can handle NULL characters. However when it comes time to pass the string to the program, it still contains the null, which signals the end of the argument.

Let's look at this in detail.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3 | xxd
0000000: 456e 676c 6973 6800 4155 0a              English.AU.

Here we simulated one of your files, passing the path through cut. Notice the xxd output which has a NULL character between English and AU.

Now lets run through and simulate the rest of the script.

$ l=$(echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3)
$ dest=/stuff2/${l}.utf8.txt
$ echo "$dest" | xxd
0000000: 2f73 7475 6666 322f 456e 676c 6973 6800  /stuff2/English.
0000010: 4155 2e75 7466 382e 7478 740a            AU.utf8.txt.

Notice the NULL after the English. The echo displays it properly because echo is a shell built-in. If we use an external echo, it also exhibits the issue.

$ /bin/echo "$dest" | xxd 
0000000: 2f73 7475 6666 322f 456e 676c 6973 680a  /stuff2/English.

P.S. You really should be quoting too :-)


The solution is to not use cut, use awk instead.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | awk -F/ '{ print $1$3 }' | xxd
0000000: 456e 676c 6973 6841 550a                 EnglishAU.

This is happening because cut is outputting NULL characters in the output. You can't pass a program arguments which contain a null character (see this).

In bash this works because bash can't handle NULL characters in strings, and it strips them out. Zsh is a bit more powerful, and it can handle NULL characters. However when it comes time to pass the string to the program, it still contains the null, which signals the end of the argument.

Let's look at this in detail.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3 | xxd
0000000: 456e 676c 6973 6800 4155 0a              English.AU.

Here we simulated one of your files, passing the path through cut. Notice the xxd output which has a NULL character between English and AU.

Now lets run through and simulate the rest of the script.

$ l=$(echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3)
$ dest=/stuff2/${l}.utf8.txt
$ echo "$dest" | xxd
0000000: 2f73 7475 6666 322f 456e 676c 6973 6800  /stuff2/English.
0000010: 4155 2e75 7466 382e 7478 740a            AU.utf8.txt.

Notice the NULL after the English. The echo displays it properly because echo is a shell built-in. If we use an external echo, it also exhibits the issue.

$ /bin/echo "$dest" | xxd 
0000000: 2f73 7475 6666 322f 456e 676c 6973 680a  /stuff2/English.

P.S. You really should be quoting too :-)


The solution is to not use cut, use awk instead.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | awk -F/ '{ print $1$3 }' | xxd
0000000: 456e 676c 6973 6841 550a                 EnglishAU.
added 214 characters in body
Source Link
phemmer
  • 73.9k
  • 21
  • 199
  • 231

This is happening because cut is outputting NULL characters in the output. You can't pass a program arguments which contain a null character (see this).

In bash this works because bash can't handle NULL characters in strings, and it strips them out. Zsh is a bit more powerful, and it can handle NULL characters. However when it comes time to pass the string to the program, it still contains the null, which signals the end of the argument.

Let's look at this in detail.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3 | xxd
0000000: 456e 676c 6973 6800 4155 0a              English.AU.

Here we simulated one of your files, passing the path through cut. Notice the xxd output which has a NULL character between English and AU.

Now lets run through and simulate the rest of the script.

$ l=$(echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3)
$ dest=/stuff2/${l}.utf8.txt
$ echo "$dest" | xxd
0000000: 2f73 7475 6666 322f 456e 676c 6973 6800  /stuff2/English.
0000010: 4155 2e75 7466 382e 7478 740a            AU.utf8.txt.

Notice the NULL after the English. The echo displays it properly because echo is a shell built-in. If we use an external echo, it also exhibits the issue.

$ /bin/echo "$dest" | xxd 
0000000: 2f73 7475 6666 322f 456e 676c 6973 680a  /stuff2/English.

P.S. You really should be quoting too :-)


The solution is to not use cut, use awk instead.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | awk -F/ '{ print $1$3 }' | xxd
0000000: 456e 676c 6973 6841 550a                 EnglishAU.

This is happening because cut is outputting NULL characters in the output. You can't pass a program arguments which contain a null character (see this).

In bash this works because bash can't handle NULL characters in strings, and it strips them out. Zsh is a bit more powerful, and it can handle NULL characters. However when it comes time to pass the string to the program, it still contains the null, which signals the end of the argument.

Let's look at this in detail.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3 | xxd
0000000: 456e 676c 6973 6800 4155 0a              English.AU.

Here we simulated one of your files, passing the path through cut. Notice the xxd output which has a NULL character between English and AU.

Now lets run through and simulate the rest of the script.

$ l=$(echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3)
$ dest=/stuff2/${l}.utf8.txt
$ echo "$dest" | xxd
0000000: 2f73 7475 6666 322f 456e 676c 6973 6800  /stuff2/English.
0000010: 4155 2e75 7466 382e 7478 740a            AU.utf8.txt.

Notice the NULL after the English. The echo displays it properly because echo is a shell built-in. If we use an external echo, it also exhibits the issue.

$ /bin/echo "$dest" | xxd 
0000000: 2f73 7475 6666 322f 456e 676c 6973 680a  /stuff2/English.

P.S. You really should be quoting too :-)

This is happening because cut is outputting NULL characters in the output. You can't pass a program arguments which contain a null character (see this).

In bash this works because bash can't handle NULL characters in strings, and it strips them out. Zsh is a bit more powerful, and it can handle NULL characters. However when it comes time to pass the string to the program, it still contains the null, which signals the end of the argument.

Let's look at this in detail.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3 | xxd
0000000: 456e 676c 6973 6800 4155 0a              English.AU.

Here we simulated one of your files, passing the path through cut. Notice the xxd output which has a NULL character between English and AU.

Now lets run through and simulate the rest of the script.

$ l=$(echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3)
$ dest=/stuff2/${l}.utf8.txt
$ echo "$dest" | xxd
0000000: 2f73 7475 6666 322f 456e 676c 6973 6800  /stuff2/English.
0000010: 4155 2e75 7466 382e 7478 740a            AU.utf8.txt.

Notice the NULL after the English. The echo displays it properly because echo is a shell built-in. If we use an external echo, it also exhibits the issue.

$ /bin/echo "$dest" | xxd 
0000000: 2f73 7475 6666 322f 456e 676c 6973 680a  /stuff2/English.

P.S. You really should be quoting too :-)


The solution is to not use cut, use awk instead.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | awk -F/ '{ print $1$3 }' | xxd
0000000: 456e 676c 6973 6841 550a                 EnglishAU.
Source Link
phemmer
  • 73.9k
  • 21
  • 199
  • 231

This is happening because cut is outputting NULL characters in the output. You can't pass a program arguments which contain a null character (see this).

In bash this works because bash can't handle NULL characters in strings, and it strips them out. Zsh is a bit more powerful, and it can handle NULL characters. However when it comes time to pass the string to the program, it still contains the null, which signals the end of the argument.

Let's look at this in detail.

$ echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3 | xxd
0000000: 456e 676c 6973 6800 4155 0a              English.AU.

Here we simulated one of your files, passing the path through cut. Notice the xxd output which has a NULL character between English and AU.

Now lets run through and simulate the rest of the script.

$ l=$(echo 'English/Flavors/AU/stuff1/filename.txt' | cut -d'/' --output-delimiter '' -f1,3)
$ dest=/stuff2/${l}.utf8.txt
$ echo "$dest" | xxd
0000000: 2f73 7475 6666 322f 456e 676c 6973 6800  /stuff2/English.
0000010: 4155 2e75 7466 382e 7478 740a            AU.utf8.txt.

Notice the NULL after the English. The echo displays it properly because echo is a shell built-in. If we use an external echo, it also exhibits the issue.

$ /bin/echo "$dest" | xxd 
0000000: 2f73 7475 6666 322f 456e 676c 6973 680a  /stuff2/English.

P.S. You really should be quoting too :-)