14

Context for the question: According to POSIX specs, ARG_MAX is maximum length of command-line arguments to exec() family of functions. Which lead me to believe that's actual number of arguments, however that clearly didn't work:

$ ulimit -s
8192
$ touch {1..18000}.jpg
$ rm *.jpg
$

Clearly, this works fine, despite being in length over 8192 items. According to D.W.'s answer, the 8192 is supposedly size in kB. So clearly the previous assumption was wrong.

This is where the actual question comes in: How do I figure out the amount of items that actually will get above 8192 kB limit ? In other words, what sort of computation I have to perform to ensure that *.jpg type of glob will result into Argument list too long error ?

Please note, this isn't a duplicate of What defines the maximum size of single command argument. I know about getconf ARG_MAX and ulimit -s values, that's not my question. I need to know how to generate enough arguments in size that will be above the limit. In other words, I need to find a way to get the error, not avoid it.

Improve this question
12
  • What is the result of getconf ARG_MAX for you? Commented Jun 23, 2018 at 18:03
  • @Jesse_b 2097152 . And I'm on Linux, btw Commented Jun 23, 2018 at 18:06
  • Are you trying to manually reproduce xargs? Commented Jun 23, 2018 at 18:23
  • 1
    @JeffSchaller No, I wanted to generate enough files to play around with various methods of handling Argument list too long error. So far I've figured out ulimit -s gives stack space, ARG_MAX is subset to that. I see so far that getconf ARG_MAX gives basically 2M by default for argument size, 8M for stack space. So basically all I need to know is how to create enough filenames to go beyond 2M limit, so that when I do *.jpg in any command that involves exec() family of function would throw that error. Commented Jun 23, 2018 at 18:33
  • 1
    Looks useful: unix.stackexchange.com/a/110301/117549 Commented Jun 23, 2018 at 19:11

4 Answers 4

Reset to default
10

Using getconf ARG_MAX to generate a long list of x and calling an external utility with that as its argument would generate an "Argument list too long" error:

$ /bin/echo $( perl -e 'print "x" x $ARGV[0]' "$(getconf ARG_MAX)" )
/bin/sh: /bin/echo: Argument list too long

The environment and the length of the string /bin/echo will be included in what makes the error occur, so we can try to find the biggest possible number by subtracting these:

$ env
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/usr/local/bin

(I started this shell with env -i sh, so there's only the PATH variable in the environment)

$ /bin/echo $( perl -e 'print "x" x ($ARGV[0] - length($ENV{"PATH"}) - length("/bin/echo"))' "$(getconf ARG_MAX)" )
sh: /bin/echo: Argument list too long

Still too long. By how much?

i=0
while ! /bin/echo $( perl -e 'print "x" x ($ARGV[0] - length($ENV{"PATH"}) - length("/bin/echo") - $ARGV[1])' "$(getconf ARG_MAX)" "$i" )
do
    i=$(( i + 1 ))
done

This loop exits for i=8.

So there's four bytes that I can't immediately account for (four of the eight must be for the name of the PATH environment variable). These are the null terminators for the four strings PATH, the value of PATH, /bin/echo and the long string of x characters.

Note that each argument is null terminated, so the more arguments you have to the command, the shorter the combined length of them can be.

Also, just to show the effect of a big environment:

$ export BIG=$( perl -e 'print "x" x $ARGV[0]' "$( getconf ARG_MAX )" )

$ /bin/echo hello
sh: /bin/echo: Argument list too long

$ /bin/echo
sh: /bin/echo: Argument list too long
Improve this answer
12
  • So ARG_MAX is actually a character count for the whole command line, or is it rather a buffer size? How does character encoding play into this? Commented Jun 24, 2018 at 8:21
  • 2
    @dessert ARG_MAX is the number of bytes that the combined length of the environment and the command argument list, including the command name and any null terminators on each argument (and on environment variable names/values), can be. So multi-byte characters would take more space. It may be a kernel or library buffer size. Commented Jun 24, 2018 at 8:28
  • @dessert Unquoted whitespace between the command's arguments would not count, so it's not the length of the command line. Commented Jun 24, 2018 at 8:29
  • 2
    @Kusalananda, I've understood it's the limit on the argument string size, it's discussed in the linked question. E.g. this would trigger it: /bin/echo $( perl -e 'print "x" x $ARGV[0]' 131072 ) > /dev/null (one less works). I think PATH_MAX is usually 4096 (again, on Linux), and I don't think it's related. Commented Jun 24, 2018 at 20:14
  • 1
    @ilkkachu Thanks for pointing this out. That limit does not seem to be an issue on OpenBSD, but I'll find a Linux system somewhere and may update the answer when I have time. Commented Jun 24, 2018 at 20:23
8

On many Linux distributions, you can find out what's the current value of ARG_MAX by running

getconf ARG_MAX

A simple way to generate the message "Argument list too long" is to actually supply a long argument list; for example, on my system

/bin/echo "$(find / -xdev 2> /dev/null)"

works, but

/bin/echo "$(find / -xdev 2> /dev/null)" "$(find / -xdev 2> /dev/null)"

produces

bash: /bin/echo: Argument list too long

For an in-depth discussion, see "Argument list too long error for rm, cp, mv commands" on Stack Overflow.

P.S. The length in question is the actual amount of memory needed to store the arguments and environment. It is not related to the number of arguments.

Improve this answer
1
  • It's ever so slightly related to the number of arguments since each argument would need a null terminator. So having helloworld would use 11 bytes, while hello and world would use 12. Commented Jun 24, 2018 at 8:36
6

To make it easier start a new shell (exit when done) and set the limit to a lower value (100kbytes):

$ bash
$ ulimit -s 100

With the example you used, the error could be triggered:

$ touch {1..100000}.jpg
bash: /usr/bin/touch: Argument list too long

But better to use something like echo. Understand that a builtin might not trigger an error, this command should work correctly:

$ echo {000001..100000}6789

Also note that the amount of bytes is well over the limit set above (1.1Mbyte):

$ echo {000001..100000}6789 | wc -c
1100000

But this command won't work:

$ /bin/echo {000001..100000}6789
bash: /bin/echo: Argument list too long

The limit is set by the size of the argument list in bytes added to the environment size of the shell.

Improve this answer
6

Instead of listing all the files on the system, which takes forever and doesn't work on sparse systems, the command

/bin/echo {1..200000}

is a much faster (163ms) way to generate the error. For me ARG_MAX is 2097152 (2,097,152 on the order of 2 million) but the command still errors.

If your shell doesn't have {start..end}, you can use the somewhat slower (but still faster than listing files)

/bin/echo $(seq 1 200000)

If either command doesn't work, just increase the end value until it does, until bash runs out of memory, or until seq can't count any higher.

Improve this answer
4
  • You don’t need seq for this, /bin/echo {1..200000} also works. Commented Jun 24, 2018 at 9:00
  • @LucasWerkmeister: Only if the shell supports this feature. Commented Jun 24, 2018 at 9:27
  • @DavidFoerster sure, but you mentioned Bash in your comment already. Commented Jun 24, 2018 at 13:32
  • @LucasWerkmeister good point, it's also exponentially faster Commented Jun 24, 2018 at 18:19

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.