Can ** (bash's globstar) run out of memory?

Question

Can using bash's globstar (**) operator cause an out of memory error? Consider something like:

for f in /**/*; do printf '%s\n' "$f"; done

When ** is being used to generate an enormous list of files, assuming the list is too large to fit in memory, will bash crash or does it have a mechanism to handle this?

I know I've run ** on humongous numbers of files and haven't noticed a problem, so I am assuming that bash will use something like temporary files to store some of the list as it is being generated. Is that correct? Can bash's ** handle an arbitrary number of files or will it fail if the file list exceeds what can fit in memory? If it won't fail, what mechanism does it use for this? Something similar to the temp files generated by sort?

Answer 1

Yes, it can, and this is explicitly accounted for in the globbing library:

  /* Have we run out of memory?  */
  if (lose)
    {
      tmplink = 0;

      /* Here free the strings we have got.  */
      while (lastlink)
        {
          /* Since we build the list in reverse order, the first N entries
             will be allocated with malloc, if firstmalloc is set, from
             lastlink to firstmalloc. */
          if (firstmalloc)
            {
              if (lastlink == firstmalloc)
                firstmalloc = 0;
              tmplink = lastlink;
            }
          else
            tmplink = 0;
          free (lastlink->name);
          lastlink = lastlink->next;
          FREE (tmplink);
        }

      /* Don't call QUIT; here; let higher layers deal with it. */

      return ((char **)NULL);
    }

Every memory allocation attempt is checked for failure, and sets lose to 1 if it fails. If the shell runs out of memory, it ends up exiting (see QUIT). There’s no special handling, e.g. overflowing to disk or handling the files that have already been found.

The memory requirements in themselves are small: only directory names are preserved, in a globval structure which forms a linked list, storing only a pointer to the next entry and a pointer to the string.