Functions digest is NOT treated the same as a directory of function files!

Question

I have a function that uses zcompile to create a digest like so:

function fpath-setup {
  local FLATFPATH="${TMPPREFIX}-${ZSH_VERSION}-fpath.zwc"

  function {
      typeset -a zarr

      blacklist=('ztodo' 'zed')
      blacklist="^(${(j:|:)blacklist})"

      setopt LOCAL_OPTIONS EXTENDED_GLOB

      for fp in "$fpath[@]"; do
        local ztail=(${zarr:t})
        for it in "${fp}/"$~blacklist; do
          if [[ -z "${ztail[(r)${it:t}]}" ]]; then
            if zcompile "${TMPPREFIX}-try-zcompile" "${it}" &>/dev/null; then
              zarr+="${it}"
            else
              echo "CANNOT COMPILE: ${it}"
            fi
          else
            echo "DUPLICATE: ${it}"
          fi
        done
      done

      zcompile "${FLATFPATH}" "$zarr[@]"

      zcompile -t "${FLATFPATH}" 'compinit' '_complete' || {
        print "Important functions missing from ${FLATFPATH}" >&2
        return 1
      }
  }

  if (( $? )); then
    print "The fpath is left unchanged." >&2
    return 1
  else
    fpath=("${FLATFPATH}")
  fi
}

At the point this runs, the $fpath looks like this:

/usr/local/share/zsh/site-functions
/usr/local/Cellar/zsh/5.2/share/zsh/functions

After this runs, the $fpath looks like this:

/var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-fpath.zwc

As you can see here, the core Zsh functions are in the digest:

https://ghostbin.com/paste/wgauy

This seems to work fine until I try and use this prompt theme:

https://github.com/sindresorhus/pure/blob/master/pure.zsh#L328

I get the following error at every prompt:

vcs_info: configured unknown backend: 'git'
vcs_info: use 'vcs_info_printsys' to find supported systems.

This can be resolved by not creating the digest in the first place, or by running the following command before invoking the prompt:

autoload -Uz VCS_INFO_get_data_git VCS_INFO_detect_git

So, my questions are:

1. Is it a bad idea to use zcompile on core Zsh functions?
2. Why do I need to manually autoload some stuff when using the zcompile digest, but that's not the case when using regular files in a directory?

Edit:

I've done some more investigating, and have found that with an fpath like:

/var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-circuit.zwc

The command:

prompt pure # ...vcs_info errors
print -f '%s\n' "${(k)functions[@]}" | grep VCS_INFO_get_data_git

Returns nothing. However, with an fpath like:

/var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-fpath.zwc
/usr/local/Cellar/zsh/5.2/share/zsh/functions

The command:

prompt pure # NO vcs_info errors!
type $(print -f '%s\n' "${(k)functions[@]}" | grep VCS_INFO_get_data_git)

Returns:

VCS_INFO_get_data_git is a shell function from /var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-fpath.zwc/VCS_INFO_get_data_git

So, somehow the mere existence of the core directory /usr/local/Cellar/zsh/5.2/share/zsh/functions causes the VCS_INFO_get_data_git function to be loaded from /var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-fpath.zwc.

Does that mean Zsh gives special treatment the core directory? To test that theory, I copy the core function files to a temporary location:

cp -r /usr/local/Cellar/zsh/5.2/share/zsh/functions /tmp

Then in a fresh session, with an fpath like:

/var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-fpath.zwc
/tmp/functions

The command:

prompt pure # NO vcs_info errors!!
type $(print -f '%s\n' "${(k)functions[@]}" | grep VCS_INFO_get_data_git)

Returns:

VCS_INFO_get_data_git is a shell function from /var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-fpath.zwc/VCS_INFO_get_data_git

Thus, I must conclude that Zsh is NOT treating it's core directory specially, but instead it is autoloading VCS_INFO_get_data_git if a FILE of that name exists in any of the directories on the fpath... However, Zsh will still load from the DIGEST if that comes first in the fpath.

This is strange.

I haven't seen this behavior documented anywhere, and it's not clear to me how Zsh goes about this (maybe someone that knows Zsh internals can explain).

To try and figure out WHEN autoloading happens, I fiddle with the fpath in a fresh session, by changing the fpath from:

/var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-fpath.zwc
/tmp/functions

To remove the /tmp/functions directory:

/var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-fpath.zwc

Then I try changing the prompt:

prompt pure

Which gives the original vcs_info errors, as before:

vcs_info: configured unknown backend: 'git'
vcs_info: use 'vcs_info_printsys' to find supported systems.

To be sure, the test:

print -f '%s\n' "${(k)functions[@]}" | grep VCS_INFO_get_data_git

Returns nothing! Now, if at this point I add the /tmp/functions directory BACK to the fpath again, so it looks like:

/var/folders/kb/ydt74z19765cv9vb86rwvcrr0000gn/T/zsh-5.2-fpath.zwc
/tmp/functions

The vcs_info errors continue to persist!

Can anyone explain what the heck is going on here with Zsh version 5.2?

Answer 1

I was researching similar things so I may have a few things that could be useful.

If you do man zshmisc and go to the "Autoloading Functions" section is explains a little bit of this. http://zsh.sourceforge.net/Doc/Release/Functions.html

It says two things here that I"m going to point out

First is that I think one of your issues is you were missing the -U flag above:

The usual alias expansion during reading will be suppressed if the autoload builtin or its equivalent is given the option -U. This is recommended for the use of functions supplied with the zsh distribution. Note that for functions precompiled with the zcompile builtin command the flag -U must be provided when the .zwc file is created, as the corresponding information is compiled into the latter.`

Second is how it autoloads from the fpath variable (I recommend reading the page, but here it is summarized):

element is the fpath item, and function is the function that's trying to be autoloaded.

There are 3 ways something can be autoloaded.

1. If there's a file called element.zwc it will be loaded and searched for functions. (digest). I personally need to research this one more, but you can see what's in a zwc file by using zcompile -t. (I know you used this above, just posting it here)

2. If three's a wordcode file (compiled file) named element/function.zwc

3. The common one which is if it's a file in the fpath (so just element/function)

Also check out man zshbuiltins and look for the man page for zcompile. You can use the -c and -a to compile all functions or all autoloaded functions: http://zsh.sourceforge.net/Doc/Release/Shell-Builtin-Commands.html

Also you can find another place where the zsh word codes are mentioned if you look up the . builtin on the page. You can see specifically if you use it it'll look for the *.zwc version of the file and load it instead:

If a file named 'file.zwc' is found, is newer than file, and is the compiled form (created with the zcompile builtin) of file, then commands are read from that file instead of file.

There's also a function you can autoload called zrecompile that's a little unrelated, but I recommend checking it out. It essentially recompiles your files as needed:

https://github.com/zsh-users/zsh/blob/284b5aaeb08f90520ac7dc385e8921471678e7dd/Functions/Misc/zrecompile