Is parsing scripts at script-runtime ubiquitous to shells or present in other interpreters and how does that work?

Question

I had always thought that shells parse whole scripts, constructing an AST, and then execute that AST from memory. However, I just read a comment by Stéphane Chazelas, and tested executing this script, edit-while-executing.sh:

#!/bin/bash

echo start
sleep 10

and then while it was sleeping running:

$ echo "echo end" >> edit-while-executing.sh

and it worked to cause it to print "end" at the end.

However, when trying to modify this:

#!/bin/bash

while true; do
  echo yes
done

by doing:

$ printf "%s" "no " | dd of=edit-while-executing.sh conv=notrunc seek=35 bs=1

It didn't work, and kept printing "yes".

I also wondered if other non-shell interpreters also worked like this, and tried the equivalent of the first script with python, but it didn't work. Though, maybe python is not an interpreter anymore and it's more of a JIT compiler.

So to reiterate my question, is this a behaviour ubiquitous to shells and limited to them or also present in other interpreters (those not regarded as shells)? Also how does this work such that could I do the first modification but not the second?

Answer 1

This feature is present in other interpreters that offer what is called a read eval print loop. LISP is a pretty old language with such a feature, and Common LISP has a read function that will read in here the expression (+ 2 2) which can then be passed to eval for evaluation (though in real code you may not want to do it this way for various security reasons):

% sbcl
* (defparameter sexp (read))
(+ 2 2)

SEXP
* (print (eval sexp))

4
4

we can also define our own very simple REPL without much in the way of features or debugging or pretty much anything else, but this does show the REPL parts:

* (defun yarepl () (loop (print (eval (read))) (force-output) (fresh-line)))

YAREPL
* (yarepl)
(* 4 2)

8
(print "hi")

"hi"
"hi"

Basically like it says on the nameplate data is read in, evaluated, printed, and then (assuming nothing crashed and there is still electricity or something powering the device) it loops back to the read No need to build an AST up in advance. (SBCL needs the force-output and fresh-line additions for display reasons, other Common LISP implementations may or may not.)

Other things with REPL include TCL ("a shell bitten by a radioactive LISP") which includes graphics stuff with Tk

% wish
wish> set msg "hello"
hello
wish> pack [label .msg -textvariable msg]
wish> wm geometry . 500x500
wish> exit

Or FORTH here to define a function f>c to do temperature conversion (the " ok" are added by gforth):

% gforth
Gforth 0.7.3, Copyright (C) 1995-2008 Free Software Foundation, Inc.
Gforth comes with ABSOLUTELY NO WARRANTY; for details type `license'
Type `bye' to exit
: f>c ( f -- c ) 32 - 5 9 */ cr . cr ;  ok
-40 f>c
-40
 ok
100 f>c
37
 ok
bye

Answer 2

So, this runs indefinitely in Bash/dash/ksh/zsh (or at least until your disk fills up):

#!/bin/sh
s=$0
foo() { echo "hello"; echo "foo" >> $s; sleep .1; }
foo

The thing to note, is that only stuff appended added to the script file after the last line the shell has read matters. The shells don't go back to re-read the earlier parts, which they even couldn't do, if the input was a pipe.

The similar construct doesn't work in Perl, it reads the whole file in before running.

#!/usr/bin/perl -l    
open $fd, ">>", $0;
sub foo { print "hello"; print $fd 'foo;' }
foo;

We can see that it does so also when given input through a pipe. This gives a syntax error (and only that) after 1 second:

$ (echo 'printf "hello\n";' ; sleep 1 ; echo 'if' ) | perl 

While the same script piped to e.g. Bash, prints hello, and then throws the syntax error one second later.

Python appears similar to Perl with piped input, even though the interpreter runs a read-eval-print loop when interactive.

---

In addition to reading the input script line-by-line, at least Bash and dash process arguments to eval one line at a time:

$ cat evaltest.sh
var='echo hello
fi'
eval "$var"
$ bash evaltest.sh
hello
evaltest.sh: eval: line 4: syntax error near unexpected token `fi'
evaltest.sh: eval: line 4: `fi'

Zsh and ksh give the error immediately.

Similarly for sourced scripts, this time Zsh also runs line-by-line, as do Bash and dash:

$ cat sourceme.sh
echo hello
fi
$ zsh -c '. ./sourceme.sh'
hello
./sourceme.sh:2: parse error near `fi'

Answer 3

At least one shell, fish, doesn't exhibit this behaviour (but then fish is unusual in other ways):

% for sh in zsh mksh fish dash bash tcsh; do echo 'sleep 5' > foo.$sh; $sh foo.$sh & sleep 1; echo 'echo $0' >> foo.$sh; fg; done
[2] 7381
[2]  - 7381 running    $sh foo.$sh
foo.zsh
[2] 7385
[2]  - 7385 running    $sh foo.$sh
foo.mksh
[2] 7387
[2]  - 7387 running    $sh foo.$sh
[2] 7390
[2]  - 7390 running    $sh foo.$sh
foo.dash
[2] 7393
[2]  - 7393 running    $sh foo.$sh
foo.bash
[2] 7415
[2]  - 7415 running    $sh foo.$sh
foo.tcsh

---

(A previous version of this answer had mistaken observations of Python and Ruby.)