You didn't mention on which OS you are running or which version of awk you're using, but in trying to reproduce your problem on my Linux host with GNU awk (4.1.1), it works fine for me. Given:

#!/bin/awk -f 

BEGIN { exit 1 }

Which when when placed in a file test.sh results in:

$ ./test.sh
$ echo $?
1

I agree with the commenter who said that test.sh is an odd name for an awk script. I would go with a .awk extension to make it clear that it's not actually a shell script.

You didn't mention on which OS you are running or which version of awk you're using, but in trying to reproduce your problem on my Linux host with GNU awk (4.1.1), it works fine for me. Given:

#!/bin/awk -f 

BEGIN { exit 1 }

Which when when placed in a file test.sh results in:

$ ./test.sh
$ echo $?
1

I agree with the commenter who said that test.sh is an odd name for an awk script. I would go with a .awk extension to make it clear that it's not actually a shell script.

Update

On RHEL 6.7:

$ rpm -q redhat-release-server
redhat-release-server-6Server-6.7.0.3.el6.x86_64

With awk:

$ awk --version
GNU Awk 3.1.7

Everything seems fine. The script:

$ cat test.awk
#!/bin/awk -f

BEGIN {exit 1}

The result:

$ ./test.awk
$ echo $?
1