The '#!'-magic, details about the shebang mechanism on various Unix flavours

2006-03-07 (see recent changes)

Here you'll find

• More reading about the history
• Selected details
• A table with test results on numerous systems

More reading:

• The origin: See an old mail from Dennis Ritchie introducing the new feature, found in 4.0BSD, /usr/src/sys/newsys/sys1.c

  So this mechanism was invented between Version 7 and Version 8. It was also available in 4BSD but not activated until 4.2BSD.

  (All this pointed out by Gunnar Ritter in <3B5B0BA4.XY112IX2@bigfoot.de> in de.comp.os.unix.shell.)

• The Unix FAQ, paragraph "3.16) Why do some scripts start with #! ... ?" (local copy),
  with emphasis about the history concerning shells, but not the kernel.

  That document is wrong about two details:

  • #! was not invented at Berkeley (but they implemented it first in officially distributed releases), see above.
  • The document explicitly states that only csh was modified on the BSDs.
    However, the Bourne shell was modified likewise as well.
    See the first occurence in 3BSD usr/src/cmd/sh/service.c.

• Meanwhile, i found an article from Andries Brouwer, which you shouldn't miss. It emphasizes some other things which are not explained here and follows a more generic approach (and it differs concerning a very few details).

Selected details:

• There is a rumor, that a very few and very special, earlier Unix versions (particularly 4.2BSD derivatives) require you to separate the "#!" from the following path with a blank. You may also read, that (allegedly) such a kernel parses "#! /" as a 32-bit (long) magic.

  But it turns out that it is virtually impossible to find a Unix which actually required this.

  4.2BSD in fact doesn't require it, although the GNU autoconf tutorial wrongly claims this (Autoconf Manual: 10. Portable Shell Programming).

  But instead, see (again 4.0BSD /usr/src/sys/newsys/sys1.c from above and) the first regular occurence in 4.2BSD, /usr/src/sys/sys/kern_exec.c. The source accepted a blank, but never required it.

  All the above pointed out by Gunnar Ritter in <3B5B0BA4.XY112IX2@bigfoot.de> (and due to the new Caldera license, i can reproduce the code here now.)

  Instead, the origin of this myth "of the required blank" might be a particular release of 4.1 BSD: There is a manpage in a "4.1.snap" snapshot of 4.1BSD on the CSRG CDs, /usr/man/man2/exec.2 (4/1/81), where a space/tab after the "#!" is mentioned as mandatory. However, this is not true: the source itself remained unchanged. (Hint to such a manpage from Bruce Barnett in <ae3m9l$rti$0@208.20.133.66>).

  It's not clear whether this is a bug in documentation or if Berkeley planned to modify the BSD source, but eventually did not.

  It's also unclear if at least DYNIX, also mentioned in the autoconf documentation, implemented it in a few releases (perhaps even following the abovementioned manual page). However, later releases did not implement it according to Usenet discussions.

  I asked David MacKenzie, the author of the autoconf documentation, about the actual origin of this note.
  But unfortunately neither the reporting author nor the very system are recorded anymore.

  Even intensive search of usenet archives didn't reveal any more hints to me.

• env(1) is often used with the #!-mechanism to start an interpreter, which then only needs to be somewhere in your PATH, e.g. "#!/usr/bin/env perl".

  However, env(1) is not /usr/bin/env but /bin/env at least on OpenServer 5.0.6, Unicos 9.0.2 and IIRC on at least one older Linux distribution. Free-, Net- and OpenBSD in turn only come with /usr/bin/env. So the env-mechanism is increasing convenience, but not strictly assuring portability.

  In practice, env must not be a script, because the mechanism only accepts binary executables (exept on UWIN).

• FreeBSD before 6.0 (hint by Akinori Musha), BSD/OS and Minix split up the arguments like a shell to fill up argv[].

• FreeBSD 4.0 introduced the handling of a "#" in the arguments like a comment character.
  Release 6.0 revoked this (see also a discussion on freebsd-arch).

• About the the maximum length of a #!-line:
  • Originally (Research Unix between Version 7 and 8) it was 16 bytes.

  • 32 bytes on 4.xBSD and SunOS 4. This is "sizeof(struct a.out)". The reason is a union, which contains both struct a.out/exec and a string with the same length (to access the #!-line).

    (On SVR3, earlier HP-UX and on Unicos this limit is not such a union but probably just coded.)

  • On some systems, the line doesn't get truncated if it is too long, but the kernel refuses to execute the file and returns an ENOEXEC. In some shells,you won't get an error or an according exit status in the command line. So it's a quite silent failure in both cases then. However, on a very few systems you get a more appropriate error from the kernel.

  • For the history on FreeBSD, see imgact_shell.c (and <sys/imgact.h>).

  • For the history on NetBSD, see kern/exec_script.c (MAXINTERP in <sys/param.h> or PATH_MAX in <sys/syslimits.h>, respectively).

  • For the history on OpenBSD, see kern/exec_script.c (MAXINTERP in <sys/param.h>).

  • 127 bytes on Linux, see also load_script() in linux/fs/binfmt_script.c. See lxr.linux.no.

  • On many other flavours, the maximum length varies between _POSIX_PATH_MAX (255) and PATH_MAX (f.i. 1024); see limits.h or syslimits.h on the respective system.

    Exceptions are BSD/OS(4.2) with 4096 and FreeBSD 6.0 with 8192.

    Minix also uses the limit of PATH_MAX characters (255 here) but the actual limit is 257 characters, because patch_stack() in src/mm/exec.c first skips the "#!" with an lseek() and then reads in the rest.

• If you ever wanted to be as portable as SUSv2/POSIX.2 allows, then you consequently should not use the #!-mechanism:

      Shell Introduction
      [...]
      If the first line of a file of shell commands starts with the
      characters #!, the results are unspecified.
  
      The construct #! is reserved for implementations wishing to provide
      that extension. A portable application cannot use #! as the first
      line of a shell script; it might not be interpreted as a comment.
      [...]
  
      Command Search and Execution
      [...]
      This description requires that the shell can execute shell
      scripts directly, even if the underlying system does not support
      the common #! interpreter convention. That is, if file foo contains
      shell commands and is executable, the following will execute foo:
  
        ./foo 

  On the other hand, speaking about "#!/bin/sh" on any Unix:
  This is a really rocksolid portable convention, if you accept anything from the Bourne shell family and its descendants to be called.

Test results on various systems:

I used the following as program "showargs":

    #include <stdio.h>
    int main(int argc, char** argv) {
	int i;
	for (i=0; i<argc; i++)
	    fprintf(stdout, "argv[%d]: \"%s\"\n", i, argv[i]);
	return(0);
    } 

and a one line script named "invoker.sh" to call it, similar to this,

    #!/tmp/showargs -1 -2 -3 

to produce the following results (tried them myself, but i'd like to add your results from yet different systems).

Typically, a result from the above would look like this:

    argv[0]: "/tmp/showargs"
    argv[1]: "-1 -2 -3"
    argv[2]: "./invoker.sh"

... but the following table lists the variations. The meaning of the columns is explained below.

OS (arch)	maximum length of #! line	cut-off (c), error (err) or ENOEXEC () [1]	only the 1st arg passed on	each arg in its own argv[x]	handle # like a comment	argv[0]: invoker, instead of interpreter	not full path in argv[0]	remove trailing white- space	accept inter- preter [2]
AIX 3.2.5 (rs6k)	[3] 255/80	c [3]					[3]
AIX 4.3.2 (rs6k)	256						X
BSDi/OS 4.2 (i386)	4096	err		X		?	?	X
EP/IX 2.2.1 (mips)	1024				?
FreeBSD 1.1- / 4.0-4.4	64			X	- / X			X
FreeBSD 4.5-4.x	128	err		X	X			X
FreeBSD 6.0 (alpha)	8192	c						X
HP-UX A.08.07/B.09.03	32					X
HP-UX B.10.10	[4] 128/80	c [4]				[4]	[4]	X
HP-UX B.10.20-11.11	128							X
IRIX 5.3/6.5 (mips)	256	err
Linux 2.2.9-2.6.7	127	c					X	X
Minix 2.0.3-3.1.1	257			X				X	X
MUNIX 3.1 (svr3.x, 68K)	32					X
NetBSD 0.8-1.6Q / 1.6R-	64 / 1024
OpenBSD 2.0-3.4	64
OSF1 V4.0B-T5.1	1024						X
OpenServer 5.0.6 [5]	256	err	X
SINIX 5.20 (MX300/NSC)	32
SunOS 4.1.4 (SPARC)	32	c						X
SunOS 5.x (SPARC)	1024		X					X
Ultrix 4.5 (VAX3100/RISC)	32 / 80	c
Unicos 9.0.2.2 (cray)	32
UWIN (i386)	255	c			?			X	X

Meaning of the columns:

• "maximum length of #!-line": self explanatory
• "cut-off(c), error(err) or ENOEXEC ( )": see the footnote [1] below.
• "only the 1st arg": argv[1] from above contains only "-1" then
• "each arg in its own argv[x]": argv[1]:"-1", argv[2]:"-2", etc.
• "handle # like a comment": if a # appears in the arguments, the rest of the line (including this #) is ignored
• "invoker in argv[0]": argv[0] doesn't contain "/tmp/showargs" but "./invoker.sh"
• "not full path in argv[0]": argv[0] contains the basename of the called program instead of its full path.
• "remove trailing whitespace": self explanatory
  Thanks to Matthew Garrett for pointing out this feature.
• "accept interpreter": the invoked program from the "#!"-line may be an interpreted script itself
• a questionmark means that i couldn't test a detail yet

Footnotes in the table:

And why shebang? In music, '#' means sharp. So just shorten '#!' to sharp-bang. Or it might be derived from "shell bang". All this probably under the influence of the american slang idiom "the whole shebang" (everything, the works, everything involved in what is under consideration).

See also the jargon dictionary or the Merriam-Websters for the slang idiom.

Sometimes it's also called hashbang, sha-bang/shabang, hashexclam or hashpling (british).

<http://www.in-ulm.de/~mascheck/various/shebang/>