I don't see what there would be to gain in disallowing the program path on the shebang line to be relative. The person that wrote the shebang can also write relative paths in some other part of the file.
I'm not reading it like that. The tone is just one of surprise, since this isn't something that one typically sees. Since it's obscure, it leads one to wonder if it can be bad, and I don't see how it could be.
I think it survived in the independent Linux because it's the simple and obvious way to do things, and it doesn't lead to any exceptional power of misuse one didn't already have with writing the rest of the file.
An example: if you made a language in python /bin/my_lang: #does nothing but pretend it does
#!/usr/local/bin/python3
import sys
print('my_lang args', sys.argv)
for line in sys.stdin:
print('invalid_line:', line)
my_script: #!/bin/my_lang
line of stuff
another line of stuff
chmod u+x my_script
./my_script
Probably for the best, but I was a bit sad that my recursive interpreter scheme was not going to work.Update: looks like linux does allow nested interpreters, good for them.
https://www.in-ulm.de/~mascheck/various/shebang/#interpreter...
really that whole document is a delightful read.
1. You chmod my_script twice.
2. Did you chmod u+x /bin/my_lang too? Since you put it in /bin, are you sure the owner isn't root?, in which case your user wouldn't have execute permission. Try +x instead of u+x.
3. Do you have python in that path? Try `/usr/bin/env python` instead.
4. In case you expected otherwise, my_script wouldn't be passed through stdin. It's just provided as an argument to my_lang.
(Actually, this is how the `curl install.sh | bash` anti pattern works. )
That's not what the article was actually about, as it turned out. The surprise in the article was about relative paths for script shebang lines. Which was useful to learn about, of course, but I was actually surprised by the surprise.
https://www.youtube.com/watch?v=J8nblo6BawU is some great watching on how "Plain text isn't that simple"
Sure, but who does this? All the Microsoft tooling writes 0xEF 0xBB 0xBF if you output utf8 with a BOM.
This is of course in stark contrast to dynamic linking, which is performed by a userspace program instead of the kernel, and much like the #!, this "interpreter"'s path is also hardcoded in dynamically linked binaries.
As for scripts vs elf executables, there's not much of a difference between the shebang line and PT_INTERP, just that parsing shebangs lines is simpler.
I'm not sure what the contrast is. In both cases the interpreter lives in userspace, in both cases kernel finds it via the path hardcoded in the file: shebang for text, .interp for ELF
Someone may have dropped a malicious executable somewhere in the user's path that the shebang calls. The someone shouldn't be able to do that, but "shouldn't" isn't enough for security.
Or maybe the relatively pathed executable has unexpected interactions with the shebanged script, compared to what the script author expected.
Etc.
Does the author not like Firefox or what?
Seems to be a fairly custom attempt at keeping poorly written scrapers off the site. curl works just fine for me. (I found these redirect URLs by manually setting the user agent header in curl) Fun idea.
What's your browser's user agent? You might have an extension that alters yours to not report your browser's actual version number.
This example works on many platforms that have a shell compatible with Bourne shell:
#!/usr/bin/perl
eval 'exec /usr/bin/perl -wS $0 ${1+"$@"}'
if 0; # ^ Run only under a shell
The system ignores the first line and feeds the program to /bin/sh, which proceeds to try to execute the Perl program as a shell script. The shell executes the second line as a normal shell command, and thus starts up the Perl interpreter. On some systems $0 doesn't always contain the full pathname, so the "-S" tells Perl to search for the program if necessary. After Perl locates the program, it parses the lines and ignores them because the check 'if 0' is never true. If the program will be interpreted by csh, you will need to replace ${1+"$@"} with $*, even though that doesn't understand embedded spaces (and such) in the argument list. To start up sh rather than csh, some systems may have to replace the #! line with a line containing just a colon, which will be politely ignored by Perl. Other systems can't control that, and need a totally devious construct that will work under any of csh, sh, or Perl, such as the following: eval '(exit $?0)' && eval 'exec perl -wS $0 ${1+"$@"}'
& eval 'exec /usr/bin/perl -wS $0 $argv:q'
if 0; # ^ Run only under a shell
[0]: https://perldoc.perl.org/perlrun#-SYou can have spacing after #! for some reason?
POSIX does not mandate -S, which means any script that uses it will only work on freebsd/linux
"The only reason to start your script with '#!/usr/bin/env <whatever>' is if you expect your script to run on a system where Bash or whatever else isn't where you expect (or when it has to run on systems that have '<whatever>' in different places, which is probably most common for third party packages)."
His very first point is how you should only use it don't know where to expect bash binary, when I feel like, while it's probably safe in most nix os', assuming it limits future enhancements by requiring that binary be in place. However unlikely it would need to or someone would want to move it.
Nowadays, most distros are moving towards having /bin be a symlink to /usr/bin, so it's mattering less and less, but I see no reason not to just do /usr/bin/env which is supposed to be on the same place on every distro.
Maybe `#! env <shell>` could be considered a DSL for hashbangs. My reasoning is that `/usr/bin/env` is the thing that seems to be hard-coded to a system path, in most cases.
Is it bad? Well it's less secure. But if you're worried about `/usr/bin/env` calling a malicious program then you need to call out the path for every executable in the script and there's a hell of a lot more other things to worry about too.
It's the same for `#!/usr/bin/env python3` in python scripts. Python3 itself might be ancient at system install, but you might need to be using a venv. So /usr/bin/env python3 works correctly while /usr/bin/python3 works incorrectly.
So is it bad? No.
You never have to "worry about whether the environment was activated", unless your code depends on those environment variables (in which case your shebang trick won't help you). Just specify the path to the venv's Python executable.
You aren't really intended to put your own scripts in the venv's bin/ directly, although of course you can. An installer will create them for you, from the entry points defined in pyproject.toml. (This is one of the few useful things that an "editable install" can accomplish; I'm generally fairly negative on that model, however.)
If you have something installed in a venv and want to run it regardless of CWD, you can symlink the wrapper script from somewhere on your PATH. (That's what Pipx does.)
I've done this for years to keep my individual projects separate and not changing activations when switching directories. I also make sure to only call `venv/bin/pip` or `venv/bin/python3` when I'm directly calling pip or python. So, yes -- you have to be in the root project directory for this to work, but for me, that's a useful tradeoff. Even when running code from within a docker container, I still use this method, so I make sure that I'm executing from the proper work directory.
If I think that I need to run a program (without arguments), I'll have a short shell wrapper that is essentially:
#!/bin/bash
cd $(dirname $0)
venv/bin/python3 myscript.py
As far as running a program that's managed by venv/pip, symlinks are essentially what I do. I'll create a new venv for each program, install it in that venv, and then symlink from venv/bin/program to $HOME/.local/bin/. It works very well when you're installing a pip managed program. project_root = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, project_root)What if the user doesn't have a venv created? What if they created it in a different directory? What if they created a second venv and want to use that instead? What if the user uses `.venv` instead of `venv`?
`#!/usr/bin/env python3` solves most of that.
These are programs that are largely meant to have a run.sh or install.sh script run before the main script. If the venv doesn’t exist, the it is created there and requirements installed.
The main point is that I’m trading away some flexibility to keep my ENV clean. When I submit jobs on HPC clusters, keeping my environment clean tends to make things easier to troubleshoot.
If I’m switching between different programs or commonly piping data between two different programs with their own venvs, it can be easier to just run the associated python binary directly, rather than have to manage different venv environment activations.
`source venv/bin/activate` from your .sh files will cause `/usr/bin/env python3` to use the python3 located in your venv. Switching between venvs is easy too. just call `deactivate` when one venv is activated. It drops you out of the venv. You can then cleanly `source venv2/bin/activate`.
I don't like sourcing things into my environment. I've worked this way for years. I think the idea of 'activating' and 'deactivating' an environment is an anti-pattern. But I also work on HPC clusters where all of the configuration about paths is handled by the environment. Because of this, I've learned the hard way that for my workflows, it's far too easy to have the wrong environment loaded with venvs and modules that it's often better to keep things explicitly defined. I don't like magic, so I explicitly state which venv my code (or occasionally other people's code) is loading from.
I sometimes will have to run multiple programs (that have their own venvs) and pipe data between them. If I have to source and deactivate different venvs for each tool, it just doesn't work right.
I think that's part of the power of virtualenv as a tool -- it's flexible in how it works. I can still use my explicit workflows with the same tooling as everyone else, and you can source your environments and keep happily coding along. For me, that's why I keep using them...
I'm sorry you feel that way.
Ultimately, it is your code and you can do with it whatever you like.
...until your code becomes company code or open sourced. Then your way becomes a hinder to other developers.
> I don't like sourcing things into my environment. I've worked this way for years. I think the idea of 'activating' and 'deactivating' an environment is an anti-pattern.
I completely agree with you. The whole concept of a venv is great! But the concept of needing to source an activation script is... just... completely foreign to me. It took me months to understand that's the way it was intended to work, and more years to stop fighting it.
> I sometimes will have to run multiple programs (that have their own venvs) and pipe data between them.
Me too! I pipe data around all the time because it's amazingly fast and amazingly awesome to just hook up a pipeline. It can be done with venvs, too. Consider this:
#!/usr/bin/env bash
set -euo pipefail
jq '.' < <(source venv1/bin/activate; ./script1) > >(source venv2/bin/activate; ./script2)
Here, script1 (which requires the first venv) might produce some JSON stuff to be processed by `jq`, then pipe that to script2 (which requires the second venv).I'm curious how you do it though.
The vast majority of my shell/user scripts are now in TS with Deno... The only thing I do wish worked better is that VS Code would look at the shebang to determine a file's language without a file extension.
Also works nicely with project automation and CI/CD integration as well... You can do a lot more, relatively easier than having to have knowledge of another language/tool you aren't as comfortable with.
tombert•2mo ago
NixOS is annoying because everything is weird and symlinked and so I find myself fairly frequently making the mistake of writing `#!/bin/bash`, only to be told it can't find it, and I have to replace the path with `/run/current-system/sw/bin/bash`.
Or at least I thought I did; apparently I can just have done `#!bash`. I just tested this, and it worked fine. You learn something new every day I guess.
miffe•2mo ago
tombert•2mo ago
adastra22•2mo ago
tombert•2mo ago
adastra22•2mo ago
Again, this is kernel code. I admit I'm a bit confused as to why it didn't work on your system. This shouldn't be handled at the shell level.
adastra22•2mo ago
jolmg•2mo ago
> If the program is a file beginning with ‘#!', the remainder of the first line specifies an interpreter for the program. The shell will execute the specified interpreter on operating systems that do not handle this executable format in the kernel.
Taking a quick look at the source in Src/exec.c:
I guess at some point someone added that `|| eno == ENOENT` and the docs weren't updated.opello•2mo ago
[1] https://sourceforge.net/p/zsh/code/ci/29ed6c7e3ab32da20f528a...
[2] https://sourceforge.net/p/zsh/code/ci/29ed6c7e3ab32da20f528a...
[3] https://www.zsh.org/mla/workers/2010/msg00522.html
[4] https://www.zsh.org/mla/workers/2000/msg01168.html
tombert•2mo ago
I don't have fish installed and can't be bothered to go that far, but I suspect they're right about that as well.
anotherhue•2mo ago
https://nixos.wiki/wiki/Nix-shell_shebang
SAI_Peregrinus•2mo ago
opello•2mo ago
https://github.com/torvalds/linux/blob/v6.17/fs/binfmt_scrip...
I think it makes it to calling open_exec but there's a test for BINPRM_FLAGS_PATH_INACCESSIBLE, which doesn't seem relevant since 'bash' isn't like '/dev/fd/<fd>/..', but does provoke an ENOENT.
https://github.com/torvalds/linux/blob/v6.17/fs/exec.c#L1445
Maybe someone else can explain it, I'd enjoy the details, and ran out of steam.
Crestwave•2mo ago
hamandcheese•2mo ago
tombert•2mo ago
I just tried it and they were absolutely right, so `#!/usr/bin/env bash` is definitely more portable in that it consistently works.
saurik•2mo ago
Ferret7446•2mo ago
Pedantic, but "#!" and "portable" don't belong in the same sentence
Crestwave•2mo ago
Meanwhile, `#!/usr/bin/env` is completely portable in the practical sense. Even systems with non-standard paths like NixOS, Termux and GoboLinux patch in support for it specifically.
nflekkhnnn•2mo ago
normie3000•2mo ago
eyelidlessness•2mo ago
int_19h•2mo ago
It's very common for Python. Less so for Bash for two reasons: because the person who writes the script references /bin/sh instead (which is required to be there) even when they are writing bash-isms, or because the person who writes the script assumes that Bash is universally available as /bin/bash.
jolmg•2mo ago
Other interpreters like python, ruby, etc. have more likelyhood of being used with "virtual environments", so it's more common to use /usr/bin/env with them.
tombert•2mo ago
rmunn•2mo ago
chuckadams•2mo ago
somat•2mo ago
It does get awkward, especially when porting. all your third party libraries and includes are in /usr/local/lib /usr/local/include but at least it is better than freebsd which also insists on putting all third party configs under /usr/local/etc/
Arch-TK•2mo ago
So it's not really a standard.
/bin/sh is a much more common convention but once again, not a standard.
There really isn't a truly portable shebang, but the same can be said about executables themselves. As part of the build or install step of whatever thing you're making, you should really be looking these up and changing them.
What's more, bash isn't a standard shell.
jolmg•2mo ago
> /bin is the "standard" location for bash on a subset of Linux distributions
Considering "location" such that it includes /bin symlinks, that would be nearly all distros, I would think...
> What's more, bash isn't a standard shell.
De facto and specifically among Linux distros, it is. It's probably an underestimate that 95% of all Linux distro installations have it preinstalled.
Arch-TK•2mo ago
Bash has to be explicitly installed on OpenBSD, FreeBSD, NetBSD (I think, haven't used it in a while) and probably a bunch of others. And in all of those cases (that I know of) it doesn't end up in /bin/bash once installed.
The default bash shipped on macs is so abhorrently ancient that it would be strictly better if it didn't exist because it would reduce the number of people who think bash scripts I write are broken (they're not broken, they just inevitably depend on some bash 4+ feature). Moreover, hardcoding /bin/bash as your shebang in this case will prevent anyone from remediating this problem by installing a non-ancient bash because the old one in /bin/bash will still get used.
latexr•2mo ago
You’re forgetting macOS. It has been using /bin/bash forever.
rottingchris•2mo ago
johnisgood•2mo ago
hulitu•2mo ago
if you have /usr/bin/env
kreetx•2mo ago
SAI_Peregrinus•2mo ago
> Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH , ensuring that the returned pathname is an absolute pathname and not a shell built-in.
[1] https://pubs.opengroup.org/onlinepubs/9799919799/
saghm•2mo ago
sgarland•2mo ago
saintfire•2mo ago
tombert•2mo ago
kreetx•2mo ago
jlokier•2mo ago
Contrary to popular belief, those aren't in the POSIX standard.
The following are not in the POSIX standard, they are just widely implemented:
Guvante•2mo ago
#! is certainly in the POSIX standard as the exact topic of "is /bin/sh always a POSIX" shell is a discussion point (it is not guaranteed since there were systems that existed at the time that had a non-POSIX shell there)
johnisgood•2mo ago
Shebang is a kernel feature, for example, and POSIX does define the sh shell language and utilities, but does not specify how executables are invoked by the kernel.
Similarly, POSIX only requires that sh exists somewhere in the PATH, and the /bin/sh convention comes from the traditional Unix and FHS (Filesystem Hierarchy Standard), but POSIX does not mandate filesystem layout.
... and so on.
Correct me if I am wrong, perhaps with citations?
wahern•2mo ago
IME /bin/sh is invariably sufficiently POSIX conformant to bootstrap into a POSIX shell (or your preferred shell), even on Solaris and AIX. And if you're willing to stick to simple scripts or rigorously test across systems, sufficient for most tasks. Outside Linux-based systems it's usually ksh88, ksh93, pdksh, or some derivative. OTOH, for those who are only familiar with bash that may not be particularly helpful.
I've had more trouble, including bugs, with other utilities, like sed, tr, paste, etc. For shell portability it's usually niche stuff like "$@" expansion with empty lists, for example how it interacts with nounset or IFS, independent of POSIX mode.
johnisgood•2mo ago
kevincox•2mo ago
teo_zero•2mo ago
I think you're mixing two concepts: relative paths (which are allowed after #! but not very useful at all) and file lookup through $PATH (which is not done by the kernel, maybe it's some shell trickery).
aaronmdjones•2mo ago
It's libc. Specifically, system(3) and APIs like execvp(3) will search $PATH for the file specified before passing that to execve(2) (which is the only kernel syscall; all the other exec*() stuff is in libc).