But `ssh` does deserve all the shame. It's a pity the real problems are hard to find in an article full of nonsense.
Note also that if you're using a deficient shell that supports neither `printf %q` nor `${var@Q}` it's still easy to do quoting in `sed`. GNU `./configure` scripts do this internally, including special-casing to only quote the right side of `--arg=value`.
Pls elaborate. Seems like a decent list of shell gotchas to me.
Or, let’s just look at an excerpt, here’s the section “proper solution:”
I’ve emphasized the actionable advice.
> The proper solution would be dropping that broken tool immediately, securely erasing it from your hard-drive, then running and screaming that tool's name out-loud in shame... (Something akin to Game of Throne's walk of atonement...)
Joke
> I'm not kidding... This kind of broken tools are the cause of many stupid bugs, ranging from the funny ups-rm-with-spaces (i.e. rm -Rf / some folder with spaces /some-file), to serious security issues like the formerly mentioned shellshock...
Joke/contentless stakes raising.
> So, you say someone holds you at gun point, thus you must use that tool? Check if the broken tool doesn't have a flag that disables calling sh -c, and instead properly executes the given command and arguments directly via execve(2). (For example watch has the -x flag as mentioned.)
Here it is, the paragraph that has something!
> Alternatively, given that most likely the tool in question is an open-source project written by someone in his spare time, perhaps open a feature request describing the issue, and if possible contribute with a patch that solves it.
This doesn’t seem practically actionable, at least in the short term—most projects might ignore your patch, or maybe it will take multiple years to get pushed out to distros.
> Still no luck? Make some popcorn and prepare for the latest block-buster "convoluted solutions for simple problems in UNIX town"...
Dramatic buildup/joke.
The article is mostly correct, although it makes some weird claims (e.g., the Shellshock bug had nothing to do with the class of bugs the article is complaining about - it was a vulnerability in the shell itself). It definitely has a "newcomer hates things without understanding why they are the way they are" vibe, but you actually need that every now and then. The old-timers tend to say "it was originally done this way for a reason and if you're experienced enough, you know how to deal with it", but what made sense 30-40 years ago might not make much sense today.
With the assumption that:
1. The person knows to do this weird thing 2. They do it consistently every time 3. They never forget
Also not sure how to use those solutions for the popen() example you provided.
The correct way is:
subprocess.run([
"gzip",
"-c",
"—-to-stdout",
user_input
], stdout=open("foo.gz"))
In this case, current "popen" semantics - a single shell command - works pretty well. You can pass it a custom process:
--data-handler="scripts/handle_data.py"
--data-handler="gzip -c > last_data.gz"
--data-handler "jq .contents | mail -s 'New data incoming' data-notify"
But that’s the same as saying you technically need SQL injection so that `psql -c 'command'` can work
> you cannot simulate all of this with command vector
Uhh, yes we can just call a shell:
subprocess.run(["bash", "-c", data_handler])
Not the same thing, this is vulnerable to $PATH interception. You can hardcode the path to bash to avoid that but there's no guarantee that it'll always be there. system() on the other hand is guaranteed to run the operating system's command interpreter.
This doesn’t give the attacker any access that they wouldn’t have.
Also you can just clobber the “PATH” variable if it is so inclined.
> system() on the other hand is guaranteed to run the operating system’s command intepreter
Yeah that just it is means less predictable.
Please show me python programs which support the pattern shown by the parent post and which actually work when running under powershell. (Or Oil shell or any other non-POSIX shell)
Aside: `/bin/sh` is guaranteed to exist by POSIX
Quite the contrary
> Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH,ensuring that the returned pathname is an absolute pathname and not a shell built-in.
https://pubs.opengroup.org/onlinepubs/9799919799/utilities/s...
subprocess.run(["bash", "-c", "--", data_handler])
Because yeah if your program provides “invoking the shell as a feature” then it sure as fuck needs to invoke the shell. I was just replying to this far-fetched example.
By the way, I think it is still better to do this than calling system because if I read “run([bash” I know the developer meant to do this explicitly. If I read “system()” then I’m probably gonna assume they were just lazy and probably didn’t even know about the extra shell being invoked. (I also said this in my previous comment, please read before replying)
>>> import subprocess
>>> import shlex
>>> subprocess.run(['ssh', 'host', shlex.join(['ls', '-la', 'a filename with spaces'])])
ls: cannot access 'a filename with spaces': No such file or directory
>>> layer2 = ['ls', '-la', 'a filename with spaces']
>>> layer1 = ['ssh', 'host1', shlex.join(layer2)]
>>> layer0 = ['ssh', 'host0', shlex.join(layer1)]
>>> subprocess.run(layer0)
Not in the standard library, but there are packages.
>>> subprocess.run(["fish", "-c", shlex.join(["echo", "this isn\\'t working"])])
fish: Unexpected end of string, quotes are not balanced
echo 'this isn\'"'"'t working'The systems with non-POSIX non-interactive shell are firmly in the "special" category. If a user decided to set their _non_interactive_ shell to fish - they know they are heading for trouble and should not be surprised. I would not worry about such users in my scripts for example.
I found the article hard to follow, but maybe because I was already familiar with the problem and was just skimming. Skip to "Some experiments..." for the actual useful examples.
I disagree with the conclusion, though. I think there should just be more obvious ways to escape the input so one can keep their sanity with nested 'sh -c' invocation. Maybe '${var@Q}' and print '%q' are enough (can't believe I didn't know those existed!)
What is YSH? https://oils.pub/ysh.html
I am writing a quoting module now, but the key point is that it's a powerful enough language to do so. It is more like Python or JS; you don't have to resort to sed to parse and quote strings.
I posted the quote-argv solution above -- in YSH it will likely be:
var argv = :| ls 'arg with space' | # like bash argv=()
ssh example.com $[quote.sh(argv)]
---
quote.sh follows the (subtle) idiom of replacing a single quote ' with
'\''
e.g. "isn't" in POSIX shell is quoted as
'isn'\''t'
'isn' \' 't'
- JSON, which can correctly round trip every Unicode string, without writing your own parsing functions
- JSON8, an optional extension that can round trip every byte string you get from the Unix kernel
ysh-0.29$ eval ls $dir
eval ls $dir
^~~~
[ interactive ]:11: 'eval' requires exactly 1 argument
YSH Doesn't Require Quoting Everywhere - https://www.oilshell.org/blog/2021/04/simple-word-eval.html
The nastiest case is probably `globasciiranges`.
exec* are not "better replacements" of the shell, they are just used for different use cases.
The whole article could be summarized to 3 bullet points:
1) Sanitize your inputs
2) If you want to execute a specific program, exec it after 1), no need for the shell
3) Allow the shell if there is no injection risk
You don't need to handle any quoting with exec*(). You still need to handle options, yes. But under Windows you always have to to handle the quoting yourself and it is more difficult than for the POSIX shell and it is program dependent. Without knowing what program is executed you can't know what quoting syntax you have to use and as such a standard library cannot write a generic interface to pass arguments to another process in a safe way under Windows.
I just felt it sounded like POSIX is particularly bad in that context, while in fact it is better than Windows here. Still, the system() function is a mistake. Use posix_spawn(). (Note: Do not use _spawn*() under Windows. That just concatenates the arguments with a space between and no quoting whatsoever.)
They are entirely different interfaces though. If you'd implemented system() using posix_spawn() it'd be just as bad as system()
If you say you don't make such mistakes: Yeah, but people do. People that write the code that runs on your system.
Force user to always create a wrapper script? that's just extra annoyance and if user is bad at quoting, they'll have the same problems with a script
Disable hooks at all? that's bad functionality regression
Ask for multiple arguments? this makes command-line parsing much more awkward.. I have not seen any good solutions for that.
(The only exception is writing a command wrapper that takes exactly 1 user command, like "timeout" or "xargs".. but those already using argument vector instead of parsing)
Then, you parse that out into a proper argument array and pass it to exec*/posix_spawn.
To correctly escape arbitrary shell syntax, not only do you need to handle the full POSIX syntax (which is quite complex) …
https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V...
… but you must also cover any bugs and undocumented/underspecified extensions implemented by the actual shell providing /bin/sh on every platform and platform version to which your code will be deployed.
That’s not just difficult — it’s impossible, and everyone that has tried has failed, repeatedly. Leading to security bugs, repeatedly.
https://gist.github.com/Zenexer/40d02da5e07f151adeaeeaa11af9...
There’s a reason why we use parameterized queries instead of escaping to prevent SQL injection, and SQL syntax and parsing behavior is far more rigorously specified than the shell.
There's two ways to think of "running a command:"
1. A list of strings containing an executable name (which may or may not be a complete path) and its arguments (think C's const char **argv).
2. A single string which is a space-separated list of arguments, with special characters in arguments (including spaces) requiring quoting to represent correctly.
Conversion between these two forms is non trivial. And the basic problem is that there's a lot of tools which incorrectly convert the former to the latter by just concatenating all of the arguments into a single string and inserting spaces. Part of the problem is that shell script itself makes doing the conversion difficult, but the end effect is that if you have to with commands with inputs that have special characters (including, but not limited to, spaces), you end up just going slowly insane trying to figure out how to get the quoting right to work around the broken tools.
In my experience, the world is so much easier if your own tools just break everything up into the list-of-strings model and you never to try to use an API that requires single-string model.
What GP is referring to is the fact that that solution doesn't work as well on Windows, because the OS's native idea of a command line isn't list-of-strings but rather a single-string, and how that single string is broken up into a list-of-strings is dependent on the application being invoked.
In python you have "shlex.quote" and "shlex.join". In bash, you have "${env@Q}". I've found those to work wonderfully to me - and I did crazy things like quote arguments, embed into shell script, quote script again for ssh, and quote 3rd time to produce executable .sh file.
In other languages.. yeah, you are going to have bad time. Especially on Windows, where I'd just give up and move to WSL.
In particular, failure to mention `printf -v` is horrible. Not only is it better performing than creating a whole process for command substitution, it also avoids the nasty newline problem.
So if you just take away the libcall, people will make their own version by just doing execl() of /bin/sh. If you want this to change, I think you have to ask why do people want to do this in the first place.
And the answer here is basically that because of the unix design philosophy, the shell is immensely useful. There are all these cool, small utilities and tricks you can use in lieu of writing a lot of extra code. On Windows, command-line conventions, filesystem quirks, and escaping gotchas are actually more numerous. It's just that there's almost nothing to call, so you get fewer bugs.
The most practical way to make this class of bugs go away is to make the unix shell less useful.
strace -o tmp.spawnlp -ff python3 -c 'import os; os.spawnlp(os.P_WAIT, "true", "true")'
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fdc03233310) = 225954
wait4(225954, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 225954
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=225954, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
set_robust_list(0x7fdc03233320, 24) = 0
gettid() = 225954
clock_gettime(CLOCK_MONOTONIC, {tv_sec=2458614, tv_nsec=322829153}) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=2458614, tv_nsec=323030718}) = 0
execve("/usr/local/bin/true", ["true"], 0x7ffdc5008458 /* 44 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/bin/true", ["true"], 0x7ffdc5008458 /* 44 vars */) = 0
Good. How do you pipeline commands with these?
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]
p1 = subprocess.Popen(["df"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "/$"], stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()
return p2.communicate()[0]
Python got a lot of things wrong, and it gets worse all the time, but for now spawning subprocesses is one of the things it got right. Although, unlike IIRC Tcl, it doesn't raise an exception by default if one of the commands fails.
Apart from the semantics of the operations, you could of course desire a better notation for them. In Python you could maybe achieve something like
(cmd(["df"]) | ["grep", "/$"]).output()
Shell scripts are much higher in footguns per character than most programming languages.
It is possible for a coder to understand bash so well that he never shoots his own foot off, but it requires more learning hours than the same feat in another language requires, and unless I've also put in the (many) learning hours, I have no way of knowing whether a shell script written by someone I don't know contains security vulnerabilities or fragility when dealing with unusual inputs that will surface in unpredictable circumstances.
The traditional Unix shell might be the most overrated tool on HN.
https://github.com/openjdk/jdk/blob/jdk-26%2B1/src/java.base...
But i would be lying if i said i understood what was going on there. Some googling suggests this was added around 1.7, ie in the early 2010s.
But then, that Rust CVE seems to originate in this work, and this guy claims Java said "won't fix", which suggests it is vulnerable:
https://flatt.tech/research/posts/batbadbut-you-cant-securel...
But there's no link, and i can't find any discussion about it, so i don't know what the actual situation is.
First, define this function:
quote-argv() { printf '%q ' "$@"; }
# (uses subtle vectorization of printf over args)
ssh example.com "$(quote-argv ls 'file with spaces')"
ls: cannot access 'file with spaces': No such file or directory
$ ssh example.com ls 'file with spaces'
ls: cannot access 'file': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'spaces': No such file or directory
They should both only take a SINGLE arg.
It is basically a self-own because spaces are an OPERATOR in shell! (the operator that separates words)
When you concatenate operators and variables, then you are mixing code and data, which is a security problem.
---
As for the exec workaround, I think this is also deficiency of shell. Oils will probably grow an 'invoke' builtin which generalizes 'command' and 'builtin', which are non-orthogonal.
'command true' means "external or builtin" (disabling shell function lookup), but there should be something that means "external only".
$ sh -c "$(quote-argv -echo 'file with spaces')"
sh: 0: Illegal option -h
$ sh -c "$(quote-argv-left -echo 'file with spaces')"
sh: 1: -echo: not found
$ ssh example.com "$(quote-argv-left -dashtest 'file with spaces')"
-dashtest
file with spacesIt is not hidden. It is written down in plain sight:
A complete command line may be specified as command, or it may have additional arguments. If supplied, the arguments will be appended to the command, separated by spaces, before it is sent to the server to be executed.
$ sudo ls 'file with spaces'
ls: cannot access 'file with spaces': No such file or directory
$ ssh example.com ls 'file with spaces'
ls: cannot access 'file': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'spaces': No such file or directory
$ ssh example.com "ls 'file with spaces'"
Accepting a shell string is sometimes OK, but silently joining multiple args is useless, and insecure.
"RTFM" is not a good answer when security is involved.
Tools do have rough edges, if you don't want to learn about them, you will get bitten.
The only thing it adds is insecurity.
If the feature didn’t exist, then it wouldn’t need to be documented, and the world would be better.
Yes it’s in the docs. Yes people who carefully read the docs won’t get bitten. Also yes the design could be improved so people don’t make this mistake even without reading the docs.
Both things can be true. We’re currently only talking about the latter, though.
I'm surprised, as i started this subthread explicitly to contest that the argv join is "hidden".
Bugs can be fixed.
I presume you consider INTERCAL to be a sanely designed programming language.
Edit: The INTERCAL handbook is a great read, and despite being satirical, it is more detailed and qualified than the documentation of some other popular projects.
In fact, later on the man page only mentions a shell in the part that talks about the behavior when no additional arguments are given:
When the user's identity has been accepted by the server, the server either executes the given command in a non-interactive session or, if no command has been specified, logs into the machine and gives the user a normal shell as an interactive session.
A few lines later it gets even more confusing:
The session terminates when the command or shell on the remote machine exits and all X11 and TCP connections have been closed.
"command" means exactly that. Evaluation by shell. With that in mind, the manual page should read less ambiguous to you.
I actually don't have a good source for that, but you can check the execve(2) manpage. If command would refer to the execution of an argument vector, it would have been mentioned in there.
The other meaning of "command" refers to specific programs like those in /bin.
Note this tool is only intended to be another layer in security.
[1] https://github.com/endiangroup/cmdjail [2] https://github.com/endiangroup/cmdjail/blob/main/main.go#L30... [3] https://github.com/endiangroup/cmdjail/blob/main/config.go#L...
It also provides easy access to escape whatever arguments you want to pass:
out = `bash -c #{arg.shellescape}`
It's possible to use bash for both interactive use and scripting. For example, this author claims to use bash as his scripting shell.
But Debian and the popular Debian-derived distributions do not use bash for scripts beginning with "#!/bin/sh", i.e., "shell scripts".
The interactive shell may be bash, but the scripting shell, /bin/sh, is not bash.
https://www.man7.org/linux/man-pages/man1/dash.1.html
https://wiki.ubuntu.com/DashAsBinSh
https://wiki.archlinux.org/title/Dash
https://www.oreilly.com/library/view/shell-scripting-expert/... ^1
https://www.baeldung.com/linux/dash-vs-bash-performance
https://en.wikipedia.org/wiki/Almquist_shell
https://lwn.net/Articles/343924/
https://scriptingosx.com/2020/06/about-bash-zsh-sh-and-dash-... ^2
I use an Almquist shell, not bash, for both interactive use and scripting. I often write scripts interactively. I use the same scripts on Linux and BSD. I restored tabcomplete and the fc builtin to dash so it feels more like the shell from which it was derived: NetBSD sh.
1. "This makes it smaller, lighter and faster than bash."
2. "... this is strong indicator that Apple eventually wants to use dash as the interpreter for sh scripts."
On my system there are 42 scripts in /bin -> /usr/bin (merged) that start with some variant of `#! /bin/bash` and at least two that do `bash -c`, but that's excluding who-knows-how-many scripts that look slightly different or are in other directories.
And keep in mind that on Debian, almost all first-party software is implemented in Perl, with a small minority in Python.
> python2 -c 'import os; os.system("-x")'
> sh -c -x
sh: -c: option requires an argument
But now I'm wondering: how is one supposed to use `which` (or `type`, or `file`) for a program named `-x`, supposing one had it?
command -v -- -x
Same for me. It looks like the POSIX folks accepted the author's suggestion in 2022 and system() in glibc was updated in 2023.
https://sourceware.org/git/?p=glibc.git;a=blobdiff;f=sysdeps...
#include <stdlib.h>
int main(void) {
system("-x");
return 0;
}
> [pid 172293] execve("/bin/sh", ["sh", "-c", "--", "-x"], 0x7ffe221d2f58 /* 76 vars */) = 0
1. https://web.archive.org/web/20201111203646if_/https://www.de...
https://bonedaddy.net/pabs3/log/2014/02/17/pid-preservation-...
This is not a problem on FreeBSD, if the problem is (as the article seems to say) that the documentation fails to warn about the requirement to properly encode arguments passed to the shell.
Here's the FreeBSD man page [1] for system(3):
SECURITY CONSIDERATIONS
The system() function is easily misused in a manner that enables a
malicious user to run arbitrary command, because all meta-characters
supported by sh(1) would be honored. User supplied parameters should
always be carefully santized before they appear in string.
I rarely use it, and almost never in production, but it has its place. Think of it as the eval() of the POSIX world. If you want to build pipelines, or anything a shell has to offer, and do it simply, then system() is for you.
Security-wise, if you are using system() with user input, you are essentially giving shell access to the user, which may or may not be a big deal. If the intended users are people who already have a shell, that's fine maybe even desitable, otherwise, use something else, like exec*().
As for OpenSSH, what is the problem? The "SH" at the end means "shell", it runs shell commands, what did you expect?
Point in case I encountered this week: I was editing a list and wanted to remove duplicates without changing the order of the lines. There's no ready-made program to do that, but this sequence of piped command served the purpose:
cat -n | sort -uk 2 | sort | cut -f 2-
degamad•2d ago