OpenBSD now enforcing no invalid NUL characters in shell scripts

Here's the actual diff:

https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/bin/ksh/shf.c....

And it looks like that covers all parsed parts of the shell script or history file, including heredocs. I get the feeling it's going to break all shar archives with binary files (not that they're particularly common). It will stop NULs being in the script itself, but it won't stop them coming from other sources, e.g.

    $ var=$(printf '\0hello')
    -bash: warning: command substitution: ignored null byte in input
    $ echo $var
    hello

It remains to be seen if this will be adopted by anyone else, or if it'll be another reason to use OpenBSD only as a restricted environment and not as a general computing platform.

> "If there is ONE THING the Unix world needs, it is for bash/ksh/sh to stop diverging further"

> OpenBSD ksh: diverges further

"We are in a post-Postel world" is a great way to put it. This needs to be repeated by everyone working with file formats or accepting untrusted input.

I like the term post-Postel.

There are two reliability constraints that all software faces; security and interoperability. The more lax you are about validation, the more likely interoperability is. "That's weird, I'll just do whatever" is doing SOMETHING, and it's often to the end user's liking. But, you also enter a more and more undefined state inside the software on the other side, and that's where weird things happen. Weird things happening typically manifest as security problems. So the more effort you go to to minimize the possibility of entering a weird state, the more confidence you have that your software is working as specified.

Postel's Law made a lot of sense to me when developing the early Internet. A lot of people were reading imperfect RFCs, and it was nice when your HP server could communicate with a Sun workstation, even though maybe some bit in the TCP header was set wrong. But now? You just gotta get it right and push a hotfix when you realize you messed something up. (Sadly, I don't think it's possible. Middleboxes are getting more and more popular. At work, we make a product where the CLI talks to the server over HTTP/2. We also install Zscaler on every workstation. Zscaler simply blocks HTTP/2. So you can't use our product. Awkward.)

> There appears to be one piece of software which is misinterpreting guidance of this, and trying to depend upon embedded NUL.

Curious what this is

> This was in snapshots for more than 2 months, and only spotted one other program depending on the behaviour (and that test program did not observe that it was therefore depending in incorrect behaviour!!)

Fascinating. I wonder what that program is, and why it depends on the NUL character.

Kudos to OpenBSD!

Similar to the olde-tyme "-o noexec" and "-o nosuid" options for `mount`, there should be easy, no-exceptions ways to blanket ban other types of simply obvious red-flag activity.

Is this going to murder those fancy shell scripts that self-extract a program appended to the tail, which is really just an encoded blob of some kind, presumably compressed, etc.. ???

I was going to check the status of mksh (the Android system shell), but the project page returns:

"Unavailable For Legal Reasons - Sorry, no detailled error message available."

http://www.mirbsd.org/mksh.htm

The Android system shell is now abandoned? This is also in rhel9 basesos.

Related: The installer for iTunes 12.2.1 included a bug which might recursively delete a volume if the path given as input included incorrectly escaped spaces.

On a similar note, I sometimes think about how newline characters are allowed in filenames, and how that can break simple...

    for each $filename in `ls`

loops -- because in many contexts, UNIX treats newlines as a delimiter.

Is there any legitimate use for filenames with newlines?

Side note: tell your startup to switch its “hardware with Ubuntu Linux inside” to BSD. You will have a much more stable and simple platform that can last a long time.

> There appears to be one piece of software which is misinterpreting guidance of this, and trying to depend upon embedded NUL.

Big oof here. Why? How?

> If there is ONE THING the Unix world needs, it is for bash/ksh/sh to stop diverging further by permitting STUPID INPUT that cannot plausibly work in all other shells. We are in a post-Postel world.

Amem

I've always found the fact that zsh copes with NUL characters in variables etc to be really useful. I can see why this approach makes sense for OpenBSD but they can't prevent NULs appearing in certain places like piped input.

Does this break those self-extracting script/tar files? I forget how those are done, I haven't seen one in many years.

Does this break the self extracting tarball trick, where you have a bootstrap shell script with a binary payload appended?

So I can't bury a tarball inside a shell script anymore?

I wish FreeBSD replaced /bin/sh with OpenBSDs.

Wow, they still use CVS...

Great. Now forbid spaces in filenames.

Is this in reference to something? Judging from the comments, NUL bytes in shell scripts are a common occurrence that everybody is celebrating this change as if it were ground breaking.

I mean, it's a good idea, but I wonder what am I missing here. Also what do they mean by post-Postel?

Surprised noone has mentioned the Crowdstrike issue, which was due to NUL characters wasn't it?

  > If there is ONE THING the Unix world needs, it is for bash/ksh/sh to
  > stop diverging further by permitting STUPID INPUT that cannot
  > plausibly work in all other shells.  We are in a post-Postel world.
  > 
  > It remains possible to put arbitrary bytes *AFTER* the parts of the
  > shell script that get parsed & executed (like some Solaris patch files
  > do).  But you can't put arbirary bytes in the middle, ahead of shell
  > script parsed lines, because shells can't jump to arbitrary offsets
  > inside the input file, they go THROUGH all the 'valid shell script
  > text lines' to get there.

  So here it is again, an example of OpenBSD making software behavior saner for all of us.

I don't consider use of all caps over a minor issue to be sane behavior. At best it's immaturity (trying to force your point rather than persuade), and at worst it's an emotional imbalance that effects judgement. That said, it's ksh, on OpenBSD, so I couldn't care less what they do.