Why Oxide Chose Illumos

{{citation needed}}?

When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU.

Here's a talk I gave about it that includes that analysis:

slides - https://kvm-forum.qemu.org/2019/kvmforum19-bloat.pdf

video - https://youtu.be/5TY7m1AneRY?si=Sj0DFpRav7PAzQ0Y

Even if you think it's a foregone conclusion given the history of bcantrill and other founders of Oxide, there absolutely is value in putting decision to paper and trying to provide a rational because then it can be challenged.

The company I co-founded does an RFD process as well and even if there is 99% chance that we're going to use the thing we've always used, if you're a serious person, the act of expressing it is useful and sometimes you even change your own mind thanks to the process.

Bryan Cantrill is CTO of Oxide [1].

I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion.

[1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

Given all of that, and taking into account building a product on top of it, and thus needing to support it and stand behind it, Linux wasn't the best choice. Looking ahead (in terms of decades) and not just shipping a product now, it was found that an alternate ecosystem existed to support that.

Culture of the community, design principles, maintainability are all things to consider beyond just "is it popular".

Exciting times in computing once again!

  1. Xen Type-1 hypervisor is smaller than KVM/QEMU.
  2. Xen "dom0" = Linux/FreeBSD/OpenSolaris. KVM/bhyve also need host OS.
  3. AMZN KVM-subset: x86 cpu/mem virt, blk/net via Arm Nitro hardware.
  4. bhyve is Type-2.
  5. Xen has Type-2 (uXen).
  6. Xen dom0/host can be disaggregated (Hyperlaunch), unlike KVM.
  7. pKVM (Arm/Android) is smaller than KVM/Xen.

Is this an argument for Illumos over Linux, or for translating SMF to Linux?

curious about what bugs are being thought of there. Sounds like a very interesting situation to be in

It's a small operation, but https://openbsd.amsterdam/ have absolutely proven that OpenBSD's hypervisor is production-capable in terms of stability - but there are indeed other problems that rule against it on scale.

For those who are unfamiliar with OpenBSD: the primary caveat is that its hypervisor can so far only provide guests with a single CPU core.

If I were Oxide, though, I’d be sprinting to seamless VMWare support. Broadcom has turned into a modern-day Oracle (but dumber??) and many customers will migrate in the next two years. Even if those legacy VMs aren’t “hyperscale”, there’s going to be lots of budget devoted to moving off VMWare.

Linux has a massive advantage where it comes to hardware support for all kinds of esoteric devices. If you don't need that, and you've got engineers that are capable of patching the OS to support your hardware, yep, have at it. Good call.

* https://www.youtube.com/watch?v=UvEKSqBBcZw

Certainly they already had experience with ZFS (as it is built into Illumos/Solaris), but as it was told to them by someone they trusted who ran a lot of Ceph: "Ceph is operated, not shipped [like ZFS]".

There's more care-and-feeding required for it, and they probably don't want that as they want to treat product in a more appliance/toaster-like fashion.

https://oxide.computer/podcasts/oxide-and-friends/2065190

> To mitigate all this, we’re intending to stick with the OSS build, which includes no CCL code.

    [0] https://news.ycombinator.com/item?id=41256222
    [1] https://rfd.shared.oxide.computer/rfd/0110

It is so sad that we've ended up with designs where this is the case. There is no intrinsic reason why nested virtualization should be hard to implement or should perform poorly. Path dependence strikes again.

That's with virtio, the virtual intel "card" is even slower.

They went with Illumos though, so curious if the poor performance is a FreeBSD-specific thing.

I know NetApp (stack based on FreeBSD) contributed significantly to Bhyve when they were exploring options to virtualize Data ONTAP (C mode)

https://forums.freebsd.org/threads/bhyve-the-freebsd-hypervi...

The section about Rust as a first class citizen seems to contain references to its potential use in Linux that are a few years out of date; with nothing more current than 2021.

> As of March 2021, work on a prototype for writing Linux drivers in Rust is happening in the linux-next tree.

Bryan Cantrill, ex-Sun dev, ex-Joyent CTO, now CTO of Oxide, is the reason they chose Illumos. Oxide is primarily an attempt to give Solaris (albeit Rustified) a second life, similar to Joyent before. The company even cites Sun co-founder Scott McNealy for its principles:

https://oxide.computer/principles

>"Kick butt, have fun, don't cheat, love our customers and change computing forever."

>If this sounds familiar, it's because it's essentially Scott McNealy's coda for Sun Microsystems.

Frankly I don't understand why they blogged that at all. It reeks of desperation, like they feel they need to defend their choice. They don't.

It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details.