What Color is Your Function? (2015)

I've always found the criticism leveled by the colored functions blog post a bit contrived. Yes, when you replace the words async/await with meaningless concepts I do not care about such as color, it's very annoying to have to arbitrarily mark a function as blue or red. But when you're honest with yourself and interpret the word "async" as "expensive", or as "does network calls", it becomes clear that "async/await" makes important features of your function explicit, rather than implicit. Seeing "await" gives me more information about the function, without having to read the body of the function (and the ones it calls, and the ones they call, etc). That's a good thing.

There are serious drawbacks to async/await, and the red/blue blog post manages to list none of them.

I've wrote a blog in response to OP containing a more detailed version of the above:

https://wpbindt.github.io/async/opinions/programming/2024/01...

Ah my pet peeve reaching the front page again…

The “coloring problem” has nothing to do with async at all, it's just about making effects explicit or implicit in the code.

In Go or Rust, returning an error is also a “color” that spreads out to the top of the call stack. Same for checked exception in Java. Unchecked exceptions are like blocking functions, it's invisible so you don't have to manually annotate your functions but as a result any function can now fail without the programmer knowing it.

JavaScript having async/await but unchecked exception may sound bit paradoxical in that regard, but every language has a combinations of both explicit and implicit effects: Rust has both async/await and explicit errors, but it also has panics (implicit errors) and implicit allocation for instance.

I personally live explicit effects, but at the same time when you have too many of them (for example I'd like to have effects like “pure”, “allocating” and “panics”) they start to cause combinatorial explosion of cases in libraries accepting closures as parameter.

Algebraic effects supposedly solve this particular problem but I'm not sure the additional conceptual learning effort is sustainable for a mainstream language, so in the meantime I think every language must keep its amount of explicit effects under a certain threshold.

I've spend a lot of time writing things using async in rust, python, and typescript, and I still find it un-intuitive / conceptually incorrect.

Once you're in an async function the rules then fundamentally change for how everything operates; it's almost like programming within a dialect of the same language. In particular, I'm referring to everything from function calling, managing concurrency, waiting on results, sleeping threads.

Comparatively, when you're in a go-block in go, you're still writing within the same dialect as outside of it.

I've thought about this a lot in relation to typescript over the years and had various opinions -- For some time I thought it'd be better if there was an implicit `await` on every line and require `void` or some other keyword to break execution like `go` in Golang.

But, eventually I realized the difference in pre-emption between languages -- Go can (now) preempt your code in many places, so locks and thread-safety are very important.

The javascript runtime only preempts at certain places, `await` being one. This means I can know no other code can be running without explicit locks around all critical sections.

Finally understanding the trade-offs, I no longer am as frustrated when recoloring a bunch of functions. Instead, I can appreciate the areas where I'm not required to lock certain operations that I would in other languages.

My web scripting language, https://hyperscript.org, tries to hide the difference between sync and async by resolving promises in the runtime. My theory is that this is something that web script writers should not be concerned with (this theory makes more sense when you consider hyperscript is a companion to htmx, and favors a particular approach to scripting[1].)

  on click
    fetch /whatever as json
    put the result's data into #some-div
    wait 2s
    put '' into #some-div

You don't have to mark the script as sync or async, and the runtime will resolve everything for you, making everything feel like synchronous scripting.

This obviously has limitations and foot guns, but it works reasonably well for light scripting. More info here:

https://hyperscript.org/docs/#async

And the implementation of it in the runtime here:

https://github.com/bigskysoftware/_hyperscript/blob/c81b07ce...

https://github.com/bigskysoftware/_hyperscript/blob/c81b07ce...

--

[1] - https://htmx.org/essays/hypermedia-friendly-scripting/

This is one of the things I love most about Lua. You can coroutine.yield inside of any function, it's up the the caller to handle it correctly.

This means I can write my tech stack to run in either mode, then swap out async/sync functions in the application. This is exactly what I do in https://Lua.civboot.org#Package_lap

I've spent some time thinking about language design recently. Coloured functions have a surprising number of downsides:

- They're sometimes much too explicit. When writing a complicated generator, I don't necessarily want to annotate every call to a sub-generator with `yield*`, especially if I need to drill that annotation through wrapper functions which aren't really generators themselves.

- Colours show up everywhere. If you have several functions with a `context: GodObject` parameter, then that's a function colour. Most third-party code will be unable to forward a `context` argument to a callback, so you'll have to manually smuggle it in using closures instead.

- Different "colour channels" don't compose nicely with one another. Even though JavaScript ES2017 provided both async functions and generators, ES2018 had to add multiple new pieces of syntax to permit async generators.

- It's normally impossible to write code which is generic over a function's colours. For example, if you have `function callTwice(f)` in JavaScript, you'd need a separate `async function callTwiceAsync(f)`, and a `function* iterateTwice(iterable)`, and an `async function* iterateTwiceAsync(iterable)`, despite the fact that all of those functions are doing the same thing.

Several small languages are experimenting with algebraic effect systems [1], which would make all functions colourless. If JavaScript had this feature, the syntax for defining and calling functions would be the same, no matter whether you're dealing with a normal function, a generator, or an async function. No more `async`, `await`, `function*`, `yield`, `yield*`, `for-of`, `for await`...

This can make everything too implicit and vague. The state of the art is for algebraic effect systems to be implemented in a statically-typed language, which then uses type inference to quietly tag all function types with their colours. This means that the compiler can enforce requirements like "you can only call generators within a scope which is prepared to collect their results", but that rule wouldn't prevent you from wrapping a generator in `callTwice`.

[1]: https://github.com/ocaml-multicore/ocaml-effects-tutorial

A slightly separate but still related question that bothers me every time I, a Python programmer, read about function coloring and different approaches to concurrency:

Does anyone understand why asyncio-style approach to async won over gevent-style in Python? Why was the former approach accepted into to stdlib, got special syntax and is preferred by the community while the latter is a fringe niche thing?

It is similar to IO in Haskell which also force you to explicitly declare it in the signature (and in turn force callers to be IO), where most other languages (even functional) allow you to do IO anywhere.

There is also coloured data: stack vs heap, threadsafe vs not, disk vs memory vs SQL, mutable vs immutable, etc ...

Another comment: Python's async/await is precisely syntactic sugar for generators. It just changes the keywords and checks that you only use await in a function labeled async.

Edit: my rate limit appears to have been increased.

I think the main issue with async/await specifically is that it is abstracted away from most developers.

I may have a sync program running a main loop and I want to have a subset of that program using something like the reactor pattern. In any multithreading aware programming language I can do that by just sticking that in another thread and syncing however I want, even using coroutines I can do that, it doesn't need to follow the reactor pattern.

However libraries (tokio) and languages (js) don't make that the obvious default. In something like JS it isn't even possible, the entire language is built on top of coroutines. For tokio the default is "put tokio main here and voila async" and you need to actually understand what it's doing under the hood to know that that isn't the behaviour you want in all cases.

This is more bottom-up vs top-down learning. Most people learn and teach top-down because it gets you to productive really quickly. Bottom-up is much harder and has the chance of getting lost in the weeds but makes what is happening very obvious.

Does tokio use reactor or coroutines? It depends...

This is how we got "fibers" in PHP and they are absolutely worthless. With async/await, I can choose or choose not to wait for the result. Or I can trigger the work asynchronously and wait on it later, in an entirely different function. With Fibers, you cannot choose. You must wait, and you must wait now.

Do any relatively-popular languages avoid “two colors” by essentially making everything async?

This article gets referenced a lot, but it does a poor job of defining what color actually is.

The biggest issue the article describes — inability for sync code to wait for an async result — is limited mostly to JavaScript, and doesn't exist in most other languages that have async and a blocking wait (they can call red functions from blue functions).

If this architectural hurdle is meant to be the color, then lots of languages with async functions don't have it. This reduces the coloring problem down to a minor issue of whether async needs to be used with a special syntax or not, and that's not such a big deal, and some may prefer async to be explicit anyway.

Related. Others?

Ruby methods are colorless - https://news.ycombinator.com/item?id=41001951 - July 2024 (235 comments)

On 'function coloring' (2018) - https://news.ycombinator.com/item?id=36199499 - June 2023 (21 comments)

I Believe Zig Has Function Colors - https://news.ycombinator.com/item?id=30965805 - April 2022 (158 comments)

In Defense of Async: Function Colors Are Rusty - https://news.ycombinator.com/item?id=29793428 - Jan 2022 (101 comments)

The Function Colour Myth, Or: async/await is not what you think it is - https://news.ycombinator.com/item?id=28904863 - Oct 2021 (7 comments)

What color is your function? (2015) - https://news.ycombinator.com/item?id=28657358 - Sept 2021 (58 comments)

What Color Is Your Function? (2015) - https://news.ycombinator.com/item?id=23218782 - May 2020 (85 comments)

What Color is Your Function? (2015) - https://news.ycombinator.com/item?id=16732948 - April 2018 (45 comments)

The Function Colour Myth (or async/await is not what you think it is) - https://news.ycombinator.com/item?id=12300441 - Aug 2016 (4 comments)

What Color Is Your Function? - https://news.ycombinator.com/item?id=8984648 - Feb 2015 (146 comments)

Wouldn't adding a way to block on an async function make javascript colorless?

No one would want that though right? Like in what context would you even have a consequenceless block? A CLI maybe?

Maybe Chomsky was on to something…

Colorless green functions async/await furiously.

I still haven't seen a good "solution" to this "problem" for imperative style programming. Why the quotes? Because the functions have different signatures, thus different functions(colors).

And IT MATTERS. That's why you have structured concurrency hot on the heels of Loom. And once you are using structured concurrency constructs... Ehh, it's not the magic "solution" we were sold anymore is it?