Visual programming should start in the debugger

I sympathize a lot with this post. Today, modern IDE's still don't provide automatic ways to inspect program execution and nicely visualize it. Instead, they rely on having you place all the breakpoints, step-step-step and restart everything again if you did not inspect something in time. I don't want to insert print statements that end up clogging the console completely. This workflow also quickly breaks with a lot of logging.

I'm quite a bit into developing more tooling around that, and it's really hard - developers particularly want tools that completely integrate with existing IDE's. Debugger infrastructure is usually not built to be very exstensible or accessible. For my extension for Visual Studio https://d-0.dev/ I had to put serious effort into making sure it all still works correctly with VS breakpoints which is extremely difficult when the debugger expects all of the code to be in the same place all the time.

I see three issues with the nice ideas presented:

1. Incentives.

I’ve made my contributions to debuggers (Firebug mostly), and devs are not willing to pay for it (we had donations, but I’ve seen people create projects in the space) and so tool makers (browser makers in this case) eat the cost and see it as a cost center. Because of this, the debuggers are at the abstraction layer of the tool itself. The tools change and the cost of keeping the debugger in sync (usually well after the feature is out there) is already high enough.

I put hooks into Firebug to make it better for higher level debugging of frameworks. Later I made a paid extension [*]. It paid for a new computer, and spawned many framework specific extensions over time. I pushed Chrome's DevTools team to add some of the same hooks. Sadly, a decade later, many of those framework level extensions don't use what is available. Again, it is a cost center for the framework development, so just enough gets done, no more.

2. Tools are Siloed.

You may have OpenTelementry, etc., but that data is siloed off somewhere. The IDE doesn't have it. But the IDE has extensions, so maybe someone like Microsoft can bridge the gap and put up a sidebar that is a heatmap of the time that that code takes on the CPU.

3. Levels of Abstraction.

When you talk about state and data modeling, not only is it siloed, but the levels of abstraction are plentiful. It is somewhat possible, but I refer you back to 1--Incentives.

[*] https://vimeo.com/showcase/2541003/video/75271334

I'm still shocked how limited our tools are. People in the early 2000s could animate complex 3d scenes in real time with physics and soft body dynamics. But tracing a few variables feels like an herculean effort.

Maybe it's time to flip the game on its head. Bring reactive, lispmachines, responsive UIs.. I don't know.

I've always said so. The essence of visual programming is watching code and runtime values juxtaposed, not connecting code primitives with fancy connectors.

Modern IDEs have lots of visual features, specially tailored to support production languages. Inspection panels, tooltips with variable values and function definitions...

Surely we could add more spatial visualization tools to them. But the step up from printf to a debugging inspector panel was on its own a great advance in building a mental model of the program state and making sense of its behaviour, which is what visual programming is all about.

For programmers interested in advanced visual tools tailored to software creation, see Bret Victor's essays on notations:

https://worrydream.com/MediaForThinkingTheUnthinkable/

https://worrydream.com/LadderOfAbstraction/

If you search for the keywords “program understanding” and “program visualization” in Google Scholar, you’ll see a lot of research on how to visualize and comprehend systems. Around 2007, one visualization that caught the internet’s attention was CodeCities by M. Lanza, and there is also Moose[1], which is based on Pharo and provides abstractions to query and visualize code bases.

But, for some reason, none of these tools catch the attention of the most popular IDEs. I hypothesize that there are major roadblocks to implementing this in a generic and useful way.

The first one is the variety of languages, frameworks, and build tools. For example, analyzing a TypeScript+React code base is not the same as analyzing a TypeScript+Vue code base, even when both use TypeScript and the TSC API is very easy to use.

The second roadblock is that useful visualizations also depend on the characteristics of your system, and creating them is not easy.

Maybe things will change with the addition of AI to analyze code bases, but so far, all the tools I’ve seen are either very niche or very short-lived.

[1]: https://moosetechnology.org/

Having actually partially designed and maintained a visual programming product with I guess what would be called a time travel debugger (among many other fascinating and strange features) inside of a major technology company, the biggest push back was always from developers themselves. These things sound good, and often they're very useful to those without strong programming backgrounds, but inevitably developers are pretty aggressive about poo-pooing tools like this as lots of things are much harder to implement, like diffing for example. The tool itself was honestly very radical (we were able to do a ton of really cutting edge things around resource scheduling, containerization pre-Docker, real actual shared time travel debugging at massive scale), but in the end, I think the company shitcanned it for something much simpler.

Agreed. Visual programming focuses on visualising the structure of the _program_, but what matters more is visualising the structure of the _data_ and, even more importantly, what the program is doing to the data (visualising in the time domain).

A major bottleneck for people learning programming is developing a mental model of what is in the memory during the runtime and what it looks like. The most successful introductions to programming start by making drawings or small games since it's more intuitive for beginners to iterate and have immediate feedback.

Screenshot at the end of the article is from this video. He calls it spatial programming and mentions some other tools which did that in the start of the video.

https://www.youtube.com/watch?v=eQgxFuw8f1U

I developed Call Stacking as a visual debugger.

It captures everything for a given request - each method call, params/argument/return values, class name/method/line of execution.

You can visibly see which method called which, as each method call is nested relative to its parent callee, in a single consolidated timeline.

https://callstacking.com/

Similar post from a few days ago:

"We need visual programming. No, not like that." https://news.ycombinator.com/item?id=40937119

Shoutout to https://whitebox.systems/

It only really works with C-with-Classes style C++. But, the fact that it works at all is pretty neat.

People talk about using LLMs to write code, but is there any work on using RNNS (perhaps LLMs) to explain what a piece of code does? Seems like that might be a good feature for a debugger.

Yeah. Debugging is getting _way_ less attention than it should. I know some people who _boast_ that they don't use a debugger. To me it feels like a doctor saying: "We don't use new-fangled X-Rays here, you should be able to find problems with just a stethoscope".

And good debugging support doesn't need to be some fancy "Minority Report" style 4D animated UI. Just give us good type rendering support, an easy ability to chain breakpoints and set conditional breakpoints, "watch variable" support. Some other things: "where did this value came from?", "watch for value use".

Go is an especially bad example. Its debugger doesn't even allow custom type rendering, so if you have something like a custom generic map container, you can't inspect it at the debug time.

Paper and pen. Sure might not work on LARGE projects, but when I’m reverse engineering something or trying to debug, I always start with a control flow diagram… works a treat!

But in the past 6 months I’ve started to use graphviz and other tools to quickly document code paths for others

I had a similar idea. I was working with some complex recursive functions and it was impossible to figure out what was going on. Trying to parse through 1000 console logs was nearly impossible so I had the idea of rendering the function call stack and attaching the logs to where in the stack it was called from and rendering it as an interactive graph.

Its mostly just a demo but I think something like this a standard tool would be really helpful. https://github.com/ando818/Svelte-Logger

The debugger? You mean console.log("here") ?

The article is missing all industrial visual programming.