Where should visual programming go?

When this last round of discussion of visual programming happened I had a minor epiphany.

For a little background, I’ve maintained that visual programming goes absolutely nowhere until we have visual diffs that work (work as in workflow). I’ve been saying that since before the UML Trough of Disillusionment kicked into high gear. Without diffs, without analysis, we are going nowhere fast. Almost every other link of the lifecycle is still intact with visual code but focusing on an editor without diffs breaks the chain. There’s no producing and maintaining commercial grade software without it. It’s either toy programs, or absolutely heroic effort, and what else could you have done with that much energy?

The epiphany was this: motion detection has been a feature of digital video since at least the MPEG days. Someone with a background in video compression needs to tackle the problem of doffing visual code. Figuring out how the code should look (mechanical sympathy) to facilitate this.

I've been experimenting with a project I'm calling guish, a bidirectional command line editor for writing bash pipelines with GUI modules. The best way to explain it is that text written in the CLI prompt updates the GUI and changes to the GUI updates the CLI prompt in turn. It operates by working on the bash AST and has step-by-step outputs so you can see the result of each step in the pipeline's output.

Plenty of screenshots, a demo video, and more at https://github.com/williamcotton/guish

I'd love feedback, positive or negative!

I don't believe in visual programming ever replacing code for general programming. However, many examples exist when visual programs work for narrow domain-specific applications. This works because such applications allow exposing only high-level domain-specific abstractions. This reduces complexity enough to be a good fit for visual representation.

Non-programmer here. I’ve long desired to program Arduino microcontrollers but haven’t had the ability and/or the patience to learn programming. I’m a very visual person and get confused if I have to keep more than a few lines of code in my head. Even if I understand how I want a signal to be processed into an action, when the flow chart in my head needs to be translated into text I get overwhelmed.

A solution arrived for me in the form of a number of visual arduino IDEs. Flprog, Outseal, and Open PLC. They all allow programming in ladder logic, and Flprog and Open PLC allows for function block programming. Ladder logic is useful for many tasks involving discrete I/O (Ex. A button latches a contact which turns on a relay which turns off a light and turns on a motor.) Function block programming has enabled me to perform more complicated functions like creating an HMI using a menu system displayed on an LCD connected via I2C. These are tasks that I wouldn’t have a chance of accomplishing without the complex arduino code being contained in graphical functions blocks which I can wire together and configure with a gui.

Prior to compilation, Flprog displays the raw arduino code in the arduino IDE, which gives me a chance to exam the actual code produced by my function block program. I find it to be very educational.

As a non-programmer, I have no idea if this embedded programming use case can be generalized to more complex programming tasks, but for my limited use case it’s been transformational.

not a well thought out take, but i think visual programming should stop trying to encode application logic and focus on:

- building good UIs rapidly (e.g. improving & simplifying layout options, look & feel, etc.)

- make debugging trivial & powerful

- make deployment trivial

Visual Basic was on the right track in almost every way except as a language. Hypercard maybe had the language thing figured out: use english as a base?

Hooking up loops and conditionals visually is a waste of type, people can type it out with a little instruction with a good IDE & code completion that caters to non-coders. Focus on the tooling around the code & the things visual development has proven good for (laying out UIs, debugging) and then the stuff that people don't want to think about (deploy)

> Sebastian argues that we shouldn’t try to replace all code with visual programming but instead only add graphics where it makes sense

I just interviewed Daniel Kallin, the maker of the language/tool Nomnoml (https://www.nomnoml.com/), for making beautiful diagrams, which is one such place where it makes sense [0].

He had an expression I liked, wanting to make a tool/lang that felt like "drawing with ascii".

I think this is the way for visual programming. You must do nice DSLs side by side with the visual representation. The text must be source of truth. It _has_ to be that way! That's how computers work!

---

Don't let the hard to pronounce name throw you off, Nomnoml is a super fast and handy tool for making beautiful visualizations, hand crafted over 10 years and with no ads or login and fully open source.

[0] interview: https://pldb.io/blog/danielKallin.html

For anyone interested in working on visual programming professionally, we use it for creation in Rec Room in a system I built called Circuits https://blog.recroom.com/posts/2021/5/03/the-circuits-handbo...

It has a real place among novice programmers. We even have some experts who use it as a fun alternative to writing text. I don't see visual systems as an effective way to replace everything us experts are doing but they've gotten a ton of mileage in the jr. and learning domain. Scratch (https://scratch.mit.edu/) being another obvious example.

See the email in my profile if you are interested in roles and I'll see if we can find something that fits.

It's interesting to see how different programming languages approach building software.

Languages like Lisp are all about planning out the whole program structure (the "Abstract Syntax Tree") upfront, kind of like a blueprint. But modern tools and visual programming often take a more piece-by-piece approach. You build smaller parts first and then connect them together, gradually building up the whole thing.

This bottom-up method feels more natural to how we solve problems in real life. Visual tools make it even easier because you can just drag and drop components instead of worrying about complex code structures.

I wonder if this kind of visual, incremental approach could make programming more accessible to everyone. It might help bridge the gap between our everyday thinking and the formal world of code.

Isn't this how some UI frameworks work(ed)? You had graphical design tool that then generated some blob (or xml) that you then included in your application.

More extreme example would be the various business process etc things, like KIE/Drools/jBPM, or BPEL/BPMN, that also allow various ways of mixing various visual and traditional practices.

I think the common thing with all of these is that in the end no developer actually liked them.

Now thinking of this, one of the problems is that there is no suitable generic graphics format that could be used as a base for these sort of visual programming ideas. So there isn't any text-editor analog for visual programming currently, instead all visual programming environments I've seen use some specialized custom tools and formats.

Recent and related:

We need visual programming. No, not like that - https://news.ycombinator.com/item?id=40937119 - July 2024 (494 comments)

Gamify and functionalize it, make it factorio but with application controll as the end goal.

Puckishly: it need go nowhere at all.

As a matter of fact, we _do_ have something that is more 'visual' or 'symbolic' than written speech, and it has a critical role in certain abstract forms of writing. It's mathematical notation (or, sometimes, symbolic logic) and it's already great.

Thus, any sufficiently advanced visual programming environment is indistinguishable from rendered LaTeX.

Visual Programming is fine for very specific cases: Graph-based connection of inputs and outputs. It was used for DirectShow GraphEdit and ComfyUI, and works find there. Just please provide a way to also type things in manually as well.

Reminds me of this: http://mbeddr.com/

Not sure how much it is still being used at Itemis, at least is is still maintained.

Visual programming has a long history. See Nassi-Shneiderman-Diagram's, which are even a German (Now eu) DIN standard. And you have Drakon from the Russian space program.

I've wanted the ability to embed diagrams in code for sooooo long. How is this basic and obviously useful feature still not possible??

Wow, that mouse pointer effect is obnoxious.

Please check this out

http://www.codersnotes.com/notes/a-constructive-look-at-temp...

Jump to the section titled "HyperText" (DolDoc)

Dataflow, and finite state machines.

Have a look at Ballerina,

https://ballerina.io/

Yeah but it turns so niche you write visual 3D interfaces in Unity for your specific application. Terminal editor improvements are visual programming.work iterm2 I can see command history and dig into problems.

There’s a huge advantage to see programming for non programmers. Do you know how hard it is to explain hypervisors, VMMs and OverlayFS to non programmers? I gave them a simple example they need to make a simple text update to a Photoshop file. It costs a renewed font license for the latest but essentially an application library they can open the legal version they bought on windows see it look sort of okay looking on osx make changes to the file and be done with it.

One thing ChatGPT taught me was people don’t understand code. SQL was supposed to be ChatGPT of the day. Executives had no concept the “power” of essentially an NLP on top of structured data.

And for developers a high level AutoCad like view of what’s talking to what, etc. is valuable. I don’t need to visualize a merge sort but maybe a realistic abstract demo non programmers and programmers can work against.

I got tired of corrupt paste programmers mocking up design in Figma. I want to show you acquired Product X on SAP and it’s a different version. Using token data I want to be like as of right now you see realistic data and users, zoom in, show it merge in a datalake and produce an auditable report.

And that it runs strategies so algorithms (probably AI) can show aggregated date and where things might be failing. Zoom out see two icons from two company’s on different software and a generic data lake, zoom in and see and update data. See a message in Skack/email/etc. indicating whatever. Show elementary that consultant leads products like Twilio run off AWS and telemetry shows they don’t use but one feature and we can swap it out. All ok a running system without awkwardly switching windows. It helps executives get the big picture,’itnactualu works and you drill down enough you see type forwarding and an IDL for the programmers.

When was the last time you used merge sort not in std library or visualization would have helped you solve it? And yes I’m not talking just about corporate software.

Just the other day I was wanting to show an overlayfs/squashfs. I mean I’ll go down to C or god help me ASM so it’s not technical ability. It’s usually not even big vs little indian that trips people up but the big picture.

I wish so bad things like Salesforce would release their code, it is very rare someone would “steal” it and companies I know would benefit greatly and still pay full cost if I could rip out things and see their documentation is wrong and not have to go to COBOL or disassemble to me that’s visual programming. “INT32 actually a mutable type oh yeah they also made up their own types. That’s in red. In fact here’s a point map of all types that don’t match specs. First question why (in 1994 a driver didn’t work right so we did some weird type forwarding) ok… but in huge systems I’m not sure visual programming would help. Maybe weird things like btree in rust is better until you make an roc call outside of rust for some reason?

Offtopic: since someone is going to notice the timestamp discrepancies in this thread - see https://news.ycombinator.com/item?id=41080646 for how this submission came to be.