Pipes: A spiritual successor to Yahoo Pipes

This is my project :) Nice to see this here again! If there are questions, I'm ready to answer (I have a rather young baby here though that wanted attention while writing this, but she sleeps now, so I think we are good).

Pipes saw some internal updates recently. It was fighting with some instability. My first measure was an internal re-architeture, on a small scale. Before, the data was transported as text from block to block. (Almost) Every block parsed the input, created an RSS feed object and then worked with that on the data, only to then output the created RSS feed as string. This was changed to just work with the RSS object directly and move that from block to block. That had some consequences for some existing pipes, mainly because it also changed when the input feeds gets normalized.

As that did not help, next step was a server upgrade. I'm not sure whether it was directly the additional processing capacity or the dependency upgrades, but that one helped so far. Knock on wood. Also reconfigured the amount of threads and puma workers, that might have solved a bottleneck somewhere. I sadly never pinpointed exactly what made the server process stop before.

I had played with a proper split between web frontend and pipes processing, but at least my approach was not viable. Worked nicely while developing it, immediately crashed with the production workload. And Pipes is not all that huge... I'll have to try again some time.

I see a lot of folks here asking about what happened to the original Yahoo Pipes.

We had the same question, so we went and talked with a lot of the original team and wrote up the story[0]. We also made a fun mini-site that contains a lot of easter eggs (e.g. if you click on the "Memory Pipes" folder on the desktop, you'll see a bunch of candid photos of the original team circa 2007)

[0] https://retool.com/pipes

Related:

The architecture and software behind Pipes - https://news.ycombinator.com/item?id=22959013 - April 2020 (12 comments)

Show HN: Early-stage Yahoo Pipes spiritual successor - https://news.ycombinator.com/item?id=14440993 - May 2017 (126 comments)

n8n is a similar tool and can be self-hosted: https://n8n.io

And NodeRED is big in the home automation space: https://nodered.org

Finally! Yahoo Pipes was so ahead of its time. The whole concept of microdata, RSS and wrangling data around in pipes was amazing and it is sad that it died. The web was so promising at that time.

I have no experience with this particular implementation but I do have experience with some important business process developed in Azure Logic Apps, which is kind of the same in spirit I believe.

From what I have seen, these kind of frameworks facilitates the worst possible combination of factors: A solution designed, created and implemented by non-programmers, not using a programming language ending up in a production environment.

One could argue this is a success in its own but I have only seen these kind of things hit their limits almost immediately after the initial POC and evolve into terrible tech debt. Just learn to program or script already.

That being said, I do not want to disrespect the effort into building this and perhaps there is a place for these kind of solutions I yet have to experience.

At work, we use extensively Palantir Foundry and its [excellent] PipelineBuilder. I was reluctant at first with this low-code no-code stuff. But after porting custom python code, I must admit it covers 99% of the transformation needs we have. Plus it keeps a semantic all along, that is much more difficult to understand/maintain by code [especially in Python].

Can Zapier and similar tools be considered a spiritual successor to Yahoo Pipes too?

Interesting to see the evolution of Visual programming across many tools. We are working on something similar[0] for low-code/no-code devs focusing on creation of APIs, backend tasks, and AI workflows.

[0] https://buildship.com

Back in the day I had hundreds of pipes, sub-pipes for all kinds of stuff from porn to price gathering, comparisons, to banking. When Yahoo Pipes died so did my will to port all that to something else.

If I had to do it all again in 2024, what’s a robust self hosted project like this?

every form of visual programming I've ever seen is a singular style of some of a box that does processing and a line that connects that box to another box.

This is the code for a YC startup I ran in this space. Very similar. Build on AWS Lambda: https://github.com/refinery-labs/refinery

I need to fix the docs because they are dead: https://web.archive.org/web/20200819221527/https://docs.refi...

Yahoo Pipes was an inspiration though. It was similar to Heroku in many ways.

It is on GitHub: https://github.com/pipes-digital/pipes.

> The default output format of a pipe is RSS.

Do people still use RSS feeds?

I work on a similar product[0], conceptually at least, to Pipes, and many of our users have asked for RSS feeds as output.

Until now I’ve rationalized that as low priority, but maybe I’m wrong?

[0] https://magicloops.dev

Very cool project. A great demonstration that when you're useful you don't need a fancy interface!

If anyone is interested in a more generic visual-programming language, check out Flyde - https://www.flyde.dev - an open-source visual-programming language I'm working on

tiny bit of feedback : I tried to sign-in with a non gmail address and did not receive the confirmation email. Switched to a gmail address and everything was smooth in the sign-in process.

I wonder if we could pull off some wackiness with custom RSS feeds. Might have some utility in keeping an LLM constantly up to date as well.

Yahoo pipes was amazing!

My vague memory of how this ended last time was processing cost and lack of a business model.

You know, there is also Apache Nifi.

rss-funnel[0] is a self-hosted feed filtering pipeline with similar goal.

[0]: https://github.com/shouya/rss-funnel

disclaimer: rss-funnel is my own project.

Yahoo pipes was awesome. Way ahead of its time. I don’t know why yahoo shut it down.

I want to support this idea, though If you use their service it's free to play around with 3 feeds, but then it gets more pricey. Maybe worth it for some?https://www.pipes.digital/pricing

kinda related but does anyone know anything similar to yql, in the sense that you can query everything using the same syntax?

the twitter example link goes nowhere.

I think this has some value potentially now, but they should not use “pipes” to describe it. Yahoo pipes is dead and there is no point associating one self with that brand anymore. The concept is easy enough to explain without using that word.