June 26th, 2024

Show HN: a Rust lib to trigger actions based on your screen activity (with LLMs)

The GitHub project "Screen Pipe" uses Large Language Models to convert screen content into actions. Implemented in Rust + WASM, inspired by `adept.ai`, `rewind.ai`, and `Apple Shortcut`. Open source under MIT license.

Read original articleLink Icon
Show HN: a Rust lib to trigger actions based on your screen activity (with LLMs)

The GitHub project "Screen Pipe" focuses on utilizing Large Language Models to turn screen content into actions. It is implemented in Rust + WASM and draws inspiration from various sources like `adept.ai`, `rewind.ai`, and `Apple Shortcut`. The tool enables screen capture and text extraction, showcasing a TypeScript server-side example using a Large Language Model for analyzing sales conversations. Currently in the Alpha stage, the project operates on the creator's computer, capturing screen elements to trigger actions. To test the tool, users can clone the repository and follow the instructions in the README. The project is open source under the MIT license, welcoming contributions through forks and pull requests. Additionally, a public Discord channel is available for community engagement and collaboration on related projects. More details and access to the project can be found on the GitHub repository for "Screen Pipe."

Related

AI-powered conversion from Enzyme to React Testing Library

AI-powered conversion from Enzyme to React Testing Library

Slack engineers transitioned from Enzyme to React Testing Library due to React 18 compatibility issues. They used AST transformations and LLMs for automated conversion, achieving an 80% success rate.

React Lua

React Lua

The GitHub repository focuses on translating ReactJS 17.x into Lua for Roblox and Lua community. It aims for performance and accuracy, welcoming contributions from notable developers like Paul Doyle and Matt Hargett.

Sloth search for Ruby Weekly – a 100 minute hack turned 20h open sauce project

Sloth search for Ruby Weekly – a 100 minute hack turned 20h open sauce project

Sloth Finder, a Ruby and Rails tool, curates niche articles on API and automation. It emphasizes simplicity, slow loading times, and plans to upgrade its tech stack for efficiency. Open source on GitHub.

Update on GNOME Newton, the Wayland-native accessibility project

Update on GNOME Newton, the Wayland-native accessibility project

The Newton project aims to enhance accessibility on free desktops, focusing on Wayland-protocols, AccessKit, Mutter, Orca, GTK, and libadwaita components. Progress includes running GTK 4 apps in Flatpak sandboxes, with ongoing work to improve Orca functionality on Wayland. Challenges involve optimizing performance and addressing architectural issues for assistive technologies. Future plans may extend support to GNOME Shell UI.

My weekend project turned into a 3 years journey

My weekend project turned into a 3 years journey

Anthony's note-taking app journey spans 3 years, evolving from a secure Markdown tool to a complex Electron/React project with code execution capabilities. Facing challenges in store publishing, he prioritizes user feedback and simplicity, opting for a custom online deployment solution.

Link Icon 6 comments
By @ay - 5 months
FWIW, I have been working on a variation that is supposed to be eventually on a similar topic, but it is confined to a specific terminal - https://github.com/ayourtch/prompter

For the time being it doesn’t do much other than intercepting the input/output, and is a bit buggy at that - so, no LLM there :)

But posting anyhow in case it can be useful for someone to avoid reinventing the plumbing.

By @abrichr - 5 months
By @stared - 5 months
The (post) title mentions a Rust library, I see code in TypeScript, and the installation instructions involve Python. Frankly, I am confused.

EDIT: To make it clear, my point is that the description could have been clearer, especially regarding the Python dependency.