Show HN: Relari – Auto Prompt Optimizer as Lightweight Alternative to Finetuning
Relari has introduced an Auto Prompt Optimizer to improve language model performance for specific tasks, offering transparency and ease of use, with future features planned and user feedback encouraged.
Relari, the founders of an LLM evaluation stack, have expanded their offerings to include an Auto Prompt Optimizer aimed at enhancing the performance of language models for domain-specific tasks. This tool addresses common frustrations in prompt engineering, such as the need for constant updates due to changes in models or user requirements. Unlike existing tools that require new frameworks and lack transparency, Relari's optimizer provides a user-friendly interface with clear visibility into the optimization process. It takes a dataset with expected outputs and a target metric to iteratively refine prompts, utilizing techniques like few-shot prompting and chain of thought. The optimizer can generate effective prompts with as few as 100 data points and is designed to be a practical alternative to fine-tuning. Users can test the optimizer by uploading datasets or generating synthetic ones, with the process taking up to an hour. Future developments will include advanced features like prompt chaining and custom metrics for complex use cases. The founders are seeking feedback on the challenges of prompt engineering and the potential benefits of a dataset-driven approach.
- Relari has launched an Auto Prompt Optimizer to enhance LLM performance for specific tasks.
- The tool offers transparency and ease of use compared to existing prompt optimization solutions.
- It can generate effective prompts with minimal data and is designed for quick adaptation.
- Future updates will include advanced features like prompt chaining and custom metrics.
- User feedback is encouraged to improve the prompt engineering workflow.
Related
Non-Obvious Prompt Engineering Guide
The article discusses advanced prompt engineering techniques for large language models, emphasizing structured prompts, clarity, and the importance of token prediction for optimizing interactions and achieving desired outcomes.
Perspectives for first principles prompt engineering
Prompt engineering optimizes prompts for large language models to meet user expectations. Best practices include clarity and specificity, while understanding LLM capabilities enhances prompt effectiveness through iterative refinement.
2 commentsMaybe this is too much of a tangent, but is it reasonable to want to see a la carte pricing options?
For instance, I have an enterprise project that I could see using this for, but that's a project with a discrete time budget (probably 1-3 months) and a tool like this would see heavy usage for the first few weeks, then intermittent usage, then we would only need to use it for maintenance updates far in the future.
The initial $1k/mo tier fills me with worry, because I can see blowing through my usage credits in the first month (and then I need to contact sales about signing up for the enterpri$$$e option), but then I wouldn't need nearly so many subscribed credits in the later months (and I imagine they don't build up like Audible credits, but instead are use-it-or-lose-it -- which also isn't great for how our development goes in spurts).
This is not the only AI tool that uses pricing like this (looking at you, Roboflow!) -- but I've never felt like this structure fits with the intermittent patterns of AI development that I use in my day-to-day job. I can understand wanting to have customers sign up for SAAS and the reliable income that such things would bring, but I feel like a la carte pricing for these kinds of tools (even if they were 4x or 5x more expensive than the equivalent credits in the subscription bundles) might let me try out these tools in an enterprise environment without waffling between "free tier" and "recurring SAAS budget line-item".
Side note: I’ve had a lot of luck combining automatic prompt optimization with finetuning. There is definitely some synergy https://raw.sh/posts/chess_puzzles
Related
Non-Obvious Prompt Engineering Guide
The article discusses advanced prompt engineering techniques for large language models, emphasizing structured prompts, clarity, and the importance of token prediction for optimizing interactions and achieving desired outcomes.
Perspectives for first principles prompt engineering
Prompt engineering optimizes prompts for large language models to meet user expectations. Best practices include clarity and specificity, while understanding LLM capabilities enhances prompt effectiveness through iterative refinement.