Show HN: Lemon Slice Live, a real-time video-audio AI model
Lemon Slice has developed a real-time avatar generation tool using a custom DiT model, enabling users to create talking avatars from a single image with 25fps video streaming and low latency.
Lemon Slice, co-founded by Lina, Andrew, and Sidney, has developed a custom diffusion transformer (DiT) model that enables real-time video streaming at 25 frames per second (fps). This technology allows users to create talking avatars from a single photo without the need for pre-training or rigging. The demo can be accessed at their website, where users can upload images of any style to generate custom characters for video calls. Key advancements include a fast DiT model that synchronizes lip movements and facial expressions with audio, a technique to maintain visual coherence across long video sequences, and a complex streaming architecture that minimizes latency. The system currently achieves a 3-6 second latency from user input to avatar response, with a goal of under 2 seconds. Future improvements aim to incorporate whole-body motions, enhance resolution, and enable avatars to engage in more natural conversations. The founders envision a future where generative video transforms media consumption, blending interactive and passive experiences in entertainment.
- Lemon Slice has created a real-time avatar generation tool using a custom DiT model.
- Users can generate talking avatars from a single image without prior training or rigging.
- The technology achieves video streaming at 25fps and aims for under 2 seconds of latency.
- Future developments will focus on improving avatar interactions and visual quality.
- The founders predict a shift in media consumption towards more interactive experiences.
Related
Show HN: Infinity – Realistic AI characters that can speak
Infinity AI has developed a groundbreaking video model that generates expressive characters from audio input, trained for 11 GPU years at a cost of $500,000, addressing limitations of existing tools.
A New System for Temporally Consistent Stable Diffusion Video Characters
Alibaba Group's MIMO system improves full-body avatar generation with Stable Diffusion, addressing temporal stability issues and utilizing three encodings for character, scene, and occlusion, demonstrating flexibility in video synthesis.
Show HN: A real time AI video agent with under 1 second of latency
Tavus, co-founded by Hassaan and Quinn, develops AI video models for realistic conversations, achieving under 1 second latency with the Phoenix-2 model, attracting clients like Delphi for digital twin technology.
Perceptually lossless (talking head) video compression at 22kbit/s
The LivePortrait model achieves perceptually lossless video compression at 22kbit/s by animating still images, focusing on facial keypoints. It has potential in video conferencing and social media, despite some limitations.
A16Z: AI Avatars
AI avatars are evolving to create realistic, interactive characters for various sectors, enhancing storytelling and engagement. They utilize advanced technology for speech and movement synchronization, benefiting businesses and consumers alike.
- Users are impressed by the technology and its ability to create engaging avatars, noting the potential for future improvements.
- Some commenters express concerns about the realism of the avatars, particularly in terms of voice and facial expressions.
- There are inquiries about the model's architecture and input capabilities, with curiosity about non-photo inputs like artwork.
- Suggestions for applications include using the technology for virtual interactions with deceased loved ones and enhancing customer service experiences.
- Several users mention the need for a more accessible trial experience, expressing reluctance to sign up for new services without assurance of quality.
23 commentsOverall, a fun experience. I think that MH was better than Scott. Max was missing the glitches and moving background but I'd imagine both of those are technically challenging to achieve.
Michael Scott's mouth seemed a bit wrong - I was thinking Michael J Fox but my wife then corrected that with Jason Bateman - which is much more like it. He knew Office references alright, but wasn't quite Steve Carell enough.
The default while it was listening could do with some work, I think - that was the least convincing bit; for Max he would have just glitched or even been completely still I would think. Michael Scott seemed too synthetic at this point.
Don't get me wrong, this was pretty clever and I enjoyed it, just trying to say what I found lacking without trying to sound like I could do better (which I couldn't!).
How does voice work? You mentioned Deepgram. Does it mean you do Speech-to-Text-to-Speech?
Impressive technology - impressive demo! Sadly, the conversation seems to be a little bit overplayed. Might be worth plugging ChatGPT or some better LLM in the logic section.,
How can you be sure? Investing in an ASIC seems like one of the most expensive and complicated solutions.
It seems clumsy to use copyrighted characters in your demos.
Seems to me this will be a standard way to interact with LLMs and even companies - like a receptionist/customer service/salesperson.
Obviously games could use this.
I'll pass thanks.
Related
Show HN: Infinity – Realistic AI characters that can speak
Infinity AI has developed a groundbreaking video model that generates expressive characters from audio input, trained for 11 GPU years at a cost of $500,000, addressing limitations of existing tools.
A New System for Temporally Consistent Stable Diffusion Video Characters
Alibaba Group's MIMO system improves full-body avatar generation with Stable Diffusion, addressing temporal stability issues and utilizing three encodings for character, scene, and occlusion, demonstrating flexibility in video synthesis.
Show HN: A real time AI video agent with under 1 second of latency
Tavus, co-founded by Hassaan and Quinn, develops AI video models for realistic conversations, achieving under 1 second latency with the Phoenix-2 model, attracting clients like Delphi for digital twin technology.
Perceptually lossless (talking head) video compression at 22kbit/s
The LivePortrait model achieves perceptually lossless video compression at 22kbit/s by animating still images, focusing on facial keypoints. It has potential in video conferencing and social media, despite some limitations.
A16Z: AI Avatars
AI avatars are evolving to create realistic, interactive characters for various sectors, enhancing storytelling and engagement. They utilize advanced technology for speech and movement synchronization, benefiting businesses and consumers alike.