Issues with Web UI Generation

The article discusses the challenges faced in generating web user interfaces (UI) using large language models (LLMs) within the AI and UX domain. It highlights the disconnect between various representations of UI, such as screenshots, Document Object Model (DOM), and source code, which complicates the AI's ability to manipulate source code effectively. The author points out that modern frameworks exacerbate these issues due to complex structures and naming conventions. The lack of comprehensive documentation for libraries leads to AI's reliance on guessing, which can result in errors. The article suggests that while open-ended generation is straightforward, adhering to specific constraints is significantly more challenging. It proposes that UI elements could encode their domain and range, potentially leading to self-documenting code. The discussion also touches on the dynamic nature of source code, emphasizing that AI struggles to manage changes over time. The author calls for improved models that can perceive changes in UI more effectively, possibly through video analysis, and suggests exploring the trade-offs between different representations of change. Overall, the piece advocates for a more structured approach to UI generation that considers the complexities of documentation, representation, and change.

- AI struggles with manipulating source code due to the disconnect between UI representations.

- Lack of documentation in libraries leads to reliance on guessing by AI.

- Open-ended generation is easier than following specific constraints in UI design.

- Proposes self-documenting UI elements to aid AI understanding.

- Emphasizes the need for models that can perceive UI changes over time.