SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code

SceneCraft is a Large Language Model (LLM) Agent designed to convert text descriptions into Blender-executable Python scripts for rendering complex 3D scenes with up to a hundred assets. The process involves spatial planning and arrangement, achieved through advanced abstraction, strategic planning, and library learning. SceneCraft creates a scene graph to define spatial relationships among assets, generates Python scripts based on this graph, and refines scenes using vision-language models like GPT-V. It also incorporates a library learning mechanism to improve continuously without extensive parameter tuning. Evaluation shows SceneCraft outperforms other LLM-based agents in adhering to constraints and receiving positive human assessments. The system's capabilities are demonstrated by reconstructing detailed 3D scenes from the Sintel movie and guiding a video generative model with the scenes as control signals.