Diffusion Texture Painting

Researchers from NVIDIA, the University of Toronto, and the Vector Institute have introduced a novel technique called Diffusion Texture Painting. This method utilizes 2D generative diffusion models to enable interactive texture painting on 3D meshes. Unlike existing systems, artists can paint with complex image textures and seamlessly transition between different textures in real-time. The approach involves a stamp-based method that leverages pre-trained models to inpaint patches, providing control over brush stroke shape and texture orientation. By adapting the inference of generative models, the system ensures stable texture brush identity while allowing for infinite variations of the source texture. This innovative method marks the first use of diffusion models for interactive texture painting, aiming to inspire further exploration of generative models in artist-driven workflows. The application scenarios include creating bold garment prints, detailing photogrammetry assets with forest textures, replicating physical toy textures on 3D models, and prototyping fantasy environments like gingerbread houses. The research paper will be presented at SIGGRAPH 2024.