Enhancing 3D Visualizations with Stable Diffusion AI

Hello, everyone! I’m Viktor Prestaya, and for the last 10 years, I’ve been working on interior and exterior visualizations. Today, I’d like to discuss using Stable Diffusion to enhance 3D renders.

Stable Diffusion is typically used within the Automatic 1111 interface, one of the most popular and user-friendly platforms. It allows you to generate images from text prompts and work directly with your renders using Img2img and Img2img Inpaint modes.

For a more detailed explanation of what models, Img2img, Inpaint, and VAE are, you can watch this video:

In addition to Stable Diffusion, there are other neural networks like Invoke or ComfyUI, and you can choose whichever suits your workflow best:

• https://www.invoke.com/
• https://github.com/comfyanonymous/ComfyUI

As 3D visualizers, we often spend much time adding people to our renders. For medium and long-range shots, we typically use 3D scans, and for close-up views, we insert cutout people.

Using Stable Diffusion's Inpaint mode, we can refine 3D scans of people in the foreground and enhance them to look more realistic. You can even enhance cutout figures if necessary.

For example, 3D scans of people were used in the foreground and refined using Inpaint. The same approach can be applied to elements like fire, wood, concrete, and more, saving time on detailed adjustments directly in 3ds Max.

In the following example, two 3D scans and a cutout of a cat and mouse were enhanced, along with their materials.

To maintain creative control and stay close to your initial vision, you can use ControlNet models, which provide an unprecedented level of precision over your images. You can learn more in this video:

The most useful models are Canny and Line Art, which help control shapes. Canny works best with edges, while Line Art is better for everything else. They can also be combined.

It’s also worth checking out the SD Upscale feature for Stable Diffusion:

With these new tools, we can spend less time on technical details and focus more on creativity. For example, you can start with a basic render, do some touch-ups in Photoshop, and then refine it in Stable Diffusion.

While there's no magic button to do everything for us just yet—Inpaint can still be a time-consuming process, requiring a well-prepared base image and several generations to get the desired result—it’s still faster than doing everything in 3D from scratch.

One thing is for sure: the future is exciting and unpredictable. Neural networks will bring significant changes to our workflow. Today, they can already help us with references, ideas, refining renders, and even generating videos.

Thank you for your attention! I’m happy to share my experience, and for more details about my work, feel free to visit my profile on Behance: https://www.behance.net/3d_ellesare41c

About the author

Viktor Prestaya

Awarded CG Artist