SNeRF: A stylized neurotic implicit representation of a 3D scene

A recent article on arXiv.org looks at the problem of stylization 3D Scenes to match the reference style image. For example, when a VR headset is turned on, one can see the world through the lens of Pablo Picasso’s art instead of being limited by the real world. The researchers propose a combination of neural radiation field (NeRF) and image-based neurotype transfer to perform stylized 3D scenes.

Stylized results of SNeRF on 360° scenes using NeRF++. Image credit: arXiv: 2207.02363 [cs.CV]

NeRF provides a strong inductive bias to maintain the consistency of multiple views, and the neural transition allows for a flexible stylized approach that doesn’t require specialized example input. Furthermore, a new training program reduces GPU memory requirements during training, allowing for high-resolution results on a single modern GPU.

The reviews demonstrate that the proposed method yields better image and video quality than modern methods.

This paper presents a stylized novel view synthesis method. Applying modern stylization methods to novel views frame by frame often causes confusion due to lack of consistency between views. Therefore, this paper investigates 3D scene stylization which gives a strong inductive tendency to synthesize a consistent novel view. Specifically, we use emerging neural radiation fields (NeRFs) as our choice of 3D scene representation so that they are capable of rendering high-quality novel views for a variety of scenes. However, because rendering a novel view from NeRF requires a large number of samples, training a stylized NeRF requires a large amount of GPU memory that exceeds the available GPU capacity. We introduce a new training method to solve this problem by alternating NeRF optimization and stylization steps. Such an approach allows us to make the most of our hardware memory capacity to both produce images at higher resolution and adopt more expressive visual style delivery methods. Our tests show that our method generates stylized NeRFs for a wide range of content, including indoor, outdoor, and dynamic scenes, and synthesizes quality novel views high with cross-view consistency.

Research articles: Nguyen-Phuoc, T., Liu, F., and Xiao, L., “SNeRF: Stylized Neurological Implicit Performance for 3D Scenes”, 2022. Links: https://arxiv.org/abs/2207.02363

You can provide your link to a page relevant to the topic of this post.