A VR application developer is tasked with creating a realistic office interior for employee training. Manually modeling every detail takes weeks of work. Photogrammetry requires traveling to the location.
An alternative is neural network generation of 3D scenes from a text description or a few photos. We implement such pipelines turnkey — 8+ years of experience and more than 50 projects in VR/AR. Get a consultation on integrating neural network generation into your project.
The main challenge of neural methods is the balance between quality and performance. Many models deliver high quality but require minutes per frame. VR needs a frame rate of 72+ FPS. We solve this through automatic LOD optimization and quantization of neural networks to INT8. This reduces memory consumption without sacrificing quality.
How do we generate 3D scenes from text?
We take a text description (for example, "an abandoned laboratory with fluorescent lamps") and run it through diffusion models — SceneScape or Set-the-Scene. The result is a depth map and semantic segmentation, which are converted into a mesh.
An alternative path is PanoGen for 360° panoramas: 30–60 seconds per environment for an initial prototype. To improve realism, we use few-shot fine-tuning on a small set of reference images.
Step-by-step scene generation process
- Preparing a text description or a set of photos (20–100 images).
- Choosing a method: Gaussian Splatting for photos, PanoGen for 360°, Text-to-Scene for text.
- Optimization: quantizing the model to INT8, generating LODs, configuring occlusion culling.
- Exporting to the target engine (Unreal, Unity, WebXR).
- Testing on a headset (Quest 3, Pico) with FPS and memory metrics.
Why is Gaussian Splatting better than NeRF for VR?
NeRF delivers high detail but requires seconds to render a frame. Gaussian Splatting renders in real time and does not fall short in quality. We use it to reconstruct objects from 20–100 photos.
Additionally, we apply model quantization to INT8, which reduces memory consumption by 2–3 times. Automatic LOD generation using a simplification algorithm based on quadric error metrics makes it possible to achieve 72 FPS even on mobile headsets. We guarantee compatibility with any target device after configuration.
| Method | Generation time | Quality | Application |
|---|---|---|---|
| Gaussian Splatting (50 photos) | 5–15 min | Photorealistic | Real objects and spaces |
| Text-to-Scene | 2–10 min | Medium-high | Fantasy/sci-fi environments |
| PanoGen (360°) | 30–60 sec | High for skybox | Rapid prototyping |
| Manual+AI population | 1–3 h | High | Detailed interiors |
Typical mistakes when generating 3D scenes and how to avoid them
When using Gaussian Splatting, artifacts often appear at object boundaries. The solution is to add depth regularization and tune the number of iterations.
Text-to-Scene may produce unrealistic proportions — we fix this by adding semantic maps and fine-tuning on an interiors dataset. All our pipelines include automatic quality control with PSNR/SSIM metrics output. Time savings compared to manual modeling reach 80%, and generation costs are 5–10 times lower.
Technical details of quantization
To reduce memory consumption without losing quality, we apply INT8 quantization with calibration on a representative sample. We use TensorRT and ONNX Runtime libraries to optimize inference. The process is automated in an MLflow-based pipeline.When does generative 3D generation replace classical modeling?
For mass generation of variations of a single environment — for example, 50 different offices for negotiator training — generative methods are indispensable. When time is tight, a text description turns into a draft scene within minutes.
For unique, highly detailed assets, manual modeling still remains the primary option. Budget savings on mass generation reach 60% compared to manual work.
Contact us for a preliminary assessment of your project — we will select the optimal generation method for your budget and timeline.
What is included in the work
Analytics and design
Audit of source data, selection of architecture (text/photo/geometry). Selection of methods (Gaussian Splatting, NeRF, PanoGen) and pipeline setup taking into account the target device.
Implementation and integration
Training and quantization of models, automatic optimization for VR. Export to Unreal Engine, Unity, WebXR. Development of a plugin for your project. All stages are documented.
Testing and support
Performance measurement on target devices (Quest 3, Pico, etc.) with FPS and memory metrics. Training for your team, documentation, and one month of support.
Performance metrics (example on Quest 3)
| Method | Triangles | FPS | GPU Memory |
|---|---|---|---|
| Gaussian Splatting | 2M gaussians | 72 | 1.2 GB |
| Text-to-Scene (optimized) | 500K polygons | 90 | 800 MB |
| PanoGen | 10K (skybox) | 144 | 50 MB |
Final scenes are exported in the following formats: UAsset (Unreal Engine 5), Prefab (Unity), glTF 2.0 (WebXR), OpenXR/GL (Quest 3). Our certified specialists guarantee correct operation on any headset.
Order an evaluation of your project — we will select the optimal generation method for your budget and timeline.
