Implementation

Architecture and Files

Runtime files

The main entry point for the package's runtime C# logic is the Runtime/VirtualMeshManager.cs component script. This component should be attached to the scene's main camera and handles all the processing required by the virtual mesh system during the Unity game loop. Persistent runtime objects such as GraphicsBuffers are also owned by this component.

The other important runtime C# class is Runtime/RenderFeatures/VirtualMeshRenderFeature.cs, which contains the render feature responsible for all the custom render passes that the virtual mesh system injects into Unity's rendering pipeline. These passes use the buffers and resources owned by the VirtualMeshManager to perform appropriate shader calls and dispatches that make up the GPU-driven pipeline.

Shaders used by the system during runtime can be found under Runtime/Shaders, which contains compute shaders used by the culling and LOD pipeline (Runtime/Shaders/DXCVisibilityPasses.compute and Runtime/Shaders/DepthPyramidPass.compute), compute shaders used by the streaming system (Runtime/Shaders/CopyPasses.compute), and vertex/fragment shaders for custom rendering passes used throughout the system (Runtime/Shaders/CustomPasses.shader and Runtime/Shaders/DebugPasses.shader).

Runtime/ShaderLibrary contains shader code files that are used to replace includes inside URP/Lit (files that start with "Lit") and Shader Graph (files that start with "PBR") shaders, which are used by virtual meshes. These files contain instanced versions of the vertex shaders used by each rendering pass supported by the virtual mesh system.

The virtual mesh system also uses some materials and render textures internally, which can be found under Runtime/Materials and Runtime/RenderTextures.

Lastly, the streaming system relies on an embedded version of BetterStreamingAssets, which can be found under Runtime/BetterStreamingAssets.

Baker files

Editor/VirtualMeshBaker.cs and Editor/VirtualMeshBakerAPI.cs respectively contain the editor tool and the baking algorithm.

Editor/MeshOperations.cs contains functions that encapsulate meshoptimizer APIs used for processing meshes during baking. The plugin itself can be found inside Editor/Optimizer/Plugins.

Editor/ShaderGraphHelper contains classes used to retrieve the shader source code generated from Shader Graph shaders. This source code is used instead of the Shader Graph assets because vertex shaders need to be swapped for an instanced version compatible with virtual meshes.

Streaming

The virtual mesh system uses custom binary files to stream mesh data during runtime. These files, saved inside the project's Assets/StreamingAssets folder, are called pages and are loaded dynamically based on requests made on the GPU.

These requests are given to the CPU via a GraphicsBuffer with the RequestAsyncReadback API. Based on the contents of that buffer, the CPU queues pages that need to be loaded and dispatches Unity C# jobs that handle the file I/O and streaming. Each job writes to an upload buffer comprised of several GraphicsBuffer objects with GraphicsBuffer.UsageFlags.LockBufferForWrite mode for asynchronous writing to the GPU. The buffers are wrapped in a fenced buffer implementation taken from the Entities Graphics package.

Each upload buffer corresponds to the data of an entire page, including metadata and all the cluster index and vertex buffers. After the C# jobs finish filling them, the buffers are read from a series of compute shader dispatches that copy their content into the large runtime buffers used for drawing virtual meshes (see the StreamingJobsKickoff and StreamingJobsWrapup functions in Runtime/VirtualMeshManager.cs).

There is a limited number of upload buffers, each independent from each other. When the CPU loads a page, it must look for an upload buffer that is not currently being written to by a job. If a buffer is available, it will be pooled to load the designated page and locked until the compute copies are finished.

To detect if a page needs loading, the CPU keeps a record of the status of every memory page (see the MemoryPageStatus enum in Runtime/VirtualMeshManager.cs).

• Unloaded and Loaded are the most common statuses that indicate when pages are resident or not.

• Waiting means that a page has been requested by the GPU and should ideally be loaded, but there are no slot left to put it in, so it is waiting for another page to unload.

• Loading indicates that a page has been assigned to a slot and that the CPU job reading its file has been dispatched but has not yet completed.

• TooFar is similar to Waiting because the GPU has requested the page, but its contents are considered to be too far away from the camera, so the CPU will not load it to save page slots for geometry that is closer to the camera.

Updates to page statuses are made on the GPU based on the bounding boxes surrounding the geometry contained inside each page. Pages are sorted in order of the size of their bounds' projection on the screen so that pages that have a bigger geometry size on screen are requested with higher priority. On top of this, the distance between a page and the camera is taken into account to avoid requesting pages that are very far. This distance value can be changed by adjusting the CameraLoadDistanceThreshold property in Runtime/VirtualMeshManager.cs.

Baking

To bake virtual meshes and prepare them for rendering, the system performs several tasks that are implemented inside Editor/VirtualMeshBakerAPI.cs.

First, shaders used by objects being baked need to be converted to versions that use vertex shaders compatible with the instancing and attribute unpacking schemes required by virtual meshes. The baker picks up shader source codes and replaces includes corresponding to files with vertex shader code with alternatives that support virtual meshes (see the ConvertShaders function in Editor/VirtualMeshBakerAPI.cs).

The next step is to iterate over GameObjects that need to be converted and apply the following algorithm to convert them (see the ConvertMeshes function in Editor/VirtualMeshBakerAPI.cs):

1. We first generate a list of MeshFilter objects to loop over. For now, the selection criteria is to pick only the highest LOD of every LODGroup hierarchy, or the whole MeshFilter if it is a standalone mesh without LODs (see the GetFilterList function in Editor/VirtualMeshBakerAPI.cs).

2. For every MeshFilter chosen previously, we select the filter's sharedMesh and loop over its submeshes.

3. For every submesh, we check if its material's shader is supported or not, or if its topology is not triangles (we only support triangle-based meshes for simplicity). The submesh's material is then assigned to a unique ID, which will be used to group geometry per material for draw calls.

4. Each submesh's geometry buffers (triangle indices, vertex positions and attributes) are then extracted. The vertex position values are compressed into half-precision floats to improve streaming and GPU data handling performance. This effectively quantizes vertex positions and slightly impacts the final geometry's appearance.

5. Using MeshOperations.BuildMeshlets, we split the submesh into meshlets of 64 triangles.

6. The resulting meshlets are partitioned with MeshOperations.PartitionMeshlets, which gives us groups of meshlets to use as leaf nodes (highest density LODs) for our cluster groups.

7. For every partition, we merge the meshlets into a single index buffer to perfom consecutive simplification and clustering over it until a resulting cluster hierarchy is built. During each step of the recursion we merge and simplify the clusters corresponding to a layer of the LOD hierarchy and record values that will allow us to switch between these layers during runtime.

8. For every resulting group, we find a page where the whole group's hierarchy fits while not exceeding the max number of instances (= meshlets) allowed per page. This is done by finding a suitable MemoryPageData class to record the data in an array of instances each corresponding to a page.

9. For every resulting group, we also keep a separate set of index and vertex buffers corresponding to the leaf clusters. These will be merged with other buffers from the same memory page to form a mesh representing the page's placeholder.

10. After building all the cluster groups, we generate placeholders and serialize every MemoryPageData instance into corresponding files that will be streamed during runtime. Lastly, we export asset bundles that hold all the materials and placeholder meshes generated during the process (see the WriteFiles function in Editor/VirtualMeshBakerAPI.cs).

LOD Selection

The selection of the LOD layer to draw within each cluster group is based on an error metric generated during simplification and representing how much each simplified layer deviates from its children in the hierarchy. The error is scaled to represent this deviation in screen-space based on a 1000 x 1000 pixel viewport and the main camera's FOV during baking (see the ComputeLODProjectionError function in Editor/VirtualMeshBakerAPI.cs). Changing the camera's FOV requires a rebake.

In addition, a scaling factor is applied on the GPU to account for transform scales. You can adjust this value to either expand or shrink the range of switching LODs to better cover the depth of geometry shown in the camera (see the LODProjectionErrorFactor constant define in Runtime/ShaderLibrary/ComputeCommon.hlsl).