Unreal Source Explained (USE) is an Unreal source code analysis, based on profilers.
For full Table of Contents and more infomation, see the repo in github.

Parallel Rendering

Render Commands

We usually create and enqueue a render command like this:

void DestroyRenderResource(FRenderResource* RenderResource) {
    ENQUEUE_RENDER_COMMAND(DestroySceneViewStateRenderResource)(
        [RenderResource](FRHICommandList&) {
            RenderResource->ReleaseResource();
            delete RenderResource;
        });
}

ENQUEUE_RENDER_COMMAND(link) creates a named (for easier debug) struct and calls EnqueueUniqueRenderCommand() with the lambda, hence, the above codes expand to these:

void DestroyRenderResource(FRenderResource* RenderResource) {
    struct DestroySceneViewStateRenderResourceName { 
        static const char* CStr() { return "DestroySceneViewStateRenderResource"; } 
        static const TCHAR* TStr() { return TEXT("DestroySceneViewStateRenderResource"); } 
    };
    
    // POI begin
    EnqueueUniqueRenderCommand<DestroySceneViewStateRenderResourceName>(
			[RenderResource](FRHICommandList&) {
				RenderResource->ReleaseResource();
				delete RenderResource;
			});
    // POI end
}

EnqueueUniqueRenderCommand()(link) will execute the lambda if it's in rendering thread. If not, it will use TGraphTask<> template to create a taskgraph's task to execute the lambda.

template<typename TSTR, typename LAMBDA>
FORCEINLINE_DEBUGGABLE void EnqueueUniqueRenderCommand(LAMBDA&& Lambda) {
	//QUICK_SCOPE_CYCLE_COUNTER(STAT_EnqueueUniqueRenderCommand);
	typedef TEnqueueUniqueRenderCommandType<TSTR, LAMBDA> EURCType;

	if (IsInRenderingThread()) {
		FRHICommandListImmediate& RHICmdList = GetImmediateCommandList_ForRenderCommand();
		Lambda(RHICmdList);
	}
	else {
		if (ShouldExecuteOnRenderThread()) {
			CheckNotBlockedOnRenderThread();
            // POI
		    TGraphTask<EURCType>::CreateTask().ConstructAndDispatchWhenReady(Forward<LAMBDA>(Lambda));
		}
		else { ... }
	}
}

Below is the time profiler filtered by void EnqueueUniqueRenderCommand() and only shows call tree whose samples count is more than 300 samples, which indicates their importance. We can know there are serveral threads, including game thread, are making lots of call to equeue render commands, e.g., FScene::UpdatePrimitiveTransform(), FSkeletalMeshObjectGPUSkin::Update(), FSceneViewport::EnqueueBeginRenderFrame(),

Recall from the Thread Management's Task Grpah chapter that, TaskGrpah is a multi-threaded task graph framework. With TGraphTask<EURCType>::CreateTask().ConstructAndDispatchWhenReady() called, it create TGraphTask instance who holds an instance of FRenderCommand, and dispatch this TGraphTask instance into the queue. Then in rendering thread, the task is dequeued and its DoTask() is called, hence, the user's lambda is executed.

template<typename TSTR, typename LAMBDA>
class TEnqueueUniqueRenderCommandType : public FRenderCommand {
public:
	TEnqueueUniqueRenderCommandType(LAMBDA&& InLambda) : Lambda(Forward<LAMBDA>(InLambda)) {}

    // POI begin
	void DoTask(ENamedThreads::Type CurrentThread, const FGraphEventRef& MyCompletionGraphEvent) {
		TRACE_CPUPROFILER_EVENT_SCOPE_ON_CHANNEL_STR(TSTR::TStr(), RenderCommandsChannel);
		FRHICommandListImmediate& RHICmdList = GetImmediateCommandList_ForRenderCommand();
		Lambda(RHICmdList);
	}
    // POI end
private:
	LAMBDA Lambda;
};

Each named thread has 2 queues for executing tasks. Take the rendering thread for example, the main queue (QueueIndex = 0) are those render task from game thread, these main tasks are of type TEnqueueUniqueRenderCommandType, which inherit from FRenderCommand. Because main render tasks are root tasks created from game thread to render thread, they are running in rendering thread (because their GetDesiredThread() return RenderThread). For better performance, they don't use task dependencies (their GetSubsequentsMode() returns ESubsequentsMode::FireAndForget). The local queue (QueueIndex = 1) handles rendering thread's internal sub-tasks, they are driven by task graph's prequisition dependencies (they are processed each time a FTaskGraphInterface::Get().WaitUntilTaskCompletes() is called, see image below).

Below is time profiler showing some call tree filtered by TGraphTask<TEnqueueUniqueRenderCommandType with great overhead (again, which indicating their importance). We can know they are all executing in the rendering thread, such as FRenderModule::BeginRenderingViewFamily(), FEngineLoop::Tick()::EndFrameName.

Summary

User enqueues render commands into task graph's task queue. In rendering thread, render commands are dequeued and executed.

RHI Commands

RHI stands for Render Haredware Interface, it's an abstraction of graphic APIs, so the rendering commands only depends RHI interfaces, rather not different platforms' GPU APIs.

Even if the game is starting up on a given platform (Windows/Android/iOS), it may still have to deceide which GPU API to be enabled (DirectX/OpenGL/Vulkan/Metal). FDynamicRHI is this API abstraction, and FMetalDynamicRHI, FVulkanDynamicRHI, etc. inherits from it. FDynamicRHI* GDynamicRHI is the global instance and initialized in RHIInit()(link), during FEngineLoop::PreInitPreStartupScreen()(link),

void RHIInit(bool bHasEditorToken) {
	if (!GDynamicRHI) {
		if (!FApp::CanEverRender()) {
			InitNullRHI();
		}
		else {
            // POI begins
			GDynamicRHI = PlatformCreateDynamicRHI();
            GDynamicRHI->Init();
            // POI ends
                ...
            }
        }
    }
}

Different platform has its own PlatformCreateDynamicRHI() implementation and returns corresponding FDynamicRHI instance.

IRHICommandContext is similar to FDynamicRHI, it's also an abstraction of GPU APIs hence inherited class FVulkanCommandListContext, FMetalRHICommandContext. But the context mainly handles things that can be paralelly encoded, such as those "draw" things. FDynamicRHI mainly handle the "resource" things. Note that, their methods all begin with RHI, such as FDynamicRHI::RHICreateUniformBuffer(), IRHICommandContext::RHIDrawIndexedPrimitive().

FDynamicRHI	IRHICommandContext
mainly "resource" RHI commands	mainly "draw" RHI commands

In older APIs, OpenGL and DirectX11, their dynamic RHI implements both the FDynamicRHI and IRHICommandContext

Remember that each render command accepts an FRHICommandList instance argument, take the following render command (link) for example, it generates RHI command via direct call of RHICmdList's member methods,

static void TickSlate(TSharedPtr<SWindow> SlowTaskWindow) {
    ...
    if (GIsRHIInitialized) {
        // POI
        ENQUEUE_RENDER_COMMAND(BeginFrameCmd)([](FRHICommandListImmediate& RHICmdList) { RHICmdList.BeginFrame(); });
    }

    // Tick Slate application
    FSlateApplication::Get().Tick();

    // End frame so frame fence number gets incremented
    if (GIsRHIInitialized) {
        // POI
        ENQUEUE_RENDER_COMMAND(EndFrameCmd)([](FRHICommandListImmediate& RHICmdList) { RHICmdList.EndFrame(); });
    }
    ...
}

However, most render command create new RHI Commands indirectly via further function calls, see the profiling call tree below, which is filtered by FRHICommandList:: and high sample counts. We can know that those important RHI commands are created in all kinds of render commands in the rendering thread.

Take the most important command, FRHICommandList::DrawIndexedPrimitive()(link) for example,

void FRHICommandList::DrawIndexedPrimitive(FRHIIndexBuffer* IndexBuffer, int32 BaseVertexIndex, ...) {
    if (Bypass()) {
        // POI
        GetContext().RHIDrawIndexedPrimitive(IndexBuffer, BaseVertexIndex, ...);
        return;
    }
    // POI
    ALLOC_COMMAND(FRHICommandDrawIndexedPrimitive)(IndexBuffer, BaseVertexIndex, ...);
}

It says, if current run is bypassing the RHI command, then RHI command list will calls its RHI Context to execute the RHI call directly. Otherwise, RHI command list will allocate the RHI command instance and queue them up. By default, Bypass()(link) always returns true in iOS, that may due to iPhone's great single core house power. And it returns false in Android (for v4.27), which may have many cores available and suit for having a dedicated RHI thread to handles RHI commands.

So, let's have a look at FRHICommandList's base class FRHICommandListBase:

class RHI_API FRHICommandListBase : public FNoncopyable {
    ...
    // POI begins (context)
public:
	bool Bypass() const;
    
	void SetContext(IRHICommandContext* InContext) {
		Context = InContext;
        ...
	}
	FORCEINLINE IRHICommandContext& GetContext() { return *Context; }

private:
	IRHICommandContext* Context;
    // POI ends (context)

    // POI begins (rhi command queue)
public:
	FORCEINLINE_DEBUGGABLE void* AllocCommand(int32 AllocSize, int32 Alignment) {
		FRHICommandBase* Result = (FRHICommandBase*) MemManager.Alloc(AllocSize, Alignment);
		++NumCommands;
		*CommandLink = Result;
		CommandLink = &Result->Next;
		return Result;
	}

private:
	FRHICommandBase* Root;
	FRHICommandBase** CommandLink;
	uint32 NumCommands;
    // POI ends (rhi command queue)

    ...
};

FRHICommandListBase has mainly two items: a context and a command queue. And it stores a list of instances whose type inherits from FRHICommandBase, with the Root as the list tail and CommandLink the head.

There is one global command list GRHICommandList(link), it's feed into the render command argument via static function FRHICommandListExecutor::GetImmediateCommandList()(link).

So, what is Immediate Command List? FRHICommandList wraps IRHICommandContext and provide "draw" methods. FRHICommandListImmediate inherits from FRHICommandList and providing additional FDynamicRHI methods about "resource". Hence is this following comparision table,

RHI Command List	RHI	Meaning
FRHICommandList	IRHICommandContext	"draw" methods
FRHICommandListImmediate	IRHICommandContext + FDynamicRHI	"draw" + "resource" methods

Note that, RHI Command list's member method names don't contain RHI, e.g., FRHICommandListImmediate::CreateUniformBuffer(), FRHICommandList::DrawIndexedPrimitive().

Summary

Render commands generate RHI commands. RHI commands may be executed in the rendering thread, or are queued up in the RHI command list and get executed in a dedicated RHI thread. RHI command list uses dyanmic RHI and RHI command context to run the actual GPU driver APIs.