Unreal Source Explained (USE) is an Unreal source code analysis, based on profilers.
For full Table of Contents and more infomation, see the repo in github.

Thread Management

RunnableThread and Runnable

FRunnableThread(link) is a cross platfrom abstract "native thread" interface in Unreal. It has different implementaion in different OS, for example, FRunnableThreadWin(link) in Windows, FRunnableThreadPThread(link) in POSIX-compliant OS, i.e., iOS and Android.

FRunnable(link)'s inherited class represets the actual running workload, e.g., FRenderingThread(link).

Each FRunnableThread runs one FRunnable.

Take the POSIX FRunnableThreadPThread for example, static function _ThreadProc()(link) is the thread entry point and it calls every FRunnableThreadPThread::Run(), as follows.

/**
* The thread entry point. Simply forwards the call on to the right
* thread main function
*/
static void *STDCALL _ThreadProc(void *pThis) {
	FRunnableThreadPThread* ThisThread = (FRunnableThreadPThread*)pThis;
	...
	// run the thread!
	ThisThread->PreRun();
	ThisThread->Run();
	ThisThread->PostRun();

	pthread_exit(NULL);
	return NULL;
}

then, FRunnableThreadPThread::Run() calls FRunnable::Run()(link).

RunnableThread and Runnable is the low-level thread management in Unreal. The following Async Task and Task Graph depend on Runnable.

Thread priority and affinity

Priority determines how threads are scheduled amoung each other, you first? or he first. Affinity determines where threads are running, the performant big core? or the power-efficent little core.

Thread's priority is usually set when they are created, as follow,

at the bottom, priority is set by FRunnableThreadPThread::SetThreadPriority()(link) and it calls UNIX pthread_setschedparam() (link). It translate Unreal's NewPriority to platform's actual priority integer, then override current schedule parameter's sched_priority. And change the policy to SCHED_RR which means Round Robin and each thread executes in equal time-interval and yield to next thread. Next thread is determined by the priority.

But is SCHED_RR a good default policy for all threads? Maybe we should try SCHED_OTHER as default policy, and only set those more-realtime-like threads to be SCHED_RR, such as main thread, render thread (and RHI thread if enabled).

Though Unreal implements thread affinity APIs, it doesn't call them by default and use default thread affinity. You can call them manually.

Android can use FAndroidPlatformProcess::SetThreadAffinityMask()(link) to pin thread to a specific CPU core, it internally calls __NR_sched_setaffinity.

Though in iOS it also provides FIOSPlatformProcess::SetThreadAffinityMask()(link), but note that macOS and iOS doens't support explicit thread to processor binding. They can only use thread_policy_set() to separate threads into different affinity set, i.e., threads with different should run on diffrent logical processors. Threads with default affinity are freely scheduled by system and tends to run on idle processors.

Queued Thread and Async Task

Queued Thread and Async Task is Unreal's thread pool implementation. All kinds of tasks are scheduled among a pool of threads.

FQueuedThreadPoolBase(link) manages this task queue and thread queue, as follow,

/**
 * Implementation of a queued thread pool.
 */
class FQueuedThreadPoolBase : public FQueuedThreadPool {
protected:
	/** The work queue to pull from. */
	TArray<IQueuedWork*> QueuedWork;
	/** The thread pool to dole work out to. */
	TArray<FQueuedThread*> QueuedThreads;
	/** All threads in the pool. */
	TArray<FQueuedThread*> AllThreads;
	...
};

There are 4 threads pools(link) in maximum, but the most important is the GThreadPool, which is the default thread pool for most tasks.

FQueuedThread(link) inherits from FRunnable, it represents the task worker thread. Its Run() waits for its DoWorkEvent, if signaled, it runs its current QueuedWork by calling IQueuedWork::DoThreadedWork(), as follow,

class FQueuedThread : public FRunnable {
protected:
	/** The event that tells the thread there is work to do. */
	FEvent* DoWorkEvent;
	/** The work this thread is doing. */
	IQueuedWork* volatile QueuedWork;
	/** The pool this thread belongs to. */
	class FQueuedThreadPool* OwningThreadPool;
	/** My Thread  */
	FRunnableThread* Thread;

	virtual uint32 Run() override {
		while (!TimeToDie.Load(EMemoryOrder::Relaxed)) {
			// We need to wait for shorter amount of time
			bool bContinueWaiting = true;
			while( bContinueWaiting ) {
				// Wait for some work to do
				bContinueWaiting = !DoWorkEvent->Wait( 10 );
			}

			IQueuedWork* LocalQueuedWork = QueuedWork;
			QueuedWork = nullptr;
			FPlatformMisc::MemoryBarrier();
			while (LocalQueuedWork) {
				// Tell the object to do the work
				LocalQueuedWork->DoThreadedWork();
				// Let the object cleanup before we remove our ref to it
				LocalQueuedWork = OwningThreadPool->ReturnToPoolOrGetNextJob(this);
			} 
		}
		return 0;
	}
	...
	void DoWork(IQueuedWork* InQueuedWork) {
		// Tell the thread the work to be done
		QueuedWork = InQueuedWork;
		FPlatformMisc::MemoryBarrier();
		// Tell the thread to wake up and do its job
		DoWorkEvent->Trigger();
	}
};

FEngineLoop::PreInit()(link) calls FQueuedThreadPoolBase::Create()(link) to creates FQueuedThreads.

IQueuedWork(link) is the task interface, it can't be any simpler:

class IQueuedWork {
public:
	virtual void DoThreadedWork() = 0;
	virtual void Abandon() = 0;

public:
	virtual ~IQueuedWork() { }
};

IQueuedWork's most important implementation is FAsyncTask(link).
FAsyncTask is a template class, you can create your own async task class as the sample code in FAsyncTask's comment(link). Various specific tasks start their work as the following.

Task Graph

Parallel programming is hard, multithreaded programming in fine granularity is even harder, because you may spend lots of time to take care of locking, waiting, race condition in every detailed level.

Task Graph is a parallel programming with coarse granularity, and it handles dependency among async tasks. A simplified task graph of a game may be depicted as follow:

We can divide the whole game into several big tasks, tasks in parallel can shared read some data but never shared write, tasks with dependency must finish in order.

You can create your own task as the demo code in TGraphTask's comment(link). Keep in mind that the task argument can not be references, but pointer is OK.
The following image is the call stacks filtered by "TGraphTask", you may notice the both the render thread and the game thread use the task graph to accomplish many important tasks, even include the world ticking.

FTaskGraphImplementation(link) is the most important manager of Task Graph, it manages an array of FWorkerThreads,

/**
*	FTaskGraphImplementation
*	Implementation of the centralized part of the task graph system.
*	These parts of the system have no knowledge of the dependency graph, they exclusively work on tasks.
**/
class FTaskGraphImplementation : public FTaskGraphInterface {
	...
	/** Per thread data. **/
	FWorkerThread		WorkerThreads[MAX_THREADS];
	/** Number of threads actually in use. **/
	int32				NumThreads;
	/** Number of named threads actually in use. **/
	int32				NumNamedThreads;
	/** Number of tasks thread sets for priority **/
	int32				NumTaskThreadSets;
	/** Number of tasks threads per priority set **/
	int32				NumTaskThreadsPerSet;
	...
}

FWorkerThread(link) is a wrapper of the running thread and the task.

struct FWorkerThread {
	/** The actual FTaskThread that manager this task **/
	FTaskThreadBase*	TaskGraphWorker;
	/** For internal threads, the is non-NULL and holds the information about the runable thread that was created. **/
	FRunnableThread*	RunnableThread;
	/** For external threads, this determines if they have been "attached" yet. Attachment is mostly setting up TLS for this individual thread. **/
	bool				bAttached;
	...
};

FTaskThreadBase(link) implements FRunnable, and it has only two inherited class:

• FNamedTaskThread(link), named task thread runs the engine built-in tasks, there are 5 named thread(link): StatsThread, RHIThread, AudioThread, GameThread and ActualRenderingThread.
• FTaskThreadAnyThread(link), unnamed task thread runs the user defined tasks.

They both have its own internal FThreadTaskQueue implementation and queue instance as data member. And FTaskGraphImplementation::QueueTask() schedues tasks into the corresponding task queues.

virtual void QueueTask(FBaseGraphTask* Task, ENamedThreads::Type ThreadToExecuteOn, ENamedThreads::Type InCurrentThreadIfKnown = ENamedThreads::AnyThread) final override {
	...
	ENamedThreads::Type CurrentThreadIfKnown;
	if (ENamedThreads::GetThreadIndex(InCurrentThreadIfKnown) == ENamedThreads::AnyThread) {
		CurrentThreadIfKnown = GetCurrentThread();
	}
	else {
		CurrentThreadIfKnown = ENamedThreads::GetThreadIndex(InCurrentThreadIfKnown);
		checkThreadGraph(CurrentThreadIfKnown == ENamedThreads::GetThreadIndex(GetCurrentThread()));
	}
	{
		int32 QueueToExecuteOn = ENamedThreads::GetQueueIndex(ThreadToExecuteOn);
		ThreadToExecuteOn = ENamedThreads::GetThreadIndex(ThreadToExecuteOn);
		FTaskThreadBase* Target = &Thread(ThreadToExecuteOn);
		// POI begin
		if (ThreadToExecuteOn == ENamedThreads::GetThreadIndex(CurrentThreadIfKnown)) {
			Target->EnqueueFromThisThread(QueueToExecuteOn, Task);
		}
		else {
			Target->EnqueueFromOtherThread(QueueToExecuteOn, Task);
		}
		// POI end
	}
}

Named task thread are created in various places. But the unnamed task threads are created inside the constructor(link) of FTaskGraphImplementation, we can observe this by the thread creation(link):

Named thread cannot handle infinite recusive tasks depth, they can only handle 2 layers of tasks: they have 2 queues to executes tasks.

The first one is Main Queue (QueueIndex=0), and the second one is Local Queue (QueueIndex=1). Main Queue's tasks are automatically pull out and execute until the queue is empty.

Local Queue handles intenral tasks and they won't be automatically executed, they must be manually trigged by calling ProcessThreadUntilIdle() or by WaitUntilTasksComplete() (which internally call ProcessThreadUntilIdle())

See Parallel Rendering for rendering thread Main Queue / Local Queue analysis.

Any thread has only 1 task queue and it can handle recursive tasks simply by events dependencies.