Now that projects are building, it is incredibly valuable to run tests against the code to validate it.
This is all the more important as in the future: in GitHub we can require that a pull request pass tests and be validated before it is merged into the main branch.
The next step is to get Jenkins to search GitHub for repositories to build.
I want all the projects in my GitHub repository that qualify to be built automatically, so if I add a new project with the right setup, it will automatically build it.
This can be achieved by having Jenkins scan your GitHub repository to gather all projects. GitHub supports pushing to Jenkins when repositories change, but that is more complicated when Jenkins is behind a firewall and cannot be directly contacted (we’ll be relying on polling in this case instead).
This is more of a guide for the process I went through for setting up Jenkins for my CI/CD. The following steps are only about setting up Jenkins itself and does not include subsequent configuration required for the jobs themselves.
I have a number of personal projects with code scattered about, where if I want to include code from one project into another I generally either need to copy code or find more sophisticated ways to share.
After going through interviewing coop student candidates and seeing their projects and GitHub repositories (when provided) I realized that documenting the process and demonstrating the value you can get out of it would be substantial.
The motivation for me is to take my projects to a higher level of rigour and improve reuse. Furthermore, I want to automate builds, validation by content tests, notifications, etc.
I do use GitHub for storing my projects which are primarily in C# but I want to amp up my setup to another level. The feature I want are:
Get my projects from GitHub
Build them
Run tests on them
Post main branch builds to a NuGet repository
Post main branch symbols to a symbol server
Generate notifications to Slack (I have my own setup for the family) about build status.
I already have a machine I use locally for service-related work. Switching to Linux would be non-trivial due to some of the services being run.
So the following setup will be using Windows, but I don’t think it will be entirely dissimilar to Linux.
The software I’m looking at using for all of these are:
Jenkins – this will run the CI/CD process. I already have familiarity with this from work: https://www.jenkins.io/
Symbolicator – I haven’t used this yet, so this may change when I get to it, but it does support multi-platform symbols (according to the documentation): https://getsentry.github.io/symbolicator/
Anything else that comes up during setup will be documented along the way.
The Unreal Engine has some rather nifty functionality for dealing with localization. We have also made numerous modifications to facilitate our workflow, but I’ll stick to what is in vanilla Unreal (this is based on 4.25).
There are three different string types to use within Unreal:
FString: These are traditional strings most people are used to. When creating a new string it will allocate memory to store the data. Equality comparisons need to check the actual string data.
FName: When you create a name, it is stored in a table in memory permanently (there is no clean up of unused names). Two FNames with the same value point to the same index, which makes equality comparisons very fast: they just compare indices. But the memory usage can grow without bound.
FText: These are localized strings that go through a process of being gathered, translated, and having their values stored in a separate file called a locres file. When evaluating an FText, there is a lookup for the actual value to use.
Epic provides a tokenization system using curly braces, { and }, where you can provide parameters (either positional or named) for an FText format string.
However, we added another one using square brackets, [ & ], where we would take the token within and look up the value as an FName in various tables that were provided to look up information.
A commandlet in Unreal is a mini-program that can be run from the commandline typically for automation.
A couple of significant examples are:
Resave Packages: resaves packages, typically used to resave older packages to bring them up to date.
Fixup Redirects: when assets are moved, they leave redirectors behind so that assets that reference them can be updated. Fixup Redirects updates those assets referencing the redirectors, then removes the redirectors that are no longer used.
Cooking: convert content into a format more optimized for target platforms.
Localization Pipeline: processes text in the game for translation and retrieves translations to include into the game.
How to Create A Commandlet
All commandlets inherit from UCommandlet which provides the basic infrastructure for running a commandlet. As a UObject it automatically gets registered so that it can be found at startup to run.
Typically a commandlet ends in “Commandlet” for discoverability.
Generally commandlets are editor-only and should go into an editor only project.
A commandlet has a Main method that receives commandline parameters and returns an integer exit code that you override. A non-zero exit code is an error and will appear at runtime (particularly relevant in build processes to know something went wrong).
Example:
#pragma once
#include "Commandlets/Commandlet.h"
#include "CommandletSample.generated.h"
UCLASS()
class UCommandletSampleCommandlet : public UCommandlet
{
GENERATED_BODY()
virtual int32 Main(const FString& Params) override;
};
The body of a commandlet is like a main function for a standard of C++.
You can run a commandlet from Visual Studio by modifying the properties of the project under Debugging→Command Arguments and add a -run= option for your commandlet.
Example:
From Commandline
To run the commandlet from the commandline (such as for Jenkins) you can run:
Commandlets are primarily about automating operations done over some or all assets.
The Asset Registry is the means by which one gets access to the list and operations available for assets in uasset and umap files.
NOTE: A significant limitation of the asset registry is it is limited in accessing assets that are sub-objects in other objects. For example a map can contain actors and those actors may not be returned when querying the asset registry. Instead it may be required to consider assets that can contain your desired asset.
Once you have your list of assets, you can retrieve them:
for (const FAssetData& Asset : Assets)
{
<Native Class>* Instance = Cast<Native Class>(Asset.GetAsset());
// DO STUFF HERE
}
Source Control
You can add, edit, and delete files from source control from commandlets as well.
Source control is available if it has been configured for the user in the project settings or via commandline settings. An example for Perforce is provided:
-SCCProvider=Perforce -P4Port=<your perforce server name and port> -P4User=<p4user> -P4Client=<p4client>
It is recommended to force update otherwise cached information may be used that can give erroneous results. There is a version of this method for getting the state of multiple files at once which is far more efficient than one at a time.
Adding, editing, or deleting consists of actions in the source control provider:
const ECommandResult::Type Result = SourceControlProvider->Execute(ISourceControlOperation::Create<FCheckOut>(), PackageToSave);
Saving Packages
Saving a package after modification does require some care.
There are two ways to save packages whether there is a root object (World in maps) or just the package itself.
We use functions like the following to save the package. NOTE: this is a good place to consolidate functionality across uses:
/**
* Helper function to save a package that may or may not be a map package
*
* @param Package The package to save
* @param Filename The location to save the package to
* @param KeepObjectFlags Objects with any these flags will be kept when saving even if unreferenced.
* @param ErrorDevice the output device to use for warning and error messages
* @param LinkerToConformAgainst
* @param optional linker to use as a base when saving Package; if specified, all common names, imports and exports
* in Package will be sorted in the same order as the corresponding entries in the LinkerToConformAgainst
* @return true if successful
*/
bool UGatherDialogCommandlet::SavePackageHelper(UPackage* Package, FString Filename)
{
Package->FullyLoad();
EObjectFlags KeepObjectFlags = RF_Standalone;
FOutputDevice* ErrorDevice = GWarn;
FLinkerNull* LinkerToConformAgainst = nullptr;
ESaveFlags SaveFlags = SAVE_None;
// look for a world object in the package (if there is one, there's a map)
UWorld* World = UWorld::FindWorldInPackage(Package);
bool bSavedCorrectly;
if (World)
{
bSavedCorrectly = GEditor->SavePackage(Package, World, RF_NoFlags, *Filename, ErrorDevice, LinkerToConformAgainst, false, true, SaveFlags);
}
else
{
bSavedCorrectly = GEditor->SavePackage(Package, NULL, KeepObjectFlags, *Filename, ErrorDevice, LinkerToConformAgainst, false, true, SaveFlags);
}
// return success
return bSavedCorrectly;
}
I have written a number of tools for navigating our data in Dauntless. One of these tools gathers data about how our assets are referenced and the Unreal Engine pak files they go into for streaming installs for some platforms.
What I wanted to be able to do is from a particular node in the tree of assets (tree representation of a directed graph with cycles) was to search for a subnode (breadth first search) that matched search criteria by either name and/or chunk it belonged to (for cross-chunk references). Given the results of my search through my tree nodes and the collection of objects I then needed to navigate to the corresponding TreeViewItem and scroll to it.
If a TreeViewItem is collapsed, its visual children TreeViewItem instances may not be generated yet. This could might be simplified further based on the ItemContainerGeneratorStatus but I think it helps illustrate the stages.
My nodes are all based on a single type, the template parameter T. For more heterogeneous situations a base class, interface, or even just object would suffice.
I spent an inordinate amount of time in Baldur’s Gate and sequels, Neverwinter Nights, and many others.
So I was delighted to see Pathfinder: Kingmaker coming out. I’ve been playing off and on and there is a bit of an adjustment. I think the most interesting detail is how easy it is to die. I don’t think this is a bad thing. It happens to be an adjustment to thinking that you shouldn’t expect to go wandering everywhere and expect it to adjust to your difficulty. If you get in over your head, you will suffer (looking at you Viscount Smoulderburn).