CI/CD Building a C# Project in Jenkins

The goal for this stage of development is to get our C# project listed in Jenkins and building. The next iteration will be adding testing.

This step follows the work from CI/CD Scanning GitHub for Repositories from Jenkins, where Jenkins is set up to scan for projects, but it now needs the files required to build.

Continue reading

CI/CD Scanning GitHub for Repositories from Jenkins

The next step is to get Jenkins to search GitHub for repositories to build.

I want all the projects in my GitHub repository that qualify to be built automatically, so if I add a new project with the right setup, it will automatically build it.

This can be achieved by having Jenkins scan your GitHub repository to gather all projects. GitHub supports pushing to Jenkins when repositories change, but that is more complicated when Jenkins is behind a firewall and cannot be directly contacted (we’ll be relying on polling in this case instead).

Continue reading

CI/CD Visual Studio Project Layout

As described in https://www.oneoddsock.com/2022/04/02/personal-ci-cd/, the goal of setting up the CI/CD automation is to:

  • Break up my more monolithic solutions into much smaller, more consumable components
  • Automate testing of my code
  • Improve potential for code sharing with others
  • Greater visibility of code/test status through reporting

To facilitate this, I’ll structure my Visual Studio solutions in GitHub so Jenkins can access them and run builds and tests.

The sample project is published on GitHub for reference at: https://github.com/FineRedMist/jenkins-project-sample-dotnet

For reference, I’m using Visual Studio 2022 Community Edition. I’m configuring the projects to use .NET 6.0.

Continue reading

Personal CI/CD

I have a number of personal projects with code scattered about, where if I want to include code from one project into another I generally either need to copy code or find more sophisticated ways to share.

After going through interviewing coop student candidates and seeing their projects and GitHub repositories (when provided) I realized that documenting the process and demonstrating the value you can get out of it would be substantial.

The motivation for me is to take my projects to a higher level of rigour and improve reuse. Furthermore, I want to automate builds, validation by content tests, notifications, etc.

I do use GitHub for storing my projects which are primarily in C# but I want to amp up my setup to another level. The feature I want are:

  1. Get my projects from GitHub
  2. Build them
  3. Run tests on them
  4. Post main branch builds to a NuGet repository
  5. Post main branch symbols to a symbol server
  6. Generate notifications to Slack (I have my own setup for the family) about build status.

I already have a machine I use locally for service-related work. Switching to Linux would be non-trivial due to some of the services being run.

So the following setup will be using Windows, but I don’t think it will be entirely dissimilar to Linux.

The software I’m looking at using for all of these are:

Anything else that comes up during setup will be documented along the way.

Unreal Engine & Localization

The Unreal Engine has some rather nifty functionality for dealing with localization. We have also made numerous modifications to facilitate our workflow, but I’ll stick to what is in vanilla Unreal (this is based on 4.25).

There are three different string types to use within Unreal:

  • FString: These are traditional strings most people are used to. When creating a new string it will allocate memory to store the data. Equality comparisons need to check the actual string data.
  • FName: When you create a name, it is stored in a table in memory permanently (there is no clean up of unused names). Two FNames with the same value point to the same index, which makes equality comparisons very fast: they just compare indices. But the memory usage can grow without bound.
  • FText: These are localized strings that go through a process of being gathered, translated, and having their values stored in a separate file called a locres file. When evaluating an FText, there is a lookup for the actual value to use.

Epic provides a tokenization system using curly braces, { and }, where you can provide parameters (either positional or named) for an FText format string.

However, we added another one using square brackets, [ & ], where we would take the token within and look up the value as an FName in various tables that were provided to look up information.

Continue reading

UE4: How To Write a Commandlet

Summary

A commandlet in Unreal is a mini-program that can be run from the commandline typically for automation.

A couple of significant examples are:

  • Resave Packages: resaves packages, typically used to resave older packages to bring them up to date.
  • Fixup Redirects: when assets are moved, they leave redirectors behind so that assets that reference them can be updated. Fixup Redirects updates those assets referencing the redirectors, then removes the redirectors that are no longer used.
  • Cooking: convert content into a format more optimized for target platforms.
  • Localization Pipeline: processes text in the game for translation and retrieves translations to include into the game.

How to Create A Commandlet

All commandlets inherit from UCommandlet which provides the basic infrastructure for running a commandlet. As a UObject it automatically gets registered so that it can be found at startup to run.

Typically a commandlet ends in “Commandlet” for discoverability.

Generally commandlets are editor-only and should go into an editor only project.

A commandlet has a Main method that receives commandline parameters and returns an integer exit code that you override. A non-zero exit code is an error and will appear at runtime (particularly relevant in build processes to know something went wrong).

Example:

#pragma once

#include "Commandlets/Commandlet.h"
#include "CommandletSample.generated.h"

UCLASS()
class UCommandletSampleCommandlet : public UCommandlet
{
    GENERATED_BODY()

    virtual int32 Main(const FString& Params) override;
};

The body of a commandlet is like a main function for a standard of C++.

#include "CommandletSample.h"

int32 UCommandletSampleCommandlet::Main(const FString& Params)
{
    return 0;
}

Running a Commandlet

From Visual Studio

You can run a commandlet from Visual Studio by modifying the properties of the project under Debugging→Command Arguments and add a -run= option for your commandlet.

Example:

From Commandline

To run the commandlet from the commandline (such as for Jenkins) you can run:

d:\p4>Engine\Binaries\Win64\UE4Editor-Cmd.exe <Your Project Name> -run=CommandletSample

Parsing Parameters

Commandlets have member functions to assist with parsing parameters by partitioning parameters into tokens and switches:

TArray<FString> Tokens, Switches;
ParseCommandLine(*Params, Tokens, Switches);

To get strings, booleans, and integers, you can use the collection of parsing methods in FParse such as:

bool Value = false;
if(FParse::Bool(*Switches[0], TEXT("boolparam"), &Value)
    && Value)
{
    // Do stuff
}

Output

Output for commandlets is done via UE_LOG. To do so you can declare a log category locally for your commandlet as follows:

DEFINE_LOG_CATEGORY_STATIC(LogCommandletSample, Log, All);

From there you can log as follows:

UE_LOG(LogCommandletSample, Display, TEXT("Found: %s"), *Instance->GetPathName());

Finding & Loading Assets

Commandlets are primarily about automating operations done over some or all assets.

The Asset Registry is the means by which one gets access to the list and operations available for assets in uasset and umap files.

NOTE: A significant limitation of the asset registry is it is limited in accessing assets that are sub-objects in other objects. For example a map can contain actors and those actors may not be returned when querying the asset registry. Instead it may be required to consider assets that can contain your desired asset.

Required headers:

#include "AssetRegistryModule.h"
#include "Developer/AssetTools/Public/AssetToolsModule.h"

Getting access to the Asset Registry:

auto& AssetRegistryModule = FModuleManager::Get().LoadModuleChecked<FAssetRegistryModule>(TEXT("AssetRegistry"));
auto& AssetRegistry = AssetRegistryModule.Get();

To ensure the Asset Registry is up to date use:

// Get all the asset in our content folders
AssetRegistry.SearchAllAssets(true);
while (AssetRegistry.IsLoadingAssets())
{
    AssetRegistry.Tick(1.0f);
}

Then you can query for assets by class (note there are a lot  of other methods in the asset registry to search for assets):

TArray<FAssetData> Assets;
UClass* TargetClass = <Native Class>::StaticClass();
AssetRegistry.GetAssetsByClass(TargetClass->GetFName(), Assets, true);

Once you have your list of assets, you can retrieve them:

for (const FAssetData& Asset : Assets)
{
    <Native Class>* Instance = Cast<Native Class>(Asset.GetAsset());

    // DO STUFF HERE
}

Source Control

You can add, edit, and delete files from source control from commandlets as well.

Source control is available if it has been configured for the user in the project settings or via commandline settings. An example for Perforce is provided:

-SCCProvider=Perforce -P4Port=<your perforce server name and port> -P4User=<p4user> -P4Client=<p4client>

Required headers:

#include "ISourceControlProvider.h"
#include "ISourceControlModule.h"
#include "SourceControlOperations.h"

Verify that source control is available:

SourceControlProvider = &ISourceControlModule::Get().GetProvider();
SourceControlProvider->Init();

if (!SourceControlProvider->IsEnabled())
{
    UE_LOG(LogCommandletSample, Error, TEXT("Source control not initialized. Exiting..."));
    return 2;
}

You can query the source control state of the file using:

FSourceControlStatePtr SourceControlState = SourceControlProvider->GetState(PackageToSave, EStateCacheUsage::ForceUpdate);

It is recommended to force update otherwise cached information may be used that can give erroneous results. There is a version of this method for getting the state of multiple files at once which is far more efficient than one at a time.

Adding, editing, or deleting consists of actions in the source control provider:

const ECommandResult::Type Result = SourceControlProvider->Execute(ISourceControlOperation::Create<FCheckOut>(), PackageToSave);

Saving Packages

Saving a package after modification does require some care.

There are two ways to save packages whether there is a root object (World in maps) or just the package itself.

We use functions like the following to save the package. NOTE: this is a good place to consolidate functionality across uses:

/**
* Helper function to save a package that may or may not be a map package
*
* @param Package The package to save
* @param Filename The location to save the package to
* @param KeepObjectFlags Objects with any these flags will be kept when saving even if unreferenced.
* @param ErrorDevice the output device to use for warning and error messages
* @param LinkerToConformAgainst
* @param optional linker to use as a base when saving Package; if specified, all common names, imports and exports
* in Package will be sorted in the same order as the corresponding entries in the LinkerToConformAgainst

* @return true if successful
*/
bool UGatherDialogCommandlet::SavePackageHelper(UPackage* Package, FString Filename)
{
    Package->FullyLoad();
    EObjectFlags KeepObjectFlags = RF_Standalone;
    FOutputDevice* ErrorDevice = GWarn;
    FLinkerNull* LinkerToConformAgainst = nullptr;
    ESaveFlags SaveFlags = SAVE_None;

    // look for a world object in the package (if there is one, there's a map)
    UWorld* World = UWorld::FindWorldInPackage(Package);
    bool bSavedCorrectly;

    if (World)
    {
      bSavedCorrectly = GEditor->SavePackage(Package, World, RF_NoFlags, *Filename, ErrorDevice, LinkerToConformAgainst, false, true, SaveFlags);
    }
    else
    {
      bSavedCorrectly = GEditor->SavePackage(Package, NULL, KeepObjectFlags, *Filename, ErrorDevice, LinkerToConformAgainst, false, true, SaveFlags);
    }

    // return success
    return bSavedCorrectly;
}

Other Resources

Navigating TreeViewItems in C#

I have written a number of tools for navigating our data in Dauntless. One of these tools gathers data about how our assets are referenced and the Unreal Engine pak files they go into for streaming installs for some platforms.

What I wanted to be able to do is from a particular node in the tree of assets (tree representation of a directed graph with cycles) was to search for a subnode (breadth first search) that matched search criteria by either name and/or chunk it belonged to (for cross-chunk references). Given the results of my search through my tree nodes and the collection of objects I then needed to navigate to the corresponding TreeViewItem and scroll to it.

If a TreeViewItem is collapsed, its visual children TreeViewItem instances may not be generated yet. This could might be simplified further based on the ItemContainerGenerator Status but I think it helps illustrate the stages.

My nodes are all based on a single type, the template parameter T. For more heterogeneous situations a base class, interface, or even just object would suffice.

Code follows.

Continue reading

Pathfinder: Kingmaker

I spent an inordinate amount of time in Baldur’s Gate and sequels, Neverwinter Nights, and many others.

So I was delighted to see Pathfinder: Kingmaker coming out. I’ve been playing off and on and there is a bit of an adjustment. I think the most interesting detail is how easy it is to die. I don’t think this is a bad thing. It happens to be an adjustment to thinking that you shouldn’t expect to go wandering everywhere and expect it to adjust to your difficulty. If you get in over your head, you will suffer (looking at you Viscount Smoulderburn).

Continue reading