Creation games loadorder #3963

halgari · 2025-10-08T15:20:28Z

This implements plugin sorting for SkyrimSE/FO4. But right now tests don't pass as we need a way to efficiently test against game files and have a valid game hash database

…on, and refine file indexing logic.

- Added SQL macros for plugin load order management (`load_order_plugin_files` and `plugin_sort_order`). - Implemented `PluginLoadOrderVariety` for sorting and managing plugins. - Extended `SkyrimSE` with sort order support via `ISortOrderVariety`. - Introduced models and attributes (`PluginSortEntry`, `PluginReactiveSortItem`, `ModKeyAttribute`) to enhance plugin data representation. - Updated `NexusMods.MnemonicDB` packages to version `0.28.1` for compatibility. - Adjusted project files to include SQL resources and align with the new load order functionality.

- Replaced unused and commented-out code in `PluginsFile.Write` with a simplified implementation based on database queries to generate sorted plugin lists. - Refined SQL macros and added ordering in `plugin_sort_order`. - Updated `PluginLoadOrderVariety` to improve item sorting and persist logic, including ordering and deletion checks.

…ntrinsic file path handling. This was causing a bug where disabled files would never show up as disabled in these queries

…plementation to ensure compatibility with future file scanner changes.

…sort logic - Introduced `PluginLoadOrderIntegrationTests` to validate plugin sorting and load order management behaviors. - Updated `plugin_sort_order` SQL to include explicit `ORDER BY SortIndex` for consistent results. - Extended test framework with additional dependencies for plugin file handling.

Al12rs · 2025-10-09T07:51:54Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginSortEntry.cs

+{
+    private const string Namespace = "NexusMods.Games.CreationEngine.PluginSortEntry";
+
+    public static readonly ModKeyAttribute ModKey = new(Namespace, nameof(ModKey));


We should add a comment to clarify what ModKey refers to in this context, as it could easily be equivocated to mean an Id for the LoadoutItemGroup mod, rather than an Id for the plugin.

I'm considering whether to use different names and aliases on top of the Mutagen ModKey, to avoid equivocation since in our code domain Mod means something different.

Al12rs · 2025-10-09T07:58:31Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrderVariety.cs

+
+namespace NexusMods.Games.CreationEngine.LoadOrder;
+
+public class PluginLoadOrderVariety : ASortOrderVariety<


Should be named PluginSortOrderVariety for consistency

Suggested change

public class PluginLoadOrderVariety : ASortOrderVariety<

public class PluginSortOrderVariety : ASortOrderVariety<

Al12rs · 2025-10-09T08:05:02Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrderVariety.cs

+    public override async ValueTask<SortOrderId> GetOrCreateSortOrderFor(LoadoutId loadoutId, OneOf<LoadoutId, CollectionGroupId> parentEntity, CancellationToken token = default)
+    {
+        var optionalSortOrderId = GetSortOrderIdFor(parentEntity);
+        if (optionalSortOrderId.HasValue)
+            return optionalSortOrderId.Value;
+
+        token.ThrowIfCancellationRequested();
+
+        using var ts = Connection.BeginTransaction();
+        var newSortOrder = new SortOrder.New(ts)
+        {
+            LoadoutId = loadoutId,
+            ParentEntity = parentEntity,
+            SortOrderTypeId = SortOrderVarietyId.Value,
+        };
+
+        var commitResult = await ts.Commit();
+
+        var sortOrder = commitResult.Remap(newSortOrder);
+        return sortOrder;
+    }


We should move this into ASortOrderVariety directly.

Only reason RedMods have an override is that they have the extra RedModSortOrder model from previous iteration of the system. It's superfluous, but it was simpler to keep it rather than do a migration to remove it.

Al12rs · 2025-10-09T08:18:59Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrderVariety.cs

+                $"""
+                 SELECT ModKey, GroupName, GroupId, SortIndex
+                 FROM creation_engine.plugin_sort_order({Connection}) WHERE SortOrderId = {sortOrderId.Value}
+                 """)


As final cleanup, I moved all the queries out from the SortOrderVariety file into a separate file (RedModExtensions, but maybe PluginQueries.cs would be better here).

I think it makes it easier to find them and update them if we change something with the query system (e.g. we want to use something like Dapper to generate the SQL or some other change).

In general I like to keep the implementation details of how the data is accessed in the DB separate from the logic.

Al12rs · 2025-10-09T08:43:25Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrderVariety.cs

+        // Insert new items at the start, sorted by newest creation order (ModGroupId)
+        // For cyberpunk RedMods, lower index wins, so we add new items at the start
+        results.InsertRange(0, itemsToAdd);


New items should be added at the end of the sort order for Bethesda games, rather than at the start.

We want newly added items to win by default, and for bethesda games we need to put them at the end of the order.

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrderVariety.cs

Al12rs · 2025-10-09T09:39:35Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrder.sql

+       file.Size, 
+       file.ItemType, 
+       file.Id, 
+       regexp_extract(file.Path.Path, '([^/]+)$') ModKey,


Are you interested in the entire path or just the filename? I think you might need to specify to capture group 1 of the pattern if just looking for the filename, as it should default to capturing the entire string otherwise.

Suggested change

regexp_extract(file.Path.Path, '([^/]+)$') ModKey,

regexp_extract(file.Path.Path, '([^/]+)$', 1) ModKey,

Al12rs · 2025-10-09T09:54:54Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrder.sql

+       regexp_extract(file.Path.Path, '([^/]+)$') ModKey,
+       grp.Id GroupId,
+       grp.Name GroupName
+FROM synchronizer.WinningFiles(db) file


Only querying the winning plugins here is problematic because it is excluding all the plugins in disabled mods and collections.

Including plugins from disabled mods in the sorting has been an explicit design decision.
This was the solution Design came to when considering the issue of users being able to Toggle collections on and off.

If we don't include disabled items in the sorting, users would immediately lose all their sorting information upon disabling a collection.

Enabling the collection again would result in all the items getting sorted in random or alphabetical order.

Similarly, users don't want to have to reposition a plugin in the load order after having toggled the parent off and on again.

I would agree that users don't necessarily expect plugins from disabled mods to show up in the sorting view, but not including them here at the data layer would result in a very bad user experience.

MO2 has a system in place to remember the position of recently disabled items in the load order. I would need check again how the implementation worked (my impression was that it was more of a hack), but it is there to address the case of users toggling mods on and off.

I would start by including disabled items in the sorting, and I would instead look at options later to potentially filter out disabled items in the view proposed to users if we have an issue with too many plugins being shown.

Al12rs · 2025-10-09T10:10:50Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrderVariety.cs

+                    if (!x.ModId.HasValue)
+                        return model;
+
+                    model.ModGroupId = Optional<LoadoutItemGroupId>.Create(x.ModId!.Value);
+                    var loadoutData = new SortItemLoadoutData<SortItemKey<ModKey>>(
+                        model.Key,
+                        true, 
+                        x.GroupName, 
+                        LoadoutItemGroupId.From(x.ModId!.Value));
+
+                    model.LoadoutData = loadoutData;
+                    return model;


Here is a bit of an example of what I mentioned the other day regarding GameFiles not being Loadout Files.

It would be preferrable to be able to reference them in a homogeneous way like we do with other items in the loadout, either as LoadoutItems themselves, or by using some abstraction over LoadoutItems (which sounds less doable).

Not much to do for this PR specifically, but more of a note for how we want to handle game files in the future.

Al12rs · 2025-10-09T10:15:52Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrder.sql

+WHERE file.Path.Location = nma_fnv1a_hash_short('Game')
+  AND file.Path.Path SIMILAR TO '(?i)^Data/[^/]+\.(esp|esl|esm)$';


The rows of this query should be deterministically sorted.

Ideally items that have been added most recently to the loadout should appear at the bottom, in case of parity I would go by plugin alphabetical order.

We want to ensure that new entries appear at the bottom of the order.

Al12rs · 2025-10-09T10:30:19Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrder.sql

+         LEFT JOIN mdb_LoadoutItem() itm on itm.Id = file.Id
+         LEFT JOIN mdb_LoadoutITemGroup() grp on grp.Id = itm.Parent


I think we need to pass in the db parameter to mdb_LoadoutItem() and mdb_LoadoutITemGroup(), or we might be fetching from the most recent DB rather than the one we are interested in.

Al12rs · 2025-10-09T10:34:50Z

src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrder.sql

+CREATE MACRO creation_engine.plugin_sort_order(db) AS TABLE
+    SELECT sortOrder.Id SortOrderId, items.ModKey, sortItem.SortIndex, items.GroupId, items.GroupName 
+    FROM MDB_SORTORDER() sortOrder
+    INNER JOIN creation_engine.load_order_plugin_files(null) items ON items.Loadout = sortOrder.Loadout
+    INNER JOIN MDB_PLUGINSORTENTRY() sortItem ON sortItem.ParentSortOrder = sortOrder.Id AND sortItem.ModKey = items.ModKey


If this is used when writing Plugins.txt during synchronizer step, it needs to access the correct Db revision.

So MDB_SORTORDER(), MDB_PLUGINSORTENTRY() should take the db parameter as well I think.

Not sure what the null is doing in this case creation_engine.load_order_plugin_files(null), I would have expected that to also need to have the db parameter passed to it.

…st framework with new test projects for SkyrimSE. Update `FileHashesService` for in-memory file system compatibility and adjust NexusMods library package versions.

…image error handling, and enhance SkyrimSE path resolution with `DataPath`.

…`.nx` archives and update integration test constructor visibility.

…`PackManifestToNx` helpers to simplify logic and reduce code duplication.

… error handling for missing environment variable, and add developer documentation for running integration tests.

…appings for Linux and Windows

…n tests as part of the main tests

…mprove filtering logic

github-actions · 2025-10-22T02:40:17Z

This PR doesn't conflict with main anymore. It can be merged after all status checks have passed and it has been reviewed.

…nitions

… `System.IO.Compression` in `ManagedZipExtractor`

github-actions · 2025-10-22T07:45:44Z

This PR conflicts with main. You need to rebase the PR before it can be merged.

Al12rs · 2025-10-22T12:05:27Z

src/NexusMods.Networking.Steam/DepotChunkProvider.cs

-        throw new NotSupportedException();
+        using var owner = MemoryPool<byte>.Shared.Rent(buffer.Length);
+        var sliced = owner.Memory[..buffer.Length];
+        Task.Run(() => ReadChunkAsync(sliced, chunkIndex)).Wait();


Should use .GetAwaiter().GetResult() instead of .Wait() for better handling of exceptions.

Suggested change

Task.Run(() => ReadChunkAsync(sliced, chunkIndex)).Wait();

Task.Run(() => ReadChunkAsync(sliced, chunkIndex)).GetAwaiter().GetResult();

Al12rs · 2025-10-22T12:47:56Z

tests/Games/NexusMods.Games.IntegrationTestFramework/IntegrationTestDataSource.cs

+public class IntegrationTestDataSource : DependencyInjectionDataSourceAttribute<IServiceProvider>
+{
+    public override IServiceProvider CreateScope(DataGeneratorMetadata dataGeneratorMetadata)
+    {
+        throw new NotImplementedException();
+    }
+
+    public override object? Create(IServiceProvider scope, Type type)
+    {
+        throw new NotImplementedException();
+    }


What would this be for?

…n-loadorder

…riety.cs Co-authored-by: Al <[email protected]>

…n-loadorder

github-actions · 2025-10-22T16:25:22Z

This PR doesn't conflict with main anymore. It can be merged after all status checks have passed and it has been reviewed.

github-actions · 2025-11-05T13:21:44Z

This PR conflicts with main. You need to rebase the PR before it can be merged.

github-actions · 2025-11-13T10:17:03Z

This PR has been marked as stale due to inactivity.

halgari added 6 commits October 2, 2025 09:15

Enhance SkyrimSE and Fallout4 path handling, add Preferences locati…

5288857

…on, and refine file indexing logic.

Switch LEFT JOINs to INNER JOINs in Synchronizer queries and update i…

87dca4d

…ntrinsic file path handling. This was causing a bug where disabled files would never show up as disabled in these queries

Comment out untestable code in Ingest method and add placeholder im…

01d1f3a

…plementation to ensure compatibility with future file scanner changes.

github-project-automation bot added this to App Oct 8, 2025

github-project-automation bot moved this to Development review in App Oct 8, 2025

halgari marked this pull request as draft October 8, 2025 15:21

Implement steam CLI verb for creating depot images

bc9560b

Al12rs changed the title ~~Creation loadorder~~ Creation games loadorder Oct 9, 2025

Al12rs reviewed Oct 9, 2025

View reviewed changes

Al12rs requested changes Oct 9, 2025

View reviewed changes

halgari added 14 commits October 9, 2025 07:14

Remove ChunkedStream and IChunkedStreamSource, add integration te…

87fe348

…st framework with new test projects for SkyrimSE. Update `FileHashesService` for in-memory file system compatibility and adjust NexusMods library package versions.

First integration test passes

af30050

Improve game file validation for integration tests, add missing game …

33686a4

…image error handling, and enhance SkyrimSE path resolution with `DataPath`.

Add steam manifest pack-game CLI verb to pack Steam manifests into …

a61b664

…`.nx` archives and update integration test constructor visibility.

Refactor manifest processing by introducing TryResolveManifest and …

c04d95e

…`PackManifestToNx` helpers to simplify logic and reduce code duplication.

Expose GameImagesEnvVarName as a constant, improve integration test…

22ccfb0

… error handling for missing environment variable, and add developer documentation for running integration tests.

Add GitHub Actions workflow for running integration tests

7f0d50c

Fix the actions runner

79b5c7a

Improve SkyrimSE path resolution by adding OS-specific Steam folder m…

bd37113

…appings for Linux and Windows

Inline parts of the test workflow so that we no longer run integratio…

ab2553e

…n tests as part of the main tests

Merge branch 'main' into game-integration-tests

d79ed6a

Fixup some broken namespaces, merge main

aa29799

Update networking test workflow to exclude integration projects and i…

6e44d28

…mprove filtering logic

Add PowerShell installation step for Linux in test workflow

8c12b50

halgari had a problem deploying to test October 22, 2025 02:36 — with GitHub Actions Failure

halgari added 2 commits October 21, 2025 20:37

Remove the old test runners

837869e

merge the integration test code

4a4d83f

github-actions bot removed the status-needs-rebase Set by CI do not remove label Oct 22, 2025

halgari marked this pull request as ready for review October 22, 2025 02:40

halgari added 4 commits October 21, 2025 20:42

Normalize casing for runs_on labels in GitHub Actions workflow defi…

0181408

…nitions

Remove unnecessary .NET setup step from GitHub Actions workflow

765513c

Remove unused Startup class from CreationEngine tests and switch to…

5d81c31

… `System.IO.Compression` in `ManagedZipExtractor`

pull in the Nexus API key

906d0a5

github-actions bot added the status-needs-rebase Set by CI do not remove label Oct 22, 2025

Al12rs reviewed Oct 22, 2025

View reviewed changes

halgari and others added 7 commits October 22, 2025 07:11

Fix the UI tests crashing on startup

46a7590

Merge remote-tracking branch 'origin/creation-loadorder' into creatio…

1c007dc

…n-loadorder

Update schema verification file with new attributes and namespaces

c0e9cce

Merge in main

1d12c50

Update src/NexusMods.Games.CreationEngine/LoadOrder/PluginLoadOrderVa…

0993725

…riety.cs Co-authored-by: Al <[email protected]>

Disable a flakey GC test

d809da6

Merge remote-tracking branch 'origin/creation-loadorder' into creatio…

afeedbe

…n-loadorder

github-actions bot removed the status-needs-rebase Set by CI do not remove label Oct 22, 2025

halgari added 2 commits October 29, 2025 11:23

Merge branch 'main' into creation-loadorder

ae2ed97

Fix tests (locally)

d1a7d3b

github-actions bot added the status-needs-rebase Set by CI do not remove label Nov 5, 2025

github-actions bot added the Stale label Nov 13, 2025


		namespace NexusMods.Games.CreationEngine.LoadOrder;

		public class PluginLoadOrderVariety : ASortOrderVariety<

	public class PluginLoadOrderVariety : ASortOrderVariety<
	public class PluginSortOrderVariety : ASortOrderVariety<

	regexp_extract(file.Path.Path, '([^/]+)$') ModKey,
	regexp_extract(file.Path.Path, '([^/]+)$', 1) ModKey,

		WHERE file.Path.Location = nma_fnv1a_hash_short('Game')
		AND file.Path.Path SIMILAR TO '(?i)^Data/[^/]+\.(esp\|esl\|esm)$';

		LEFT JOIN mdb_LoadoutItem() itm on itm.Id = file.Id
		LEFT JOIN mdb_LoadoutITemGroup() grp on grp.Id = itm.Parent

	Task.Run(() => ReadChunkAsync(sliced, chunkIndex)).Wait();
	Task.Run(() => ReadChunkAsync(sliced, chunkIndex)).GetAwaiter().GetResult();

Creation games loadorder #3963

Are you sure you want to change the base?

Creation games loadorder #3963

Uh oh!

Conversation

halgari commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Al12rs Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Al12rs Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 22, 2025

Uh oh!

github-actions bot commented Oct 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 22, 2025

Uh oh!

github-actions bot commented Nov 5, 2025

Uh oh!

github-actions bot commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Al12rs Oct 9, 2025 •

edited

Loading

Al12rs Oct 9, 2025 •

edited

Loading