Release Testing System

The GeoDMS release testing system is a Python-based framework for running automated regression tests against different versions of the GeoDMS software. The testing framework is implemented in the GeoDMS-Test repository and allows developers to verify that new versions don't break existing functionality.

overview

The release testing system consists of:

• batch/full.py - the main test runner script
• profiler/regression.py - helper functions for experiment management (in the main GeoDMS repository)
• profiler/profiler.py - profiling utilities (in the main GeoDMS repository)

The system runs a comprehensive suite of tests including operator tests, storage tests, network model tests, land use model tests, and GUI tests.

adding a new version to the release testing system

To test a new version of the GeoDMS, you need to:

1. install the GeoDMS version

Install the GeoDMS version you want to test. The installer places the executables in:

%ProgramFiles%/ObjectVision/GeoDms{VERSION}

For example, for version 18.0.3:

C:\Program Files\ObjectVision\GeoDms18.0.3

2. configure your machine (one-time)

Copy the gitignored settings template and adjust paths for your drives / user:

cd C:\path\to\GeoDMS-Test\batch
copy local_settings.json.template local_settings.json
notepad local_settings.json

See configuring local machine parameters below for the schema.

3. run the test script

Navigate to the batch folder in the GeoDMS-Test repository and run the test script with the version parameter:

cd C:\path\to\GeoDMS-Test\batch
python full.py -version 18.0.3        # test an installed release
python full.py -version local          # test the just-built CMake Release tree

re-running tests for an existing version

To re-run the tests for a version that has already been tested (e.g., after fixing a bug or updating test data):

1. remove the resultsfolder for the geodms test that you would like to rerun. For geodms 19.3.0 this means removing the folder 19_3_0_x64_SF_S1S2S3_OVSRV05 or remove the specific .bin coinciding with the tests that you specifically want to rerun.

2. run the same command again with the same version number:

python full.py -version 18.0.3

3. To preserve previous results, rename or move the existing results folder before re-running:

# Rename previous results
ren "results\18.0.3_x64_S1_S2_S3" "18.0.3_x64_S1_S2_S3_old"
# Then re-run
python full.py -version 18.0.3

optional parameters

The test script accepts several optional parameters for controlling multithreading behavior:

Parameter	Description	Default	Options
-version	GeoDMS version number	18.0.3	Any installed version (e.g., 17.4.6, 18.0.0)
-MT1	Multithreading setting 1	S1	S1 (single) or C1 (concurrent)
-MT2	Multithreading setting 2	S2	S2 (single) or C2 (concurrent)
-MT3	Multithreading setting 3	S3	S3 (single) or C3 (concurrent)

Example with all parameters:

python full.py -version 18.0.3 -MT1 C1 -MT2 C2 -MT3 C3

how the version lookup works

The get_geodms_paths function in full.py constructs the paths to the GeoDMS executables based on the version string. There are two modes:

def get_geodms_paths(version:str) -> dict:
    s = _load_local_settings()
    geodms_paths = {}
    geodms_paths["GeoDmsPlatform"] = "x64"
    if version == "local":
        geodms_paths["GeoDmsPath"] = f"\"{s['LocalBuildDir']}\""
    else:
        geodms_paths["GeoDmsPath"] = f"\"{os.path.expandvars(
            f"%ProgramFiles%/ObjectVision/GeoDms{version}")}\""
    geodms_paths["GeoDmsProfilerPath"]   = f"{s['ProfilerDir']}/profiler.py"
    geodms_paths["GeoDmsRegressionPath"] = f"{s['ProfilerDir']}/regression.py"
    geodms_paths["GeoDmsRunPath"]    = f"{geodms_paths['GeoDmsPath']}/GeoDmsRun.exe"
    geodms_paths["GeoDmsGuiQtPath"]  = f"{geodms_paths['GeoDmsPath']}/GeoDmsGuiQt.exe"
    return geodms_paths

• -version 18.0.3 (or any other "X.Y.Z" string) resolves to the installed GeoDms{version} folder under %ProgramFiles%/ObjectVision/.
• -version local resolves to the LocalBuildDir configured in local_settings.json — handy for testing the working tree without going through the NSIS installer.

prerequisites

Before running the tests, ensure:

1. Python 3.x is installed and available in your PATH
2. GeoDMS version is either installed via the official installer (for numeric versions) or built locally (for -version local)
3. Source data directories are configured in batch/local_settings.json:
   • RegressionTestsSourceDataDir — location of regression test data
   • LocalDataDir — local data directory for test outputs
4. Project snapshots are available in the prj_snapshots subdirectory of RegressionTestsSourceDataDir

configuring local machine parameters

Machine-specific paths live in batch/local_settings.json, which is gitignored so each working copy has its own without conflicting with others. A committed local_settings.json.template shows the expected schema and sane defaults.

one-time setup per working copy

cd C:\path\to\GeoDMS-Test\batch
copy local_settings.json.template local_settings.json
notepad local_settings.json   # adjust drive letters / user names

Example local_settings.json:

{
    "_comment": "OVSRV10 — Maarten's dev box",
    "RegressionTestsSourceDataDir":    "F:/SourceData/RegressionTests",
    "RegressionTestsAltSourceDataDir": "F:/SourceData",
    "LocalDataDir":                    "C:/LocalData",
    "ProfilerDir":                     "C:/dev/GeoDMS_2026/profiler",
    "LocalBuildDir":                   "C:/dev/GeoDMS_2026/build/windows-x64-release/bin"
}

resolution order

Each setting is resolved in this priority order (highest wins):

1. Environment variable with the same name as the JSON key (e.g. LocalDataDir)
2. local_settings.json in batch/
3. Built-in default in _BUILT_IN_DEFAULTS (top of full.py)

So for one-off overrides without editing the file:

$env:LocalDataDir = "D:\TestRunData"
python full.py -version local

test results

Test results are stored in a results folder with a naming convention:

{version}_{platform}_{MT1}_{MT2}_{MT3}

For example: 18.0.3_x64_S1_S2_S3

Results include:

• Log files for each test
• Indicator result files comparing expected vs actual values
• File comparison results for storage tests

available tests

The full regression test suite includes:

Test ID	Description
t010	Operator test - tests all GeoDMS operators
t050	Storage write (Shapefile polygon)
t060	BAG Snapshot (GeoPackage)
t100	Network connect
t101	Network OD PC4 dense
t102	Network OD PC6 sparse
t151	BL-RD coordinate conversion
t200	Grid Poly2Grid
t300	XML read/parse
t301	BAG residential type
t405	NetworkModel PBL (prepare, run, indicator)
t410	NetworkModel EU
t611	LUS demo 2023
t641	RSopen regression test
t710	2UP model
t720	2BURP model
t810	ValLuisa Czech
t910	CUSA2 Africa
t1630	GUI expand test
t1640	Value info test
t1642	Value info group_by
t1742	Command statistics

adding new tests

To add a new test to the regression suite, add an entry in the get_experiments() function:

regression.add_exp(
    exps, 
    name=f"{result_folder_name}__t{NUMBER}_{TEST_NAME}",
    cmd=f"{geodms_paths['GeoDmsRunPath']} /L{result_paths['results_log_folder']}/{TEST_NAME}.txt /{MT1} /{MT2} /{MT3} {path_to_dms} {item_to_calculate}",
    exp_fldr=f"{result_paths['results_folder']}",
    env=env_vars,
    log_fn=f"{result_paths['results_log_folder']}/{TEST_NAME}.txt",
    indicator_results_file=f"{result_paths['results_folder']}/{TEST_NAME}.txt"
)

troubleshooting

version not found

If you get an error that the version is not found, verify:

1. The GeoDMS is installed in the expected location
2. The version string matches the folder name exactly (e.g., "18.0.3" not "v18.0.3")

missing source data

Ensure the regression test source data is available at the configured paths. Contact the GeoDMS team for access to the test data repository.

permission errors

Run the test script with administrator privileges if you encounter permission issues writing to the output directories.

see also

• Debugging - debugging the GeoDMS
• Compiling the GeoDMS - building from source
• Unit testing - Boost test framework for unit tests
• GeoDMS-Test repository - test repository on GitHub