code and documentation improvements

Author: ropeless

LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Barry Drake
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

docs/1_01_front_matter.md

@@ -1,6 +1,6 @@
 # _front matter_
 
-Synthorus version 0.0.0a9, built 2025-11-19 20:28:49 (AUS Eastern Summer Time).
+Synthorus version 0.0.0a10, built 2025-11-20 08:35:07 (AUS Eastern Summer Time).
 A pre-release version.
 
 These pages form a reference for the software known as Synthorus.
@@ -9,7 +9,7 @@ They are intended for users and developers of the software to help orient them
 to key design points and functionality of the software.
 
 Not all sections of this documentation will be relevant for all readers. As well as covering top-level
-organization and usage of Synthorus, these notebooks also covers some important implementation details.
+organization and usage of Synthorus, these notebooks also cover some important implementation details.
 
 This documentation is available online at
 [synthorus.readthedocs.io](https://synthorus.readthedocs.io/).

docs/1_01_front_matter_template.md

@@ -9,7 +9,7 @@ They are intended for users and developers of the software to help orient them
 to key design points and functionality of the software.
 
 Not all sections of this documentation will be relevant for all readers. As well as covering top-level
-organization and usage of Synthorus, these notebooks also covers some important implementation details.
+organization and usage of Synthorus, these notebooks also cover some important implementation details.
 
 This documentation is available online at
 [synthorus.readthedocs.io](https://synthorus.readthedocs.io/).

docs/1_03_installation.ipynb

@@ -37,10 +37,10 @@
      "start_time": "2025-11-10T01:56:31.203505Z"
     },
     "execution": {
-     "iopub.execute_input": "2025-11-19T09:29:04.416172Z",
-     "iopub.status.busy": "2025-11-19T09:29:04.415200Z",
-     "iopub.status.idle": "2025-11-19T09:29:04.617235Z",
-     "shell.execute_reply": "2025-11-19T09:29:04.617235Z"
+     "iopub.execute_input": "2025-11-19T21:35:22.665359Z",
+     "iopub.status.busy": "2025-11-19T21:35:22.665359Z",
+     "iopub.status.idle": "2025-11-19T21:35:22.865666Z",
+     "shell.execute_reply": "2025-11-19T21:35:22.865666Z"
     }
    },
    "outputs": [
@@ -51,10 +51,10 @@
       "Entity: patient ['_id_', '_count_', 'age']\n",
       "\n",
       "patient [('_id_', 1), ('_count_', 1), ('age', 'old')]\n",
-      "patient [('_id_', 2), ('_count_', 2), ('age', 'young')]\n",
+      "patient [('_id_', 2), ('_count_', 2), ('age', 'middle_aged')]\n",
       "patient [('_id_', 3), ('_count_', 3), ('age', 'old')]\n",
       "patient [('_id_', 4), ('_count_', 4), ('age', 'old')]\n",
-      "patient [('_id_', 5), ('_count_', 5), ('age', 'old')]\n",
+      "patient [('_id_', 5), ('_count_', 5), ('age', 'young')]\n",
       "\n",
       "Finished\n"
      ]

docs/2_01_background.ipynb

@@ -15,7 +15,7 @@
     "- synthetic data\n",
     "- random variables\n",
     "- probabilistic graphical models\n",
-    "- datasets and data sources\n",
+    "- datasets and datasources\n",
     "- cross-tables\n",
     "- protecting privacy\n",
     "- entities and relationships\n",
@@ -134,7 +134,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Datasets and Data Sources ##\n",
+    "## Datasets and Datasources ##\n",
     "\n",
     "Within Synthorus a _dataset_ represents reference data from a system (the system to represent with synthetic data).\n",
     "Although in-principal synthetic data can be used as a dataset, we generally reserve _dataset_ to mean reference data for training PGMs, and reserve _synthetic data_\n",
@@ -149,11 +149,16 @@
     "\n",
     "Within Synthorus dataset values and weights are immutable and cannot be updated.\n",
     "\n",
-    "A _data source_ is merely a specification for how to create a dataset. Example data sources include:\n",
-    "- a string in CSV format\n",
-    "- a parquet file\n",
-    "- a database query\n",
-    "- a function.\n"
+    "A _dataset spec_ is a specification for how to access a dataset. Synthorus provides dataset spec classes for different kinds of datasets, including:\n",
+    "- CSV file or inline text (with generalised separators and line spacing)\n",
+    "- Table Builder file or inline text (as created by the [Australian Bureau of Statistics](https://www.abs.gov.au/statistics/microdata-tablebuilder/tablebuilder))\n",
+    "- Pickled Pandas DataFrame object, as a file\n",
+    "- Parquet file\n",
+    "- Feather file\n",
+    "- A database SQL query (using ODBC or Postgres)\n",
+    "- A mathematical function (with a defined domain).\n",
+    "\n",
+    "A _datasource spec_ (or just _datasource_) is composed of a dataset spec and other parameters. Those parameters support privacy protection and the statistical integrity of data.\n"
    ]
   },
   {
@@ -280,7 +285,7 @@
     "\n",
     "In data modeling, relationship cardinality describes how many instances of one entity can be related to instances of another entity. In general, each side of the relationship cardinality is a range. E.g., the cardinality of entities X and Y might be _a_-_b_:_c_-_d_, meaning that for any instance of Y the minimum number of X instances is _a_ and the maximum is _b_. Similarly, for any instance of X the minimum number of Y instances is _c_ and the maximum is _d_. If the minimum and maximum of a range is the same value, then just that value is written.\n",
     "\n",
-    "If a range can be any value, then an asterisk (\\*) can be used,. If the minimum of a range is greater than zero, then only the upper value is marked with an asterisk.\n",
+    "If a range can be any value, then an asterisk (\\*) can be used. If the minimum of a range is greater than zero, then only the upper value is marked with an asterisk.\n",
     "\n",
     "Here are some common examples.\n",
     "\n",
@@ -429,7 +434,7 @@
     "\n",
     "For example, the utility of a dataset may depend only on the statistical relationships between small subsets of random variables. Or it may only be concerned with marginal distributions of random variables, or correlations between pairs of random variables.\n",
     "\n",
-    "Synthorus avoids the problem of needing large numbers of synthetic data records by using the joint probability distribution from the probabilistic models of the simulator. The probabilistic models guarantee that as the number of synthetic data records increases, the emprical probability distribution approaches the models distribution. This has three advantages.\n",
+    "Synthorus avoids the problem of needing large numbers of synthetic data records by using the joint probability distribution from the probabilistic models of the simulator. The probabilistic models guarantee that as the number of synthetic data records increases, the empirical probability distribution approaches the models distribution. This has three advantages.\n",
     "1. It removes effects of sampling errors in the evaluation of utility.\n",
     "2. Evaluation can be performed without any synthetic data being generated.\n",
     "3. Synthorus probabilistic models can be directly and efficiently queried for most standard probabilities and statistics.\n",

docs/2_02_workflow.ipynb

@@ -31,14 +31,14 @@
     "Making a model involves providing a model specification. A model specification describes:\n",
     "- model meta-data\n",
     "- privacy protection and other parameter values\n",
-    "- data sources for reference data\n",
+    "- datasources for reference data\n",
     "- what are the random variables and their states\n",
     "- what are the cross-tables\n",
     "- what are the entities, their fields, and relationships\n",
     "- simulation parameters and values.\n",
     "\n",
     "The steps to make a model are as follows.\n",
-    "1. Datasets specified in the model specification will be loaded based on the specified data sources.\n",
+    "1. Datasets specified in the model specification will be loaded based on the specified datasources.\n",
     "2. Clean cross-tables are computed.\n",
     "3. Privacy protection is applied to create noisy cross-tables.\n",
     "4. The noisy cross-tables are used to create a probabilistic graphical model (PGM) for each entity.\n",
@@ -47,7 +47,7 @@
     "The result of making a model is a collection of files which are placed into a model definition folder. The files are:\n",
     "1. a JSON definition of the model ('model_spec.json')\n",
     "2. a JSON definition of an index of model components ('model_index.json')\n",
-    "2. a JSON definition of the synthetic data simulator ('simulator_spec.json')\n",
+    "3. a JSON definition of the synthetic data simulator ('simulator_spec.json')\n",
     "4. Compiled Knowledge PGMs for each entity ('pgms/{_entity_}.py')\n",
     "5. noisy cross-tables, if requested for saving ('noisy_cross_tables/{_cross_table_}.pk')\n",
     "6. clean cross-tables, if requested for saving ('clean_cross_tables/{_cross_table_}.pk')\n",
@@ -121,10 +121,10 @@
      "start_time": "2025-11-11T03:49:36.477856Z"
     },
     "execution": {
-     "iopub.execute_input": "2025-11-19T09:29:28.778980Z",
-     "iopub.status.busy": "2025-11-19T09:29:28.778980Z",
-     "iopub.status.idle": "2025-11-19T09:29:28.788590Z",
-     "shell.execute_reply": "2025-11-19T09:29:28.788590Z"
+     "iopub.execute_input": "2025-11-19T21:35:50.091245Z",
+     "iopub.status.busy": "2025-11-19T21:35:50.091245Z",
+     "iopub.status.idle": "2025-11-19T21:35:50.100528Z",
+     "shell.execute_reply": "2025-11-19T21:35:50.100528Z"
     }
    },
    "outputs": [
@@ -190,10 +190,10 @@
      "start_time": "2025-11-11T03:49:36.499060Z"
     },
     "execution": {
-     "iopub.execute_input": "2025-11-19T09:29:28.790595Z",
-     "iopub.status.busy": "2025-11-19T09:29:28.790595Z",
-     "iopub.status.idle": "2025-11-19T09:29:29.013056Z",
-     "shell.execute_reply": "2025-11-19T09:29:29.013056Z"
+     "iopub.execute_input": "2025-11-19T21:35:50.102532Z",
+     "iopub.status.busy": "2025-11-19T21:35:50.102532Z",
+     "iopub.status.idle": "2025-11-19T21:35:50.320792Z",
+     "shell.execute_reply": "2025-11-19T21:35:50.320792Z"
     }
    },
    "outputs": [
@@ -233,23 +233,23 @@
       "    }\n",
       "  },\n",
       "  \"rvs\": {\n",
-      "    \"X\": {\n",
+      "    \"Z\": {\n",
       "      \"states\": \"infer_distinct\",\n",
       "      \"ensure_none\": false\n",
       "    },\n",
       "    \"Y\": {\n",
       "      \"states\": \"infer_distinct\",\n",
       "      \"ensure_none\": false\n",
       "    },\n",
-      "    \"Z\": {\n",
+      "    \"X\": {\n",
       "      \"states\": \"infer_distinct\",\n",
       "      \"ensure_none\": false\n",
       "    }\n",
       "  },\n",
       "  \"crosstabs\": {\n",
-      "    \"_X\": {\n",
+      "    \"_Z\": {\n",
       "      \"rvs\": [\n",
-      "        \"X\"\n",
+      "        \"Z\"\n",
       "      ],\n",
       "      \"datasource\": \"xyz\",\n",
       "      \"epsilon\": 0.1,\n",
@@ -265,9 +265,9 @@
       "      \"min_cell_size\": 0.0,\n",
       "      \"max_add_rows\": 1000000\n",
       "    },\n",
-      "    \"_Z\": {\n",
+      "    \"_X\": {\n",
       "      \"rvs\": [\n",
-      "        \"Z\"\n",
+      "        \"X\"\n",
       "      ],\n",
       "      \"datasource\": \"xyz\",\n",
       "      \"epsilon\": 0.1,\n",
@@ -281,17 +281,17 @@
       "      \"count_field_name\": \"_count_\",\n",
       "      \"foreign_field_name\": null,\n",
       "      \"fields\": {\n",
-      "        \"X\": {\n",
+      "        \"Z\": {\n",
       "          \"type\": \"sample\",\n",
-      "          \"rv_name\": \"X\"\n",
+      "          \"rv_name\": \"Z\"\n",
       "        },\n",
       "        \"Y\": {\n",
       "          \"type\": \"sample\",\n",
       "          \"rv_name\": \"Y\"\n",
       "        },\n",
-      "        \"Z\": {\n",
+      "        \"X\": {\n",
       "          \"type\": \"sample\",\n",
-      "          \"rv_name\": \"Z\"\n",
+      "          \"rv_name\": \"X\"\n",
       "        }\n",
       "      },\n",
       "      \"cardinality\": [],\n",
@@ -320,7 +320,7 @@
    "source": [
     "This spec file defines one datasource, \"xyz\" that includes three random variables, \"X\", \"Y\" and \"Z\".\n",
     "\n",
-    "System random variable can be explicitly defined in a spec file using an \"rvs\" section. The demo `spec_tiny.py` does not include this section so one is internally created with a random variable defined for all random variables seen in all data sources. Each random variable definition needs to define the possible states of the random variable. Including `states: infer_distinct` at the top level of spec file dictionary means that `states: infer_distinct` will be inherited for every random variable definition. The value `infer_distinct` means that the possible values of a random variable will be defined as \"all distinct values seen for that random variable in the datasources.\"\n",
+    "System random variables can be explicitly defined in a spec file using section \"rvs\". The demo `spec_tiny.py` does not include this section so one is internally created with a random variable defined for all random variables seen in all datasources. Each random variable definition needs to define the possible states of the random variable. Including `states: infer_distinct` at the top level of spec file dictionary means that `states: infer_distinct` will be inherited for every random variable definition. The value `infer_distinct` means that the possible values of a random variable will be defined as \"all distinct values seen for that random variable in the datasources.\"\n",
     "\n"
    ]
   },
@@ -354,10 +354,10 @@
      "start_time": "2025-11-11T03:49:37.115213Z"
     },
     "execution": {
-     "iopub.execute_input": "2025-11-19T09:29:29.015061Z",
-     "iopub.status.busy": "2025-11-19T09:29:29.015061Z",
-     "iopub.status.idle": "2025-11-19T09:29:29.019192Z",
-     "shell.execute_reply": "2025-11-19T09:29:29.019192Z"
+     "iopub.execute_input": "2025-11-19T21:35:50.322911Z",
+     "iopub.status.busy": "2025-11-19T21:35:50.322911Z",
+     "iopub.status.idle": "2025-11-19T21:35:50.327511Z",
+     "shell.execute_reply": "2025-11-19T21:35:50.327511Z"
     }
    },
    "outputs": [
@@ -394,10 +394,10 @@
      "start_time": "2025-11-11T03:49:37.127892Z"
     },
     "execution": {
-     "iopub.execute_input": "2025-11-19T09:29:29.021198Z",
-     "iopub.status.busy": "2025-11-19T09:29:29.021198Z",
-     "iopub.status.idle": "2025-11-19T09:29:29.686709Z",
-     "shell.execute_reply": "2025-11-19T09:29:29.686709Z"
+     "iopub.execute_input": "2025-11-19T21:35:50.329516Z",
+     "iopub.status.busy": "2025-11-19T21:35:50.329516Z",
+     "iopub.status.idle": "2025-11-19T21:35:50.992145Z",
+     "shell.execute_reply": "2025-11-19T21:35:50.992145Z"
     }
    },
    "outputs": [