Merge pull request #11 from underworldcode/development

Development - synchronise changes to documentation / JOSS

Author: lmoresi

docs/joss-paper/paper.md

@@ -67,6 +67,8 @@ Users of `underworld3` typically develop python scripts within `jupyter` noteboo
 
 # Statement of need
 
+Typical problems in geodynamics usually require computing material deformation, damage evolution, and interface tracking in the large-deformation limit. These are typically not well supported by standard engineering finite element simulation codes. Underworld is a python software framework that is intended to solve geodynamics problems that sit at the interface between computational fluid mechanics and solid mechanics (often known as *complex fluids*). It does so by putting Lagrangian and Eulerian variables on an equal footing at both the user and computational levels.
+
 Underworld is built around a general, symbolic partial differential equation solver but provides template forms to solve common geophysical fluid dynamics problems such as the Stokes equation for mantle convection, subduction-zone evolution, lithospheric deformation, glacial isostatic adjustment, ice flow; Navier-Stokes equations for finite Prandtl number fluid flow and short-timescale, viscoelastic deformation; and Darcy Flow for porous media problems including groundwater flow and contaminant transport.
 
 These problems have a number of defining characteristics:  geomaterials are non-linear, viscoelastic/plastic and have a propensity for strain-dependent softening during deformation; strain localisation is very common as a consequence. Geological structures that we seek to understand are often emergent over the course of loading and are observed in the very-large deformation limit. Material properties have strong spatial gradients arising from pressure and temperature dependence and jumps of several orders of magnitude resulting from material interfaces.

docs/user/NextSteps.qmd

@@ -16,7 +16,7 @@ In addition to the notebooks in this brief set of examples, there are a number o
 
   - [The Underworld Website / Blog](https://www.underworldcode.org)
 
-  - [The API documentation](https://underworldcode.github.io/underworld3/main_api/underworld3/index.html) 
+  - [The API documentation](https://underworldcode.github.io/underworld3/main_api/underworld3/index.html)
     (all the modules and functions and their full sets of arguments) is automatically generated from the source code and uses the same rich markdown content as the notebook help text.
 
   - [The API documentation (development branch)](https://underworldcode.github.io/underworld3/development_api/underworld3/index.html)
@@ -46,6 +46,33 @@ Almost all of our notebook examples are annotated python for this reason.An exce
 
 The main difference between the notebook development environment and HPC is the lack of interactivity, particularly in sending parameters to the script at launch time. Typically, we expect the HPC version to be running at much higher resolution, or for many more timesteps than the development notebook. We use the `PETSc` command line parsing machinery to generate notebooks that also can ingest run-time parameters from a script (as above).
 
+#### Parallel scaling / performance
+
+Running geodynamic models on a single CPU/processor (i.e. serial) is time-consuming and limits us to low resolution. Underworld is build from the ground-up as a parallel computing solution which means we can easily run large models on high performance computing clusters (HPC); that is, sub-divide the problem into many smaller chunks and use multiple processors to solve each one, taking care to combine and synchronise the answers from each processor to obtain the correct solution to the original problem.
+
+Parallel computation can reduce time we need to wait for the our results to be computed but it does happen at the expense of some overhead The overhead does depend on the nature of the computer we are using but typically we need to think about:
+
+ - **Code complexity**: any time we manage computations across different processors, we have additional coding to reassemble the calculations correctly and we need to think about many special cases. For example, integrating a quantity of the surface of a mesh: many processes contribute, some do not, the results have to be computed independently then combined.
+
+ - **Additional memory is often required**: to manage copies of information that lives on / near boundaries, to store the topology of the decomposed domain and to help navigate the passing of information between processes.
+
+ - **The time taken to synchronise results** and the work required to keep track of who is doing what, when they are done, and in making sure everyone waits for everyone else. There is a time-cost in actually sending information as part of a synchronisation and a computational cost in ensuring that work is distributed efficiently.
+
+To determine the efficiency of parallel computation, we introduce the *strong scaling test* which measures the time taken to solve a problem in parallel compared to the same problem solved in serial. In strong scaling tests, the size of the problem is kept constant, while the number of processors is increased. The reduction in run-time due to the addition of more processors is commonly expressed in terms of the speed-up:
+
+$$
+\textrm{speed up} = \frac{t(N_{ref})}{t(N)}
+$$
+
+where $t(N_{ref})$ is the run-time for a reference number of processors, $N_{ref}$, and $t(N)$ is the run-time when $N$ processors are used. In the ideal case, $N$ additional processors should contribute all of its resources in solving the problem and reduce the compute time by a factor of $N$ relative to the reference run time. For example, using $2 N_{ref}$ processors will ideally halve the run-time resulting to a speed-up = 2.
+
+::: {#fig-strong-scaling}
+
+![](media/UW3-StrongScalingSolvers.png)
+
+Strong parallel-scaling tests run on Australia's peak computing system, [GADI, at the National Computational Infrastructure](https://nci.org.au/our-systems/hpc-systems?ref=underworldcode.org). This is a typical High Performance Computing facility with large numbers of dedicated, identical CPUs and fast communication links.
+:::
+
 
 ### Advanced capabilities
 
@@ -79,11 +106,11 @@ It is also possible to use the PETSc mesh adaption capabilities, to refine the r
 
 ```{=html}
 <center>
-<iframe src="media/pyvista/AdaptedSphere.html" width="600" height="300">
+<iframe src="media/pyvista/AdaptedSphere.html" width="600" height="500">
 </iframe>
 </center>
 ```
-*Live Image: Static mesh adaptation to the slope of a field. The driving buoyancy term is three plume-like upwellings and the slope of this field is shown in colour (red high, blue low). The adapted mesh is shown in green.*
+*Live Image: Static mesh adaptation to the slope of a field. The driving buoyancy term is a plume-like upwelling and the slope of this field is shown in colour (red high, blue low). Don't forget to zoom in !*
 
 ```python
 

docs/user/Notebooks/1-Meshes.ipynb

@@ -13,28 +13,18 @@
     }
    },
    "source": [
+    "\n",
+    "# Notebook 1: Meshes\n",
+    "\n",
     "<div style=\"float: right; width: 40%\">\n",
     "    \n",
     "![](media/SampleSphericalMesh.png)\n",
     "\n",
     "</div>\n",
     "\n",
     "\n",
-    "## Notebook 1: Meshes\n",
-    "\n",
-    "Introducing meshes: how to build them, interrogate them and visualise them.\n",
-    "\n",
-    "  - `mesh.view()`\n",
-    "  - Mesh refinement options\n",
-    "  - Mesh coordinates\n",
-    "  - Mesh coordinate systems\n",
-    "  - Mesh deformation\n",
-    "\n",
-    "\n",
     "\n",
-    "Mesh adaptivity is a work-in-progress.\n",
-    "\n",
-    "\n"
+    "This notebookIntroduces the mesh discretisation that we use in `Underworld3` and how you can build one of the pre-defined meshes. This notebook also show you how to use the `pyvista` visualisation tools for `Underworld3` objects. The mesh holds information on the mesh geometry, boundaries and coordinate systems and you can attach data to the mesh (see Notebook 2: [Variables](2-Variables.ipynb)). \n"
    ]
   },
   {
@@ -71,15 +61,7 @@
     },
     "tags": []
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "PostHog telemetry failed: HTTPSConnectionPool(host='eu.i.posthog.com', port=443): Max retries exceeded with url: /capture/ (Caused by NameResolutionError(\"<urllib3.connection.HTTPSConnection object at 0x3049912b0>: Failed to resolve 'eu.i.posthog.com' ([Errno 8] nodename nor servname provided, or not known)\"))\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "#| output: false # Suppress warnings in html version\n",
     "\n",
@@ -105,7 +87,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 3,
    "id": "1c928d60-24a9-4415-ab22-41834695c70a",
    "metadata": {
     "editable": true,
@@ -157,7 +139,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 4,
    "id": "b9cfac97-5c19-46b1-aebd-0fb77678b9ed",
    "metadata": {},
    "outputs": [
@@ -173,7 +155,7 @@
        "       [ 0.51755571,  0.41031492,  0.44421012]])"
       ]
      },
-     "execution_count": 23,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -232,7 +214,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
+   "execution_count": 5,
    "id": "1cb8378c-cdb8-4e75-bd12-e85f12169ef7",
    "metadata": {
     "editable": true,
@@ -266,7 +248,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 27,
+   "execution_count": 6,
    "id": "fd6cbed0-3159-4c6c-a482-6a9636158854",
    "metadata": {
     "editable": true,
@@ -291,10 +273,10 @@
        "        "
       ],
       "text/plain": [
-       "<IPython.lib.display.IFrame at 0x35f726660>"
+       "<IPython.lib.display.IFrame at 0x33d867770>"
       ]
      },
-     "execution_count": 27,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -406,19 +388,19 @@
      "text": [
       "\n",
       "\n",
-      "Mesh # 0: .meshes/uw_cubed_spherical_shell_ro1.0_ri0.547_elts8_plexFalse.msh\n",
+      "Mesh # 0: .meshes/uw_cubed_spherical_shell_ro1.0_ri0.547_elts8_plexTrue.msh\n",
       "\n"
      ]
     },
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "f18b77ba1d8d44ae96a3832ea3ffa5a3",
+       "model_id": "9b169322ec7d469e81893a05b2cbfffb",
        "version_major": 2,
        "version_minor": 0
       },
       "text/plain": [
-       "Widget(value='<iframe src=\"http://localhost:50188/index.html?ui=P_0x307b3b5f0_1&reconnect=auto\" class=\"pyvista…"
+       "Widget(value='<iframe src=\"http://localhost:58527/index.html?ui=P_0x33d89cf80_1&reconnect=auto\" class=\"pyvista…"
       ]
      },
      "metadata": {},
@@ -428,28 +410,42 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Number of cells: 3072\n",
+      "Number of cells: 7615\n",
       "\n",
       "No variables are defined on the mesh\n",
       "\n",
-      "| Boundary Name            | ID    |\n",
-      "| -------------------------------- |\n",
-      "| Lower                    | 1     |\n",
-      "| Upper                    | 2     |\n",
-      "| Null_Boundary            | 666   |\n",
-      "| All_Boundaries           | 1001  |\n",
-      "| All_Boundaries           | 1001  |\n",
-      "| UW_Boundaries            | --    |\n",
-      "| -------------------------------- |\n",
+      "| Boundary Name            | ID    | Min Size | Max Size |\n",
+      "| ------------------------------------------------------ |\n",
+      "| Lower                    | 1     | 1062     | 1062     |\n",
+      "| Upper                    | 2     | 1062     | 1062     |\n",
+      "| Null_Boundary            | 666   | 1672     | 1672     |\n",
+      "| All_Boundaries           | 1001  | 1536     | 1536     |\n",
+      "| All_Boundaries           | 1001  | 1536     | 1536     |\n",
+      "| UW_Boundaries            | --    | 5332     | 5332     |\n",
+      "| ------------------------------------------------------ |\n",
       "\n",
       "\n",
-      "Use view(1) to view detailed mesh information.\n",
-      "\n"
+      "DM Object: uw_.meshes/uw_cubed_spherical_shell_ro1.0_ri0.547_elts8_plexTrue.msh 1 MPI process\n",
+      "  type: plex\n",
+      "uw_.meshes/uw_cubed_spherical_shell_ro1.0_ri0.547_elts8_plexTrue.msh in 3 dimensions:\n",
+      "  Number of 0-cells per rank: 1672\n",
+      "  Number of 1-cells per rank: 10053\n",
+      "  Number of 2-cells per rank: 15998\n",
+      "  Number of 3-cells per rank: 7615\n",
+      "Labels:\n",
+      "  depth: 4 strata with value/size (0 (1672), 1 (10053), 2 (15998), 3 (7615))\n",
+      "  All_Boundaries: 1 strata with value/size (1001 (1536))\n",
+      "  Elements: 1 strata with value/size (99999 (7871))\n",
+      "  Lower: 1 strata with value/size (1 (1062))\n",
+      "  Upper: 1 strata with value/size (2 (1062))\n",
+      "  celltype: 4 strata with value/size (0 (1672), 1 (10053), 3 (15998), 6 (7615))\n",
+      "  Null_Boundary: 1 strata with value/size (666 (1672))\n",
+      "  UW_Boundaries: 4 strata with value/size (1 (1062), 2 (1062), 666 (1672), 1001 (1536))\n"
      ]
     }
    ],
    "source": [
-    "mesh.view()"
+    "mesh.view(1)"
    ]
   },
   {
@@ -470,14 +466,6 @@
     "\n",
     "See Notebook 8 for a short mesh-deformation example.\n"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "2374ad95-c893-49a8-9915-de68374135ba",
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

docs/user/Notebooks/2-Variables.ipynb

@@ -15,7 +15,10 @@
    "source": [
     "# Notebook 2: Variables\n",
     "\n",
-    "We can add discrete variables (unknowns associated with the mesh points)."
+    "We can add discrete \"variables\" (unknowns associated with the mesh points) to a mesh, assign values to them and build expressions that `sympy` can understand, manipulate and simplify.\n",
+    "\n",
+    "This notebook introduces the concept of `MeshVariables` in `Underworld3`. These are both data containers and `sympy` symbolic objects. We show you how to inspect a `meshVariable`, set the data values in the `MeshVariable` and visualise them, and build expressions that `sympy` can understand, manipulate and simplify.\n",
+    "\n"
    ]
   },
   {
@@ -50,15 +53,7 @@
     },
     "tags": []
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "PostHog telemetry failed: HTTPSConnectionPool(host='eu.i.posthog.com', port=443): Max retries exceeded with url: /capture/ (Caused by NameResolutionError(\"<urllib3.connection.HTTPSConnection object at 0x33cee0560>: Failed to resolve 'eu.i.posthog.com' ([Errno 8] nodename nor servname provided, or not known)\"))\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "#| output: false # Suppress warnings in html version\n",
     "\n",
@@ -139,22 +134,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 5,
    "id": "333ea2f7-1df3-49f5-9f28-0dbebabae49b",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Expression to be evaluated: Matrix([[N.x/sqrt(N.x**2 + N.y**2 + N.z**2), N.y/sqrt(N.x**2 + N.y**2 + N.z**2), N.z/sqrt(N.x**2 + N.y**2 + N.z**2)]])\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "with mesh.access(scalar_var, vector_var):\n",
-    "    scalar_var.data[:,0] = uw.function.evaluate(r, scalar_var.coords, evalf=False )\n",
-    "    vector_var.data[:,:] = uw.function.evaluate(r_vec, vector_var.coords, evalf=False, verbose=True)"
+    "    scalar_var.data[:,0] = uw.function.evaluate(r, scalar_var.coords)\n",
+    "    vector_var.data[:,:] = uw.function.evaluate(r_vec, vector_var.coords)"
    ]
   },
   {
@@ -167,7 +154,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": 6,
    "id": "9e778873-9ba9-4778-a0d2-828c1647cf50",
    "metadata": {},
    "outputs": [
@@ -264,7 +251,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 25,
+   "execution_count": 7,
    "id": "1ad62dad-6247-4818-a7bd-8227fc5ace69",
    "metadata": {
     "editable": true,
@@ -310,7 +297,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
+   "execution_count": 8,
    "id": "4754590a-23c4-42d7-ab04-97f6ea8421a9",
    "metadata": {
     "editable": true,
@@ -335,10 +322,10 @@
        "        "
       ],
       "text/plain": [
-       "<IPython.lib.display.IFrame at 0x3503a9490>"
+       "<IPython.lib.display.IFrame at 0x34f6b1370>"
       ]
      },
-     "execution_count": 26,
+     "execution_count": 8,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -363,8 +350,24 @@
    "source": [
     "## More information\n",
     "\n",
-    "The meshVariable code is described in the API docs: https://underworldcode.github.io/underworld3/main_api/underworld3/meshing.html"
+    "The meshVariable code is described [**API docs** here.](https://underworldcode.github.io/underworld3/development_api/underworld3/discretisation.html#underworld3.discretisation.MeshVariable)"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b13b41c5-8b5c-417f-b60e-0f82376ef686",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d3d9c223-b7b9-4bb3-a81c-a33521392d30",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {