Merge pull request #829 from mimiframework/params

Simplify Parameter Handling

Author: davidanthoff

README.md

@@ -9,9 +9,11 @@ Mimi is a [Julia](http://julialang.org) package that provides a component model
 
 ## NEWS
 
+We recently tagged and released a feature relese revamping the API surrounding parameters, please see https://www.mimiframework.org/Mimi.jl/dev/howto/howto_5/ and https://www.mimiframework.org/Mimi.jl/dev/howto/howto_9/.
+
 On 7/15/2020 we officially tagged and released Mimi v1.0.0, which has some new features, documentation, and quite a bit of internals work as well.  Since this is a major version change, there are some breaking changes that may require you to update your code.  We have done the updates for the existing models in the Mimi registry (FUND, DICE, etc.), and will release new major versions of those today as well, so if you are using the latest version of Mimi and the latest version of the packages, all should run smoothly.
 
-**Please view the how to guide here: https://www.mimiframework.org/Mimi.jl/stable/howto/howto_6/ for a run-down of how you should update your own code.**
+**Please view the how to guide here: https://www.mimiframework.org/Mimi.jl/stable/howto/howto_7/ for a run-down of how you should update your own code.**
 
 In addition please do not hesitate to ask any questions on the forum, we are working hard to keep this transition smooth.
 

contrib/test_all_models.jl

@@ -10,24 +10,27 @@
 #   julia --color=yes test_all_models.jl
 #
 
+using Pkg
+pkg_that_errored = []
 
+# first set of packages to test
 packages_to_test = [
-    "MimiDICE2010" => ("https://github.com/anthofflab/MimiDICE2010.jl", "master"),
-    "MimiDICE2013" => ("https://github.com/anthofflab/MimiDICE2013.jl", "master"),
-    "MimiRICE2010" => ("https://github.com/anthofflab/MimiRICE2010.jl", "master"),
-    "MimiFUND" => ("https://github.com/fund-model/MimiFUND.jl", "master"),
-    "MimiPAGE2009" => ("https://github.com/anthofflab/MimiPAGE2009.jl", "master"),
+    "MimiDICE2010" => ("https://github.com/anthofflab/MimiDICE2010.jl", "df"), # note here we use the df branch 
+    "MimiDICE2013" => ("https://github.com/anthofflab/MimiDICE2013.jl", "df"), # note here we use the df branch 
     "MimiDICE2016" => ("https://github.com/AlexandrePavlov/MimiDICE2016.jl", "master"),
+    "MimiDICE2016R2" => ("https://github.com/anthofflab/MimiDICE2016R2.jl", "master"),
+    "MimiRICE2010" => ("https://github.com/anthofflab/MimiRICE2010.jl", "master"), 
+    "MimiFUND" => ("https://github.com/fund-model/MimiFUND.jl", "mcs"), # note here we use the mcs branch 
+    "MimiPAGE2009" => ("https://github.com/anthofflab/MimiPAGE2009.jl", "mcs"), # note here we use the mcs branch 
+    "MimiPAGE2020" => ("https://github.com/lrennels/MimiPAGE2020.jl", "mcs"), # note using lrennels fork mcs branch, and testing this takes a LONG time :) 
     "MimiSNEASY" => ("https://github.com/anthofflab/MimiSNEASY.jl", "master"),
     "MimiFAIR" => ("https://github.com/anthofflab/MimiFAIR.jl", "master"),
     "MimiMAGICC" => ("https://github.com/anthofflab/MimiMAGICC.jl", "master"),
-    "MimiHector" => ("https://github.com/anthofflab/MimiHector.jl", "master")
+    "MimiHector" => ("https://github.com/anthofflab/MimiHector.jl", "master"),
 ]
 
-using Pkg
-
 mktempdir() do folder_name
-    pkg_that_errored = []
+
     Pkg.activate(folder_name)
 
     Pkg.develop(PackageSpec(path=joinpath(@__DIR__, "..")))
@@ -54,6 +57,39 @@ mktempdir() do folder_name
     for p in pkg_that_errored
         println(p)
     end
-    
 end
 
+# test separately because needs MimiFUND 3.8.6
+packages_to_test = [
+   "MimiIWG" => ("https://github.com/rffscghg/MimiIWG.jl", "mcs") # note here we use the mcs branch
+]
+
+mktempdir() do folder_name
+
+    Pkg.activate(folder_name)
+
+    Pkg.develop(PackageSpec(path=joinpath(@__DIR__, "..")))
+
+    Pkg.add([i isa Pair ? PackageSpec(url=i[2][1], rev=i[2][2]) : PackageSpec(i) for i in packages_to_test])
+
+    Pkg.resolve()
+
+    for pkg_info in packages_to_test
+        pkg = pkg_info isa Pair ? pkg_info[1] : pkg_info
+        @info "Now testing $pkg."
+        try
+            Pkg.test(PackageSpec(pkg))
+        catch err
+            push!(pkg_that_errored, pkg)
+        end
+    end
+end
+
+println()
+println()
+println()
+
+println("The following packages errored:")
+for p in pkg_that_errored
+    println(p)
+end

docs/make.jl

@@ -20,10 +20,12 @@ makedocs(
 			"1 Construct + Run a Model" => "howto/howto_1.md",
 			"2 Explore Results" => "howto/howto_2.md",
 			"3 Monte Carlo + SA" => "howto/howto_3.md",
-			"4 Timesteps, Params, and Vars" => "howto/howto_4.md",
-			"5 Update Time Dimension" => "howto/howto_5.md",
-			"6 Port to v0.5.0" => "howto/howto_6.md",
-			"7 Port to v1.0.0" => "howto/howto_7.md"
+			"4 Timesteps" => "howto/howto_4.md",
+			"5 Parameters + Variables" => "howto/howto_5.md",
+			"6 Update Time Dimension" => "howto/howto_6.md",
+			"7 Port to v0.5.0" => "howto/howto_7.md",
+			"8 Port to v1.0.0" => "howto/howto_8.md",
+			"9 Port to New Param API" => "howto/howto_9.md"
 		],
 		"Advanced How-to Guides" => Any[
 			"Advanced How-to Guides Intro" => "howto_advanced/howto_adv_main.md",

docs/src/explanations/exp_pkgs.md

@@ -23,6 +23,8 @@ run(m)
 
 ## Registries and The Mimi Registry
 
+*Update (5/3/2020): Please note that going forward we are moving away from this model and encouraging registration in the General Registry to keep things simple and seamless for users instead of requiring extra maintenance and communication by our team. This will not be a breaking change, so current models registered with the Mimi registry will work as expected.*
+
 Packages can be registered in a [Registry](https://julialang.github.io/Pkg.jl/v1/registries/), and "registries contain information about packages, such as available releases and dependencies, and where they can be downloaded. The [General registry](https://github.com/JuliaRegistries/General) is the default one, and is installed automatically".
 
 The Mimi registry is a custom registry maintained by the Mimi development team that colocates several Mimi models in one central registry in the same way julia colates packages in the General registry, where `Mimi` and other packages you commonly may use are located. While the development team maintains this registry and has some basic requirements such as continuous integration tesing (CI) and proper package structure as dictated by julia, they do not claim responsibility or knowledge of the content or quality of the models themselves. 

docs/src/howto/howto_1.md

@@ -1,6 +1,6 @@
 # How-to Guide 1: Construct and Run a Model
 
-This how-to guide pairs nicely with Tutorial 4: Create a Model and Tutorial 6: Create a Model with Composite Components, and serves as a higher-level version and refresher for those with some experience with Mimi.  If this is your first time constructing and running a Mimi model, we recommend you start with Tutorial 4 (and Tutorial 6 if you are interested in composite components), which will give you more detailed step-by step instructions.
+This how-to guide pairs nicely with [Tutorial 4: Create a Model](@ref) and [Tutorial 6: Create a Model Including Composite Components](@ref), and serves as a higher-level version and refresher for those with some experience with Mimi.  If this is your first time constructing and running a Mimi model, we recommend you start with Tutorial 4 (and Tutorial 6 if you are interested in composite components), which will give you more detailed step-by step instructions.
 
 ## Defining Components
 
@@ -48,13 +48,13 @@ The API for using the fourth argument, represented as `t` in this explanation, i
 
 To access the data in a parameter or to assign a value to a variable, you must use the appropriate index or indices (in this example, either the Timestep or region or both).
 
-By default, all parameters and variables defined in the `@defcomp` will be allocated storage as scalars or Arrays of type `Float64.` For a description of other data type options, see How-to Guide 4: Work with Timesteps, Parameters, and Variables 
+By default, all parameters and variables defined in the [`@defcomp`](@ref) will be allocated storage as scalars or Arrays of type `Float64.` For a description of other data type options, see How-to Guide 5: Work with Parameters and Variables 
 
 ### Composite Components
 
 Composite components can contain any number of subcomponents, **which can be either leaf components or more composite components**. To the degree possible, composite components are designed to operate in the same way as leaf components, although there are a few necessary differences:
 
-- Leaf components are defined using the macro `@defcomp`, while Composite components are defined using `@defcomposite`. Each macro supports syntax and semantics specific to the type of component.
+- Leaf components are defined using the macro [`@defcomp`](@ref), while Composite components are defined using [`@defcomposite`](@ref). Each macro supports syntax and semantics specific to the type of component.
 
 - Leaf components support user-defined `run_timestep()` functions, whereas composites have a built-in `run_timestep()` function that iterates over its subcomponents and calls their `run_timestep()` function.
 
@@ -103,9 +103,9 @@ Now we construct a composite component `MyCompositeComponent` which holds the tw
 end
 ```
 
-The `connect` calls are responsible for making internal connections between any two components held by a composite component, similar to `connect_param!` described in the Model section below. 
+The `connect` calls are responsible for making internal connections between any two components held by a composite component, similar to [`connect_param!`](@ref) described in the Model section below. 
 
-As mentioned above, conflict resolution refers to cases where two subcomponents have identically named parameters, and thus the user needs to explicitly demonstrate that they are aware of this and create a new external parameter that will point to all subcomponent parameters with that name.  For example, given leaf components `A` and `B`: 
+As mentioned above, conflict resolution refers to cases where two subcomponents have identically named parameters, and thus the user needs to explicitly demonstrate that they are aware of this and create a new shared model parameter that will point to all subcomponent parameters with that name.  For example, given leaf components `A` and `B`: 
 
 ```julia
 @defcomp Leaf1 begin
@@ -161,21 +161,29 @@ add_comp!(m, ComponentA)
 add_comp!(m, ComponentA, :GDP)
 ```
 
-The first argument to `add_comp!` is the model, the second is the name of the ComponentId defined by `@defcomp`. If an optional third symbol is provided (as in the second line above), this will be used as the name of the component in this model. This allows you to add multiple versions of the same component to a model, with different names.
+The first argument to [`add_comp!`](@ref) is the model, the second is the name of the ComponentId defined by [`@defcomp`](@ref). If an optional third symbol is provided (as in the second line above), this will be used as the name of the component in this model. This allows you to add multiple versions of the same component to a model, with different names.
 
-The `add_comp` function has two more optional keyword arguments, `first` and `last`, which can be used to indicate a fixed start and/or end time (year in this case) that the compnonent should run for (within the bounds of the model's time dimension).  For example, the following indicates that `ComponentA` should only run from 1900 to 2000.
+The [`add_comp!`](@ref) function has two more optional keyword arguments, `first` and `last`, which can be used to indicate a fixed start and/or end time (year in this case) that the compnonent should run for (within the bounds of the model's time dimension).  For example, the following indicates that `ComponentA` should only run from 1900 to 2000.
 
 ```julia
 add_comp!(m, ComponentA; first = 1900, last = 2000)
 ```
 
-The next step is to set the values for all the parameters in the components. Parameters can either have their values assigned from external data, or they can internally connect to the values from variables in other components of the model.
+The next step is to set the values for all the parameters in the components. Parameters can either have their values assigned from external data, or they can internally connect to the values from variables in other components of the model. When assigned from external data, parameters are externally connected to a model parameter, which can be a shared model parameter with its own name and connected to more than one component-parameter pair, or an unshared model paarameter accessible only through the component-parameter pair names and connected solely to that parameter.
 
-To make an external connection, the syntax is as follows:
+To make an external connection to an unshared model parameter, the syntax is as follows:
 
 ```julia
-set_param!(m, :ComponentName, :ParameterName, 0.8) # a scalar parameter
-set_param!(m, :ComponentName, :ParameterName2, rand(351, 3)) # a two-dimensional parameter
+update_param!(m, :ComponentName, :ParameterName1, 0.8) # a scalar parameter
+update_param!(m, :ComponentName, :ParameterName2, rand(351, 3)) # a two-dimensional parameter
+```
+
+To make an external connection to a shared model parameter, the syntax is as follows:
+
+```julia
+add_shared_param!(m, :ModelParameterName, 1.0) # add a shared model parameter to the model
+connect_param!(m, :ComponentName, :ParameterName3, :ModelParameterName) # connect component parameter
+connect_param!(m, :ComponentName, :ParameterName4, :ModelParameterName)
 ```
 
 To make an internal connection, the syntax is as follows.  
@@ -301,11 +309,11 @@ end
 m = Model()
 set_dimension!(m, :time, 2005:2020)
 add_comp!(m, top, nameof(top))
-set_param!(m, :fooA1, 1)
-set_param!(m, :fooA2, 2)
-set_param!(m, :foo3, 10)
-set_param!(m, :foo4, 20)
-set_param!(m, :par_1_1, collect(1:length(2005:2020)))
+update_param!(m, :top, :fooA1, 1)
+update_param!(m, :top, :fooA2, 2)
+update_param!(m, :top, :foo3, 10)
+update_param!(m, :top, :foo4, 20)
+update_param!(m, :top, :par_1_1, collect(1:length(2005:2020)))
 run(m)
 ```
 Take a look at what you've created now using `explore(m)`, a peek into what you can learn in How To Guide 2!

docs/src/howto/howto_2.md

@@ -32,9 +32,9 @@ getdataframe(m, :Component1=>:Var1, :Component2=>:Var2) # request variables from
 
 Mimi provides support for plotting using [VegaLite](https://github.com/vega/vega-lite) and [VegaLite.jl](https://github.com/fredo-dedup/VegaLite.jl).
 
-Plotting support is provided by the **Explorer UI**, rooted in `VegaLite`.  The `explore` function allows the user to view and explore the variables and parameters of a model run.  The explorer can be used in two primary ways.
+Plotting support is provided by the **Explorer UI**, rooted in `VegaLite`.  The [`explore`](@ref) function allows the user to view and explore the variables and parameters of a model run.  The explorer can be used in two primary ways.
 
-In order to invoke the explorer UI and explore all of the variables and parameters in a model, simply call the function `explore` with the model run as the required argument as shown below.  This will produce a new browser window containing a selectable list of parameters and variables, organized by component, each of which produces a graphic.  The exception here being that if the parameter or variable is a single scalar value, the value will appear alongside the name in the left-hand list.
+In order to invoke the explorer UI and explore all of the variables and parameters in a model, simply call the function [`explore`](@ref) with the model run as the required argument as shown below.  This will produce a new browser window containing a selectable list of parameters and variables, organized by component, each of which produces a graphic.  The exception here being that if the parameter or variable is a single scalar value, the value will appear alongside the name in the left-hand list.
 
 ```julia
 run(m)
@@ -43,7 +43,7 @@ explore(m)
 
 ![Explorer Model Example](../figs/explorer_model_example.png)
 
-Alternatively, in order to view just one parameter or variable, call the (unexported) function `Mimi.plot` as below to return a plot object and automatically display the plot in a viewer, assuming `Mimi.plot` is the last command executed.  Note that `plot` is not exported in order to avoid namespace conflicts, but a user may import it if desired. This call will return the type `VegaLite.VLSpec`, which you may interact with using the API described in the [VegaLite.jl](https://github.com/fredo-dedup/VegaLite.jl) documentation.  For example, [VegaLite.jl](https://github.com/fredo-dedup/VegaLite.jl) plots can be saved as [PNG](https://en.wikipedia.org/wiki/Portable_Network_Graphics), [SVG](https://en.wikipedia.org/wiki/Scalable_Vector_Graphics), [PDF](https://en.wikipedia.org/wiki/PDF) and [EPS](https://en.wikipedia.org/wiki/Encapsulated_PostScript) files. You may save a plot using the `save` function.  
+Alternatively, in order to view just one parameter or variable, call the (unexported) function `plot` as below to return a plot object and automatically display the plot in a viewer, assuming `plot` is the last command executed.  Note that `plot` is not exported in order to avoid namespace conflicts, so needs to be called with Mimi.plot or a user may import it if desired. This call will return the type `VegaLite.VLSpec`, which you may interact with using the API described in the [VegaLite.jl](https://github.com/fredo-dedup/VegaLite.jl) documentation.  For example, [VegaLite.jl](https://github.com/fredo-dedup/VegaLite.jl) plots can be saved as [PNG](https://en.wikipedia.org/wiki/Portable_Network_Graphics), [SVG](https://en.wikipedia.org/wiki/Scalable_Vector_Graphics), [PDF](https://en.wikipedia.org/wiki/PDF) and [EPS](https://en.wikipedia.org/wiki/Encapsulated_PostScript) files. You may save a plot using the `save` function.  
 
 Note that saving an interactive plot in a non-interactive file format, such as .pdf or .svg will result in a warning `WARN Can not resolve event source: window`, but the plot will be saved as a static image. If you wish to preserve interactive capabilities, you may save it using the .vegalite file extension. If you then open this file in Jupyter lab, the interactive aspects will be preserved.
 
@@ -55,4 +55,4 @@ save("figure.svg", p)
 ```
 ![Plot Model Example](../figs/plot_model_example.png)
 
-These two functions, `explore` and `plot` also have methods applicable to the sensitivity analysis support described in the next section. Details can be found in the sensitivity analysis how-to guide How-to Guide 3: Conduct Sensitivity Analysis as well as Tutorial 4: Sensitivity Analysis (SA) Support.
+These two functions, [`explore`](@ref) and `plot` also have methods applicable to the sensitivity analysis support described in the next section. Details can be found in the sensitivity analysis how-to guide How-to Guide 3: Conduct Sensitivity Analysis as well as Tutorial 4: Sensitivity Analysis (SA) Support.