Submodule update

[Emre-Yorat89]

Closes # (if applicable).

Changes proposed in this Pull Request

Hello,

I have updated pypsa-earth submodule and also fetched changes after the Version 0.0.2 is published. I hope this is okay. If not I can do correction to this pull requests. Beside this, I am going to try to create new environment using pypsa-earth/envs/linux-64.lock.yaml.
I will let you know how the process is going on.
Thank you.

Checklist

• I consent to the release of this PR's code under the GPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to the main environment at PyPSA-Earth repository
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[Emre-Yorat89]

Hi,

I am happy to announce that I was able generate an environment with the updated pypsa-earth submodule.
The code I used for this inside pypsa-distribution folder using WSL terminal is: conda env create -f pypsa-earth/envs/linux-64.lock.yaml ( if you would like to try that do not forget to change the environment name inside linux-64.loc.yaml to avoid risking your current pypsa-earth environment.)
I have added rampdemand package upon this environment with pip install rampdemand command.

Now the error I get when I tried to run the tutorial (snakemake -j 1 solve_network) as is:
I would like to document this to follow what changes I made to overcome these issue and take your opinions on what changes you suggest me to do.
Thank you so much in advance.

[Emre-Yorat89]

Hi,
I have experienced some errors when I tried snakemake -j 1 solve_network after updating the submodule and the environment. I have suggested some changes to correct some of those errors but still I could not manage the start the subworkflow to download data yet.
The error at the top is about invalid escape sequence:
/mnt/c/Users/Emre/Documents/pypsa-distribution-fork/submodule/pypsa-distribution/pypsa-earth/Snakefile:90: DeprecationWarning:
invalid escape sequence '.'
I was able to solve this issue by putting the character r before all lines inside wildcard_constraints dictionary in pypsa-earth/Snakefile.

A warning for a mismatch between the updated pypsa-earth submodule and the version specified in pypsa-distribution/config.pypsa-earth.yaml
It is solved by changed the pypsa-earth version in config.pypsa-earth.yaml from 0.3.0 to 0.8.0.

An error (string indices must be integers, not 'str') for agg_p_nom_limits dictionary.
I was able to solve this by updating values of agg_p_nom_limits key inside config.pypsa-earth.yaml from pypsa-earth/config.yaml values of agg_p_nom_limits key.

The current problem is I cannot run the subworkflow. I was expecting after clearing those error the subworkflow should run but I could not yet. I am trying to figure out where this issue stems from. If you have any ideas for the issue I would be very happy to implement the changes that you would suggest. Thank you in advance.

The figure that shows the subworkflow is not running.

[Emre-Yorat89]

Hello,

I have some good news and bad news for the subworkflow. I was able to run the subworkflow. Actually, why it did not start at first still unclear for me. But changing the bundles_to_download list in retrieve_databundle rule helped me to run the subworkflow and download the inital data.
Then I have experienced some errors in the usual workflow, the first one was in pypsa-earth/build_renewable_profiles.py.
Our current pypsa-earth submodule has a line for progressbar https://github.com/pypsa-meets-earth/pypsa-earth/blob/c1d4b993dc52d1f22a46efab983bcef5289afb95/scripts/build_renewable_profiles.py#L500
And the updated pypsa-earth submodule has different approach for getting progressbar info as in https://github.com/pypsa-meets-earth/pypsa-earth/blob/fdf7f4ae6f9abe1cd4b60c9f9296643351c6452a/scripts/build_renewable_profiles.py#L496
To solve this issue, I have copied the option for progressbar from the updated pypsa-earth config.default.yaml to the config.pypsa-earth.yaml file specifically I have put it inside the "enable" dictionary to fetch informartion config["enable"]["progress_bar"] in build_renewable_profiles.py. The added line to the config is here: https://github.com/Emre-Yorat89/pypsa-distribution/blob/de689df36c81ce1423cf22a852b187bf23c96d7e/config.pypsa-earth.yaml#L35

The last issue I have experienced was in pypsa-distribution/solve_network.py script in the line https://github.com/Emre-Yorat89/pypsa-distribution/blob/fd88dd351b0729e4d55feb4fcde5966c377ba49d/scripts/solve_network.py#L43
The error is No Module pypsa.linopf Found. So my initial guess for this error is related to the updated environment that I use. Probably, pypsa.linopf has been removed from pypsa in latest version. That is probably the reason why I get the module not found error.
Upon this error I want to try n.optimize() to solve the network. I wanted to make a comparison of solved networks with the old environment and the new environment. But n.optimize() has also given me the below error when I try to solve the network.

I would like to also mention that I was able to solve the network (networks/elec.nc) in a jupyter notebook using the new environment with n.optimize() function. I could not yet figure out why n.optimize() function is not working inside the solve_network.py script.
Update to solving network error: I am now able to solve the network with n.optimize() function in solve_network.py. I was using it with the solve_name option as parameter to this function so I found that not using solver_name option as parameter helps to remove the solving error. Now I get the objective value also. I wanted to test if the old functionality (network_lopf) and the new functionality(n.objective()) are giving the same objective value. To do that I have solved the tutorial with the new and old environment separately and check their objective values as well as lines_t dictionary which are closely matching.
These are the changed lines in solve_network.py: https://github.com/Emre-Yorat89/pypsa-distribution/blob/1a5bf25187f3bae1607d973041fad68bb9ab387c/scripts/solve_network.py#L94-L97
https://github.com/Emre-Yorat89/pypsa-distribution/blob/1a5bf25187f3bae1607d973041fad68bb9ab387c/scripts/solve_network.py#L116
These are the objective values to compare new and old pypsa optimization functionalities.

The last issue is about the changes I have made to the pypsa-earth submodule and pushed to my submodule_and_environment_update branch. The problem here is since I have changed some parts in the submodule I cannot no longer create local copies of this branch with the command in CMD "git clone --branch submodule_and_environment_update --recurse-submodules https://github.com/Emre-Yorat89/pypsa-distribution.git" The error I get on the terminal is "fatal: remote error: upload-pack: not our ref d1e1c0d7965b79d7ac22f5c9e1197a772efad99d
fatal: Fetched in submodule path 'pypsa-earth', but it did not contain d1e1c0d7965b79d7ac22f5c9e1197a772efad99d. Direct fetching of that commit failed."
Also I am unable to open the submodule on github page and on trying I get below error.

The only change I did in the pypsa-earth submodule was to place r before the values of wildcard_constraints dictionary in the Snakefile of pypsa-earth.

So I probably need to open a new branch and inside it I again repeat the same solution steps except changing the pypsa-earth/Snakefile. After fixing the problem in solve_network.py. May be we can search for the ways to change some parts of the submodule before releasing the version.

Thank you for your suggestions in advance.