Commodity prices in MUSE

[tsmbland]

A few notes about how prices are used and calculated in MUSE:

User inputs

Prices are specified in a csv file with region, year, and all of the commodities as columns. Each row holds the price of all commodities for a specific year. This means that (annoyingly), if a user want to specify the price of one commodity for a specific year, then they need to specify all commodities for that year, even if many of these input prices will never be seen by the model (more on that below).

I believe it's also possible to specify prices on a timeslice level (the model certainly holds timeslice-level prices), although I've never seen that being done.

How are prices used?

Prices are used in the calculation of objectives (calculated at the timeslice level), which the solver aims to minimize. Each year, the objectives are given prices from the current year up to the forecast year (current year + forecast length), however it varies between the objectives whether they actually use the full price forecast. The LCOE objective does use the full forecast (with flat-forward extension beyond the forecast year if necessary), however other objectives such as fuel consumption cost, capital cost and emissions cost only use prices for the forecast year.

NOTE. The sector is only ever given prices for the current year and the investment year, and if the forecast year is further in the future than the investment year (e.g. a forecast length of 10 with a 5-year time framework), then it will just use linear extrapolation to calculate prices for the forecast year.

Updating prices

Each year of the simulation, the model will recalculate the prices for some or all of the commodities. The way that prices are dealt with differs depending on the type of commodity. We can split commodities into three categories:

Environmental products (i.e. CO2f)

Prices for these are never recalculated, and the model will always use the projections given in the input file.
EDIT: Except when using a carbon budget (see comments below)

Input commodities that are not outputted by any process

If a commodity is not outputted by any process in the model (e.g. maybe oil is an input commodity to some processes, but there is no process in the model outputting oil), then prices will not be recalculated (same as above).

Commodities that are outputted by a process/processes

Any commodities that are outputted by a process will have their price recalculated for future years by the supply_cost function (more on this later). For example, if it's 2020 investing for 2025, then the price for 2025 will be replaced with the recalculated value, and all future years will also be set to this price (regardless of the forecast length). Then, in the next year, 2030 and all future years will be replaced again.

Price updates happen each iteration of the MCA.

This means that, for these commodities, the only input prices that will ever be seen by the solver are the first year and the first investment year (e.g. 2020 and 2025). And since prices for the investment year get replaced each iteration of the MCA, only the input values for the first year will be used directly in the final result.

The exception here is if maximum_iterations is set to 1. In this case, no prices will be updated, and the input projections for all years will be maintained.

How are prices calculated?

Prices are re-calculated using the supply_cost function.

If a commodity is produced by a single technology, the new price is just the LCOE of the commodity produced by that technology in the investment year. This will depend on the lifetime of the technology (technical_life), the interest rate for the technology, capital costs (cap_par), fixed/variable costs (fix_par and var_par), the utilization factor, the commodity prices of the technology inputs, and the commodity price of environmental outputs. Exponents (cap_exp, fix_exp and var_exp) are not used (equivalent to having exponents of 0). If a commodity is produced by multiple technologies in the same sector*, a weighted average is taken according to the production by each technology. Prices are calculated at the timeslice-level, however the only price-determinant that can vary on a timeslice-basis (apart from the price of other commodities) is the utilization factor.

(*This only works if all technologies that produce a commodity are in the same sector. If this isn't the case, then you can run into all sorts of problems, beyond just the calculation of prices)

I can understand the logic, as the (weighted-average) LCOE is the price that must be charged for each commodity for the technologies to break even. However, I find it really strange that the price of a commodity in t=2 does not directly depend on its own price in t=1. I also find it strange (perhaps naively) that prices aren't influenced by demand and supply...

Also, because the exponents aren't used in the calculation of commodity prices, they are only going to be accurate it capacity/production are equal to 1 (or if the exponents are zero), which may not be the case. Why are we not using the exponents?

[ahawkes]

Thanks very much for this @tsmbland. I have a few initial questions and a couple of clarifications:

1. Is the forecast price ever different to the endogenously calculated price?
2. When calculating LCOE I assume a flat-forward price is assumed that is equal to the investment year price? i.e. flat forward to the end of the life of the process.
3. What happens when output of a process in a time slice is zero? Remember the divide by zero issue with objective calculation we came across a few weeks ago? I recall you said the LCOE is calculated in several places without a common function - is this part of the code using the same function that is used for objective calculation?
4. For environmental products, is CO2 a special case somehow? I thought the carbon budget functionality allowed for changing (usually escalating) CO2 price to try to encourage investment in low carbon technologies - i.e. keep increasing the CO2 price until the CO2 budget is satisfied?
5. You mention price is calculated a time slice level. How is this done exactly? As discussed previously, it's hard to know how to distribute capital cost and fixed operating cost to time slices.
6. You wrote that "however other objectives such as fuel consumption cost, capital cost and emissions cost only use prices for the forecast year." - Did you mean to say "investment year" here? It might be odd to use values for the forecast year if it's far in the future.

And some clarifications:

1. Weighted average of LCOE is OK for estimating price is certain markets. But most markets are priced at the marginal price of the most expensive process (i.e. variable costs only, including input/outputs commodity costs, variable opex) - capital costs ignored etc. I think we're OK to stick with weighted average LCOE for now. But wasn't there a toml input parameter that could switch this to something else?
2. Supply/demand balance does still influence price in this formulation - this is because as demand increases you have to buy more capacity - and as you buy more capacity you may hit a constraint and need to move to the next most expensive process - thus increasing overall price.
3. cap_exp and fix_exp and var_exp: all these are a bit annoying as they are hard to get data for and not that intuitive to use. I advise people to set them to a value where they do not influence the value of cap_par etc in the calculations (I think that's zero, right?).

Overall:
Perhaps the way forward here is (a) making sure one common function is used for LCOE calculation everywhere in the model where it's needed, and (b) modifying the code such that input prices are not needed for every commodity when you input them for one commodity (and modifying so that you can specify time slice level input prices), and (c) improving the documentation so that the behaviour you've observed is clearly documented - and maybe incorporating some of these features into tutorials (e.g. how a price of an environmental commodity is used in the model, and input-only commodity, and how maximum_iterations influences price calculations - in fact maybe this last one also needs a code modification because it seems silly - instead user should be able to specify which commodities they want endogeneous prices for and which they want to use input prices for)?

[tsmbland]

1. Is the forecast price ever different to the endogenously calculated price?

If the forecast length is equal to the time framework (5 years for most models), then no (except the very first MCA iteration for the first investment year which uses the input projection, but this will be overwritten anyway).

If the forecast length is longer (e.g. 10 years), then prices beyond the investment year will be a linear extrapolation based on the current price and the investment year price (so endogenously calculated, but not directly based on LCOE)

2. When calculating LCOE I assume a flat-forward price is assumed that is equal to the investment year price? i.e. flat forward to the end of the life of the process.

Yes, but again, if the forecast year is later than the investment year, the flat-forward extension will be from the extrapolated forecast year price.

3. What happens when output of a process in a time slice is zero? Remember the divide by zero issue with objective calculation we came across a few weeks ago? I recall you said the LCOE is calculated in several places without a common function - is this part of the code using the same function that is used for objective calculation?

The LCOE used to calculate prices is a different function to the objective calculation, so no, the divide-by-zero fix doesn't apply here. However, I think it's fine because the weighted average multiplies by zero if the output of a process is zero.

4. For environmental products, is CO2 a special case somehow? I thought the carbon budget functionality allowed for changing (usually escalating) CO2 price to try to encourage investment in low carbon technologies - i.e. keep increasing the CO2 price until the CO2 budget is satisfied?

Yes, if using a carbon budget then CO2 (or anything you specify in the carbon_budget_control.commodities list in the settings file) is treated differently. The new carbon price gets calculated for the investment year, and this gets propagated to all future years, so any input prices after the first investment year will not be used.

5. You mention price is calculated a time slice level. How is this done exactly? As discussed previously, it's hard to know how to distribute capital cost and fixed operating cost to time slices.

The LCOE function used to calculate prices is essentially trying to find the levelized cost of supplying 1 unit of the commodity in the investment year (accounting for the lifetime of the technology). I think there are mistakes, but that's the intention.

I would interpret the calculated LCOE value in each timeslice as the cost of supplying 1 unit of the commodity in a year, if the whole year had the properties of that timeslice. This means you take the full capital and fixed costs when calculating the price in each timeslice, with an adjustment according to the utilization factor in that timeslice (if the output in a timeslice is capped at half the maximum rate, then the capital costs per unit of commodity output are doubled).

When it comes to prices, if you're charging the this timeslice-level LCOE for the length of each timeslice, then I think you'll end up breaking even.

Obviously things would get a lot more complicated if the exponents were included, so I can understand why they were omitted.

6. You wrote that "however other objectives such as fuel consumption cost, capital cost and emissions cost only use prices for the forecast year." - Did you mean to say "investment year" here? It might be odd to use values for the forecast year if it's far in the future.

No, I do mean the forecast year. Should we change this?

1. Weighted average of LCOE is OK for estimating price is certain markets. But most markets are priced at the marginal price of the most expensive process (i.e. variable costs only, including input/outputs commodity costs, variable opex) - capital costs ignored etc. I think we're OK to stick with weighted average LCOE for now. But wasn't there a toml input parameter that could switch this to something else?

Seems like there's something in place if you use the "match" dispatch method (https://muse-os.readthedocs.io/en/latest/inputs/toml.html#standard-sectors), but I'd need to look into this more

2. Supply/demand balance does still influence price in this formulation - this is because as demand increases you have to buy more capacity - and as you buy more capacity you may hit a constraint and need to move to the next most expensive process - thus increasing overall price.

I see, good point

3. cap_exp and fix_exp and var_exp: all these are a bit annoying as they are hard to get data for and not that intuitive to use. I advise people to set them to a value where they do not influence the value of cap_par etc in the calculations (I think that's zero, right?).

Ok. Actually 1 would be the more appropriate value. Zero means the cost is fixed regardless of the amount of installed capacity or the production level, which doesn't make much sense. 1 means the cost is linear with capacity/production (e.g. cap_par * capacity, var_par * production)

Perhaps the way forward here is (a) making sure one common function is used for LCOE calculation everywhere in the model where it's needed,

Yes, I agree. I've homogenized things a lot in #462, but there are still 2 functions for calculating LCOE: lifetime_levelized_cost_of_energy and annual_levelized_cost_of_energy

The former aims to calculate the LCOE across the lifetime of the technology, and is used for the objectives, whereas the latter aims to calculate the LCOE in a single year, and is used to set commodity prices.

I think most of the code in these two functions could be combined into a single function, however there are subtle differences between the two functions that might need to be addressed (use of exponents, interest rates, how the capacity/production level is accounted for).

and (b) modifying the code such that input prices are not needed for every commodity when you input them for one commodity (and modifying so that you can specify time slice level input prices)

This will be possible with the new database format that I have in mind. In theory this part of the database is 90% coded up already, so it might be possible to slot this in even though the full database is some way off.

and (c) improving the documentation so that the behavior you've observed is clearly documented - and maybe incorporating some of these features into tutorials (e.g. how a price of an environmental commodity is used in the model, and input-only commodity, and how maximum_iterations influences price calculations

Agreed. Also could do with a tutorial using a carbon budget.

in fact maybe this last one also needs a code modification because it seems silly - instead user should be able to specify which commodities they want endogenous prices for and which they want to use input prices for)?

Agreed

[ahawkes]

1. Is the forecast price ever different to the endogenously calculated price?

If the forecast length is equal to the time framework (5 years for most models), then no (except the very first MCA iteration for the first investment year which uses the input projection, but this will be overwritten anyway).

If the forecast length is longer (e.g. 10 years), then prices beyond the investment year will be a linear extrapolation based on the current price and the investment year price (so endogenously calculated, but not directly based on LCOE)

Can we change this so that instead of linear extrapolation, the price is assumed to be flat-forward from the investment year?

2. When calculating LCOE I assume a flat-forward price is assumed that is equal to the investment year price? i.e. flat forward to the end of the life of the process.

Yes, but again, if the forecast year is later than the investment year, the flat-forward extension will be from the extrapolated forecast year price.

As above - to be changed to flat-forward.

3. What happens when output of a process in a time slice is zero? Remember the divide by zero issue with objective calculation we came across a few weeks ago? I recall you said the LCOE is calculated in several places without a common function - is this part of the code using the same function that is used for objective calculation?

The LCOE used to calculate prices is a different function to the objective calculation, so no, the divide-by-zero fix doesn't apply here. However, I think it's fine because the weighted average multiplies by zero if the output of a process is zero.

OK. I guess zero multiplied by infinity is OK?

4. For environmental products, is CO2 a special case somehow? I thought the carbon budget functionality allowed for changing (usually escalating) CO2 price to try to encourage investment in low carbon technologies - i.e. keep increasing the CO2 price until the CO2 budget is satisfied?

Yes, if using a carbon budget then CO2 (or anything you specify in the carbon_budget_control.commodities list in the settings file) is treated differently. The new carbon price gets calculated for the investment year, and this gets propagated to all future years, so any input prices after the first investment year will not be used.

OK.

5. You mention price is calculated a time slice level. How is this done exactly? As discussed previously, it's hard to know how to distribute capital cost and fixed operating cost to time slices.

The LCOE function used to calculate prices is essentially trying to find the levelized cost of supplying 1 unit of the commodity in the investment year (accounting for the lifetime of the technology). I think there are mistakes, but that's the intention.

I would interpret the calculated LCOE value in each timeslice as the cost of supplying 1 unit of the commodity in a year, if the whole year had the properties of that timeslice. This means you take the full capital and fixed costs when calculating the price in each timeslice, with an adjustment according to the utilization factor in that timeslice (if the output in a timeslice is capped at half the maximum rate, then the capital costs per unit of commodity output are doubled).

When it comes to prices, if you're charging the this timeslice-level LCOE for the length of each timeslice, then I think you'll end up breaking even.

Obviously things would get a lot more complicated if the exponents were included, so I can understand why they were omitted.

We need to think this through a bit. Not sure LCOE at time slice level should assume the properties of that time slice apply to the whole year.

6. You wrote that "however other objectives such as fuel consumption cost, capital cost and emissions cost only use prices for the forecast year." - Did you mean to say "investment year" here? It might be odd to use values for the forecast year if it's far in the future.

No, I do mean the forecast year. Should we change this?

Yes I think so! Forecast year is the one further in the future right? But the investment year is the year the process comes online? If yes, capex etc should relate to the investment year, not the forecast year. In fact the whole current year, investment year, forecast year thing is annoying and confusing - MUSE 2.0 only uses current year (which is also assumed to be the investment year) - but I guess it's too hard to MUSE 1.0 to try to simplify these things.

1. Weighted average of LCOE is OK for estimating price is certain markets. But most markets are priced at the marginal price of the most expensive process (i.e. variable costs only, including input/outputs commodity costs, variable opex) - capital costs ignored etc. I think we're OK to stick with weighted average LCOE for now. But wasn't there a toml input parameter that could switch this to something else?

Seems like there's something in place if you use the "match" dispatch method (https://muse-os.readthedocs.io/en/latest/inputs/toml.html#standard-sectors), but I'd need to look into this more

2. Supply/demand balance does still influence price in this formulation - this is because as demand increases you have to buy more capacity - and as you buy more capacity you may hit a constraint and need to move to the next most expensive process - thus increasing overall price.

I see, good point

3. cap_exp and fix_exp and var_exp: all these are a bit annoying as they are hard to get data for and not that intuitive to use. I advise people to set them to a value where they do not influence the value of cap_par etc in the calculations (I think that's zero, right?).

Ok. Actually 1 would be the more appropriate value. Zero means the cost is fixed regardless of the amount of installed capacity or the production level, which doesn't make much sense. 1 means the cost is linear with capacity/production (e.g. cap_par * capacity, var_par * production)

Perhaps the way forward here is (a) making sure one common function is used for LCOE calculation everywhere in the model where it's needed,

Yes, I agree. I've homogenized things a lot in #462, but there are still 2 functions for calculating LCOE: lifetime_levelized_cost_of_energy and annual_levelized_cost_of_energy

The former aims to calculate the LCOE across the lifetime of the technology, and is used for the objectives, whereas the latter aims to calculate the LCOE in a single year, and is used to set commodity prices.

I think most of the code in these two functions could be combined into a single function, however there are subtle differences between the two functions that might need to be addressed (use of exponents, interest rates, how the capacity/production level is accounted for).

As above, needs some thought regarding exactly how LCOE is used to generate the prices. My hunch is that it should be identical to how this is done for objective calcuation re the scipy solver, but need to revisit that to remind myself how it worked.

and (b) modifying the code such that input prices are not needed for every commodity when you input them for one commodity (and modifying so that you can specify time slice level input prices)

This will be possible with the new database format that I have in mind. In theory this part of the database is 90% coded up already, so it might be possible to slot this in even though the full database is some way off.

Perfect

and (c) improving the documentation so that the behavior you've observed is clearly documented - and maybe incorporating some of these features into tutorials (e.g. how a price of an environmental commodity is used in the model, and input-only commodity, and how maximum_iterations influences price calculations

Agreed. Also could do with a tutorial using a carbon budget.

in fact maybe this last one also needs a code modification because it seems silly - instead user should be able to specify which commodities they want endogenous prices for and which they want to use input prices for)?

Agreed

[tsmbland]

I agree that the forecast year thing is extremely confusing.

Referring back to #196, I think the suggestions here are equivalent to suggestion (a) in #196, right? (i.e. no foresight). If we also want (b) to be an option, this would require using price data from the input file beyond the investment year, and that should only be permitted when using maximum_iterations=1 (i.e. using exogenous prices throughout the simulation).

Do you still want to go ahead with this two-option approach?

[tsmbland]

Also, what about demand and technology parameters? We should always be flat-forwarding values from the investment year, right?

[ahawkes]

@tsmbland - in relation to this see closed issue #246 and remaining open issue #288