Level of abstraction in the model design

[jkiviluo]

We've had couple of telco discussions concerning the level of abstraction in the model design. Here are some guidelines based on those discussions:

• Aim for high level of abstraction in model equations
• Nodes are fundamental and they can have a state variable
• Both transfer and conversion are processes between nodes. They are expressed as flow variables.
• Input data structure can be more specific: units can be units and transmission lines can be transmission lines
• Between input data and the abstract model structure there is an interpretation layer (when needed)
• Reservation of capacity (reserve) is treated separate from a deployment of a process (energy)

[jkiviluo]

Discussion topic: will markets require another index in variables or are we handling them with a structure that is not directly in the constraints (code around the solves).

[jkiviluo]

I would suggest to keep the market structure outside of the equations. It's in the solve looping structure (market decisions and market prices are stored in the looping code). That keeps the model insides more understandable.

[Tasqu]

I think it might be possible to represent markets as nodes, as I believe most of the markets only require a constraint ensuring that the sold product (be they energy or reserve capacity) equals or at least doesn't exceed the corresponding production. However, I'm not 100% familiar with all the possible rules and regulations of different markets, and if they can pose more exotic requirements for production that need to be handled via specific constraints.

The main challenge I see regarding treating markets as nodes is figuring out a nice way to treat the different intervals and closings of multiple different markets at different times. Most likely, this would require supporting different temporal and stochastic structures on different nodes, somewhat similar to what we're doing in SpineOpt. Furthermore, we'd need some way of fixing variables for markets as soon as they close, which I think could maybe be handled using the ever-elusive multi-model framework (which has some ties with the static vs rolling discussion as well). I'll try to illustrate my idea:

1. Each market would have a separate model defined for it, with different outputs linked with it. E.g. a day-ahead market would have day-ahead market variables as its output, a 2-days-ahead market would have 2-days ahead market variables as its output, etc.
2. The different models are solved in a desired sequence, and all the variables output by a previous model would be treated as fixed for the subsequent models.
3. All the models would use a common library for accessing/storing the variables, so controlling the sequence and output of the models would determine which variables are fixed, and when.

This is certainly not the easiest way to handle this, but could permit the users quite a lot of freedom how to set up different sequences of markets or other models with interacting outputs if it could be pulled off.

[Mastomaki]

For (1) we need to prepare also for future markets. For example day-ahead (DA) market would anticipate that FCR market is approaching also for the next day, and thus FCR offering is also solved. And every hour there is an update of energy storage content (possibly other things too), which means that scheduling must be updated, although it is not related to any market.

[jkiviluo]

Discussion topic: what temporal and stochastic properties should be included?
Includes Juha's proposal for Predicer

• Time resolution can change between time steps (does someone disagree?)
  • Yes
• Time resolution can change between nodes?
  • Yes (can be distinguishing factor from other similar models and allows to make faster solves for e.g. complicated hydro systems)
• Single stochastic structure to handle both 'samples' and 'forecasts'?
  • Yes, time dimensions would be: [sample/forecast, t]
• Stochastic branches can have arbitrary form (or just single branching)?
  • No, just single branching
• Stochastic branches can get back to single forecast?
  • Yes
• Rolling and static temporal windows? Would it be ok to just have rolling (static is then '1-roll')?
  • Yes, just rolling and static is made by having only one solve.

[Tasqu]

My two cents:

• Time resolution can change between time steps (does someone disagree?)
  • Agreed. In my experience, this isn't terribly difficult to achieve and can help reduce computational times in simulations requiring long-model horizons, e.g. hydro-thermal scheduling.
• Time resolution can change between nodes?
  • Can be useful for energy system integration stuff over energy networks that operate on different time-scales, sure. However, I'm not sure how crucial it is for market-oriented single-actor optimisation, as the markets tend to operate on similar time-scales, right? This makes the code quite a bit more complicated, because all constraints over two or mode nodes need to account for the possibility of different temporal resolutions on each node.
• Single stochastic structure to handle both 'samples' and 'forecasts'?
  • Agreed. We never should've used separate indexes for them in the first place, in my opinion. As far as I'm concerned, the only difference between the two is how they connect to each other.
• Stochastic branches can have arbitrary form (or just single branching)?
  • I believe that single branching should be sufficient for single-actor market-oriented optimisation.
• Stochastic branches can get back to single forecast?
  • I agree that this is again useful for long horizons (e.g. hydro-thermal scheduling), since we typically don't have meaningful stochastic data spanning months/years into the future. However, if we want to handle this properly, it necessitates the "stochastic path indexing" of constraints similar to SpineOpt, which is somewhat complicated and makes the model code quite a bit harder to read.
• Rolling and static temporal windows? Would it be ok to just have rolling (static is then '1-roll')?
  • I've never really seen the point of this discussion. "Rolling" can be seen as a number of static problems run in a sequence, or "Static" as a single-step rolling. It's the same thing in the end. Personally, I'd like to try and implement a more generic framework for rolling with "multi-model support" as I mentioned in the discussion about markets, but I'm not sure if this is the project for trying something new like this.

[jkiviluo]

Personally, I'd like to try and implement a more generic framework for rolling with "multi-model support" as I mentioned in the discussion about markets, but I'm not sure if this is the project for trying something new like this.

That should be working in FlexTool3 and can give an example. However, the looping of the solves (and hence different models) happens in the code around the model (in this case, it's Python code).

[Mastomaki]

• Stochastic branches can have arbitrary form (or just single branching)?
  • at least in some cases single branching has been found sufficient [1]. But in our reality there are several decision stages and random variables which are realized after decisions, so is it sufficient? E.g. DA spot market -> DA FCR market.

1. Badesa, L., Teng, F., and Strbac, G., Simultaneous Scheduling of Multiple Frequency Services in Stochastic Unit Commitment, IEEE Trans. Power Syst., vol. 34, no. 5, pp. 3858–3868, 2019, https://doi.org/10.1109/TPWRS.2019.2905037

[Tasqu]

at least in some cases single branching has been found sufficient [1]. But in our reality there are several decision stages and random variables which are realized after decisions, so is it sufficient? E.g. DA spot market -> DA FCR market.

Hmm, I suppose you're right. Since the different markets have different closing windows (and if all markets have price forecast data and hence price scenarios), the scenarios for the different markets branch at different times. If we want to properly account for this, we need to support at least some measure of interacting stochastic structures between the markets. That's going to make dealing with the stochastics a bit more complicated already.

If the converging of forecasts is to be supported as well, I suppose we'll have to implement a similar "stochastic path indexing" for constraints as is used in SpineOpt.

[Tasqu]

Drafting abstract core structure - Part I: Overall model structure

(Updated 1.10.2021)

In my opinion, the main benefit of an abstract structure is the possibility to minimise the number of different entities in the model, and comprehensively handle all possible interactions between them. This should, in principle, make the model extremely flexible, although not necessarily easy to use. However, it should be possible to implement ease-of-use later in the descriptive-to-abstract input layer.

I thought I'd throw some rough draft of a potential model variables and the corresponding here to serve as a basis for further discussion. Eventually, we might want to open a new discussion board for this topic, but for now, I've kept things here.

To me, the model structure can be separated into two to four distinct parts, depending on how we want to handle things.

1. System structure: This category contains the different entities we need to define the system itself: the commodities, their flows, generation assets, transfer/transportation assets, etc.
2. Temporal structure: This category contains the entities needed to define how time and stochastics work. E.g. time resolution, forecasts/samples, potential stochastic branching/converging, etc.
3. Market structure?: Depending how markets are implemented, they might not fit neatly into the system structure, but this remains to be seen.
4. Meta structure? In case we want to implement some sort of multi-model support, we need some data structure able to define the properties of individual models, as well as defining how they interact with each other.

System structure

Here's what I believe to be the absolute minimum number of entities for describing energy systems as we've modelled them so far:

1. node is an entity responsible for recording the "state" of things, e.g. enforcing commodity balance and recording any potentially stored commodities. In principle, unit online status and investment variables can be represented using nodes as well.
2. process is an entity responsible for modifying commodity flows between nodes. Essentially, it's a group of "flow" variables bound together by constraints governing how the "flows" should behave with respect to one another, or in respect to node "state" variables.
3. connection? is an entity responsible for direct transfer of commodities between nodes. connections aren't strictly required, as processes are more generic by nature, but depending on how the variables are indexed, there might be a difference between the two.

System variables

Depending on the application, all the variables can be either continuous, integer, or binary.

1. node_state(node, stochastic, temporal) depicts the status of a node on each stochastic time step.
2. process_flow(process, direction, node, stochastic, temporal) depicts the "flow" between the process and the nodes.
3. connection_flow(connection, node, node, stochastic, temporal) depicts the "flow" directly between two nodes.

Challenges with the minimal abstract system structure

For extremely simple use cases, the above structure remains rather easy to use. However, more advanced use-cases like unit online states, investments, reserves, ramps, etc. require quite complicated definitions and constraints between the entities and variables. In principle, the complexity can be dealt with during the descriptive-to-abstract layer, but it might nonetheless make interpreting the core model more challenging.

Furthermore, implementing different conversion efficiency approximations for processes, and especially making them time-dependent like in Backbone, would require further thought.

How to make the descriptive-to-abstract layer so that it can also be used in reserve to obtain descriptive results from the abstract results?

Temporal structure

The simplest and also possibly the most flexible temporal structure is pretty much what we've used in SpineOpt.

1. temporal_block or whatever one wants to call it. Essentially, it represents a period of time with a set time resolution.

SpineOpt uses another entity called stochastic_scenario for defining the stochastics, but I believe it's not strictly necessary. temporal_blocks by themselves can be interpreted as being stochastic entities: two temporal_blocks overlapping each other can be interpreted as being "parallel timelines", or "forecasts", and "samples" are no different. The only challenge is to define how the temporal_blocks interact with each other.

Market structure? (Updated 1.10.2021)

Depending on how the markets are handled, they might not fit neatly into the system structure and require a layer of their own.

Personally, I'd like to try handling them via "multi-model" interactions using the meta structure.

1. Instead of having any dedicated market variables or data structures in the model, I'd try to use nodes to represent the market balance.
2. Then, a different model would be defined for each market we want to solve with different definitions for which market variables are recorded as output and thus fixed for any subsequent solves.
3. By solving the models of the different markets in a desired sequence, we obtain the optimal bids for each market while the already sold market products remain fixed based on previous solutions.

Separating each market phase into a standalone model hopefully also helps with robustness and customizability of the daily multi-market bidding loop as well.
As far as I've understood, there might be a need to tweak the input data between market steps e.g. based on actual market clearance.

Meta structure?

Essentially, the point of the meta structure would be to enable defining different model objects with different instructions, and allow for executing these models in sequence.
I believe such structures could be used to fix desired market variables after market closures without separate handling of markets in the model code.
However, whether or not this is actually doable, I can't say for certain.

There are other benefits that I believe a more modular approach to multi-model and rolling simulations via better meta structure could achieve:

1. Parallelization: The way I see it, the main bottleneck for large-scale rolling simulations in particular, is the performance of the solver. This means that everything besides the solving process should be handled in parallel, if at all possible.
   • If the model could be separated into functions e.g. preprocess_model, solve_model, postprocess_model, it might be possible to e.g. start pre-processing data for the next rolling iteration parallel to solving the first iteration, and then start the next iteration immediately as soon as the current solve finishes.
2. Robustness: Currently, our rolling optimisations tend to be one massive undertakings, that don't write any intermediary results at any point. If a single solve somewhere in the process fails, the whole process fails. Treating rolling optimisation as a sequence of "static" solves might help with this, since intermediary results could be saved at desired intervals, and a new sequence could be started using previous results as initial values.
   • Of course, performance could be impacted by constant saving of values, but hopefully this too could be parallelised if need be.

[Tasqu]

There are probably a few pitfalls related to pre- and post-processing that use results from the solves, but hopefully most of the work can be done in parallel.

[Mastomaki]

Regarding the market structure, for example Aasgård et al. [1] model energy market just as set of stochastic scenarios with different prices and associated bid power variables. So no separate structures would be needed. For reserve markets this would be stochastic scenarios with different reserve capacity prices and associated v_reservedemand variables. Then there is the reserve activation, whose stochasticity could also be taken into account. All this creates a lot of scenarios of course.

[1] Ellen Krohn Aasgård et al. Optimizing day-ahead bid curves in hydropower production

[jkiviluo]

System structure

As Topi notes above, it is not necessary to have connection object an the abstract model. It can also be implemented by having just v_flow variable that is indexed with [process, source, sink, t]. 'Process' may be extra in principle, but helps to form the conditionals and to keep the data organized. Anyway, this means that a flow variable can be set between:

• a source node and a sink node (simple unit or a 2-way transfer connection without losses)
• a source node and a unit/connection
• a unit/connection and a sink node

The latter two are use for units an connections that need more than one variable (and there can be multiple of them to allow multiple inputs and outputs).

That's how it's done in FlexTool3 - with methods defining how to create variables and constraints. I think the main benefit is that the node has only v_flow coming in an going out, which simplifies the constraints.

[jkiviluo]

Temporal structure

I think this requires bit more complexity (at least on the 'descriptive' side). The issue is that there are historical time series and future assumptions that need to get mixed in models in various ways (multi-period investment model vs. rolling dispatch). However, the abstract model probably needs just a sample and t with appropriate temporal links and weights.

[Tasqu]

The issue is that there are historical time series and future assumptions that need to get mixed in models in various ways (multi-period investment model vs. rolling dispatch)

As far as I understand, rolling dispatch is essentially just solving a number of "static" problems in a sequence, so in my draft above the "rolling" part of dispatch would mostly be handled using the Meta Structure. However, I'm not familiar with the concept of "multi-period investments", so I'm not sure what type of temporal/stochastic structures are required to represent those.

[Tasqu]

Drafting abstract core structure - Part II: Temporal and stochastic structures the "generic" way

As far as I understand the objectives of our part in the HOPE project and the aim of the Predicer, the main focus should be on solving a single stochastic optimization problem corresponding to a desired market, resulting in optimal bidding under multi-market uncertainty. Based on the above discussions, the key features we seem to want are:

1. Variable time resolution for the timesteps (as a function of model horizon).
2. Different time resolution for different parts of the system (e.g. nodes and units).
3. Branching and converging stochastic scenarios.
4. Rolling and static temporal windows (support for representative day type of approach).

Essentially, it seems we want maximum flexibility in our temporal/stochastic structure, which requires quite a bit of definitions.

Temporal and stochastic data structure

While the structure we use in SpineOpt can basically handle all of the above, I believe it could be streamlined and clarified to make it appear less intimidating. I would propose using only a single type of data object to represent both time and stochastic scenarios, and potentially another to act as a "handle" for easier linking with the desired parts of the modelled system:

Objects

Based on my experiences with SpineOpt, supporting temporal structures ranging from continuous rolling-window-types to potentially discontinuous representative-day-types requires at least the following:

• temporal_block or timeline or whatever we want to call it. The basic building block for defining temporal and stochastic structures.
• temporal_structure or whatever. A handle representing a group of temporal_blocks with defined parameters for their properties and interactions.

In principle, the temporal_structure is not strictly necessary, but without it every system object (node, unit, etc.) needs to be connected to each and every relevant temporal_block forming the desired structure.

Relationships

In order to define different temporal_structures, we need to include the necessary temporal_block objects with parameters determining how they are interpreted.

• temporal_structure__temporal_block defines which temporal_block objects are included in which temporal_structure, as well as basic parameters to control the "shape" of the temporal_structure:
  • block_start & block_end define when a specific temporal_block starts and when it ends. For Predicer, these parameters are mostly defined relative to when the model starts, but for representative days it's often easier to fix the exact timestamp.
  • time_resolution defines the time resolution of the temporal_block in the current temporal_structure.
  • weight_relative_to_parents defines the weight given to the temporal_block in the objective function. Easiest to give relative to parents in case of potentially different branching/converging scenarios.
• parent_temporal_block__child_temporal_block defines how the temporal_blocks are related to each other. Required for defining the "stochastic scenario tree". Note that this cannot be temporal_structure dependent without massively complicating interactions between different temporal_structures.

Modularity of the temporal/stochastic structure

Every energy system model requires some way to define how time and stochastics work, and it's always a modicum of work to handle that. I wonder if we could isolate handling the temporal/stochastic structure into an isolated module, so that it could potentially be reused with later model-development efforts to save time? All the energy system models really need from a temporal/stochastic structure are the (scenario, timestamp) indexes for generating variables and constraints, as well as convenient ways to navigate them.

Known challenges based on experiences with SpineOpt

1. Having a temporal/stochastic structure as flexible as this permits defining extremely challenging cases with both "relative" (defined relative to model start time) and "static" (defined using fixed timestamp) temporal_blocks. The same structure cannot be repeated in a rolling fashion without eventually running into conflicts between overlapping relative and static temporal_blocks.
2. The possibility to define discontinuous time (e.g. representative days here and there) needs to know what happens during/over these missing time periods when dynamic constraints try to access variables on these missing periods. This is also related to how initial values for the model are handled.
3. Supporting converging scenarios means that we'll have use something like the "stochastic path indexing" of constraints to keep them unambiguous, which can make constraint code more difficult to understand and adapt.

[Tasqu]

Drafting abstract core structure - Part III: System structure, model variables, and constraints

In Part I I already proposed a minimal example of a possible system structure. However, it's not enough to really start trying things on the implementation side. Here I'm trying to draft things further to the variables and constraint level, so that things hopefully become more concrete from the model development perspective.

System data structure

Here's what I believe to be the absolute minimum number of entities for describing energy systems as we've modelled them so far:

1. node is an entity responsible for recording the "state" of things, e.g. enforcing commodity balance and recording any potentially stored commodities. In principle, unit online status and investment variables can be represented using nodes as well.
2. process is an entity responsible for modifying commodity flows between nodes. Essentially, it's a group of "flow" variables bound together by constraints governing how the "flows" should behave with respect to one another, or in respect to node "state" variables.
3. connection? is an entity responsible for direct transfer of commodities between nodes. connections aren't strictly required, as processes are more generic by nature, but depending on how the variables are indexed, there might be a difference between the two.

The core system data structure objects probably don't need further additions, do they?

System variables

1. node_state(node, stochastic, temporal) depicts the status of a node on each stochastic time step.
2. process_flow(process, direction, node, stochastic, temporal) depicts the "flow" between the process and the nodes.
3. connection_flow?(connection, node, node, stochastic, temporal) depicts the "flow" directly between two nodes.

4. node_slack(node, pos/neg, stochastic, temporal) for allowing and penalizing breaching commodity balance on nodes. Might actually contain a group of variables with different costs, similar to process_flow conversion efficiency approximations.

The above is the absolute minimum number of variables I believe to be necessary to represent our types of energy system models. However, representing more complex technical aspects of energy systems using only these not be practical.

Known challenges based on Backbone/SpineOpt

• Ramps, as they are a rather specific constraint for certain types of power plants. Depending on whether one needs to capture a cost of ramping and in what manner, modelling ramps can require quite complicated dynamic node_state and process_flow constraints.
• Reserves, as they have rather distinct properties for general energy system models. However, for Predicer, they might not be as problematic, as they can be mostly treated as just another product for sale similar to energy output. Problems mainly arise for storages, which need to consider how their state would change if the reserve product was actually activated.
• Conversion efficiency approximations, as depending on their complexity the process_flow variable might in fact comprise a group of SOS2 variables, for example. Fortunately, JuMP has the @expression macro that Per used quite nicely in aFXR to "hide" that complexity in the majority of constraints.
  • These become even more challenging if we want to implement the possibility to change the approximation as a function of model horizon, like Backbone does.
• Startup, shutdown, online, and investment decisions, as these typically have interactions with considerable time delays, e.g. minimum up and downtime constraints or investment lifetimes. Technically, these could be represented using node_state and process_flow variables. However, that would require more complicated forms of input data we've used in the past, as we'd need to be able to define coefficients with multiple time delays.

For easier writing of the core constraints, it might be helpful to include dedicated variables for some/all of the above. However, each additional type of variable makes the model core less flexible, unless appropriate constraints are also added, which can quickly result in a large number of constraints.

System constraints

Based on the core system data structure objects node, unit, and possibly connection, we can derive a few core system constraints based on them:

1. Node commodity balance constraint, which ensures that incoming and outgoing process_flows are in equilibrium with the node_state. Depending on the desired application, there can be a few potential variations for this constraint
   • Regular ==, <=, or => as with most constraints.
   • "Dynamic" balance, meaning the node has a node_state variable that has to account for its value on previous time steps.
   • "Instantaneous" balance, meaning the node_state variable only acts as an instant "mismatch" variable, without considering its value on previous time steps.
2. Process flow balance constraint ensuring that the incoming and outgoing process_flow variables are in equilibrium.
3. Connection flow balance constraint, if connections are a thing.

Furthermore, since Predicer is supposed to handle market bidding, we'll at least need the following market constraint:

4. Bidding constraint, which enforces a "price order" for the market variables across the price forecasts (stochastic scenarios). Essentially, the bidding constraint prevents selling more on scenarios with lower prices than on scenarios with higher prices. As far as I understand, it's required for forming a logical bid matrix.

Generic system constraints

In addition to the above core system constraints, if we want the abstract core to be truly generic, we'll need constraints that permit constraining any variables by any other variables. Naturally, the more types of different variables are in the model core, it becomes increasingly more difficult to account for every potential interaction.

Naturally, there probably won't ever be a need to actually account for every potential interaction. However, if such a need arises due e.g. new techologies to be modelled or more detailed technology description, it can't be modelled without adding new code to the model core without comprehensive generic constraints.

Furthermore, if we want to be able to constraint variables by other variables or themselves across timesteps, we'll need more complicated input data in order to be able to define parameter values at different lags.

Technical system constraints

These are the foreseen constraints dealing with the detailed technical aspects of energy systems, as highlighted in the previous sections. Depending on the level of abstraction of the model core, these constraints could be formulated using the above core system constraints, but whether that's feasible in practise remains to be seen. If ramps, reserves, conversion efficiency approximations, startups, shutdowns, online status, and investment decisions are modelled using dedicated variables, then some/all of the following constraints become necessary as well:

• Ramp constraints enforcing process_flow doesn't change too fast between timesteps.
• Reserve constraints enforcing the peculiarities of providing reserves e.g. from storages.
• Conversion efficiency constraints are required separately, unless the process flow balance constraint and generic system constraints can handle them.
• Startup, shutdown, and online constraints for e.g. min uptime and downtime, unless they are handled via node commodity balance constraint and the generic system constraints.
• Investment constraints, unless handled via node commodity balance constraint and generic constraints.

Known challenges based on Backbone/SpineOpt development

Including dedicated variables and constraints for the technical aspects of e.g. power systems is an easy way to build a model, but eventually the amount of variables and constraints handling them seems to become so large that no one person has a handle on the whole. Learning, understanding, and customizing the model becomes increasingly more difficult as the size of the codebase increases.

However, while a more abstract approach can (in principle) drastically reduce the amount of model code required, it mostly shifts the complexity to the input data side. Will it be practical having to define complex interactions via complex definitions in the input data?