From 6bd7c7b0aca44fa640c2b1a950653348311acd28 Mon Sep 17 00:00:00 2001 From: orichters Date: Fri, 12 Jul 2024 12:56:59 +0200 Subject: [PATCH 1/2] fail on duplicated column names --- CHANGELOG.md | 33 +++++++++++++++++-------- scripts/start/readCheckScenarioConfig.R | 6 ++++- 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f80ca6929..cb3b09855 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,35 +11,48 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). anymore; can be re-activated using `cm_wastelag` ### added -- added realizations diffExp2Lin and diffLin2Lin to 45_carbonprice [#1723](https://github.com/remindmodel/remind/pull/1723) +- **50_damages**, **51_internalizeDamages** add KotzWenz realization based on Kotz & Wenz (2024) + [[#1601](https://github.com/remindmodel/remind/pull/1601)] +- **45_carbonprice** added realizations diffExp2Lin and diffLin2Lin + [[#1723](https://github.com/remindmodel/remind/pull/1723)] +- **scripts** define defaults for script selections in output.R + [[#1739](https://github.com/remindmodel/remind/pull/1739)] +- **scripts** fail transparently on duplicated column names in scenario_config*.csv files + [[#1742](https://github.com/remindmodel/remind/pull/1742)] ### fixed - included CCS from plastic waste incineration in CCS mass flows so it is subject to injection constraints (but did not add CCS costs, see https://github.com/remindmodel/development_issues/issues/274 +- **scripts** fix tax convergence reporting in modelSummary + [[#1728](https://github.com/remindmodel/remind/pull/1728)] +- **scripts** cleanup non-existing realizations from settings_config.csv + [[#1718](https://github.com/remindmodel/remind/pull/1718)] ### removed ## [3.3.2] - 2024-07-04 ### changed -- fix output generation [[#1715]] (https://github.com/remindmodel/remind/pull/1715) +- fix output generation [[#1715](https://github.com/remindmodel/remind/pull/1715)] ## [3.3.1] - 2024-06-18 ### changed -- new input data (6.77) including new GDP and population data([#83](https://github.com/pik-piam/mrdrivers/pull/83)) [[#1684]](https://github.com/remindmodel/remind/pull/1684) +- new input data (6.77) including new GDP and population data + [[#83](https://github.com/pik-piam/mrdrivers/pull/83)] [[#1684](https://github.com/remindmodel/remind/pull/1684)] - **37_industry** remove subsector-specific shares of SE - origins in FE carriers for performance reasons [[#1659]](https://github.com/remindmodel/remind/pull/1659) -- **37_industry** make process-based steel production model the default over the ces-based model [[#1663]](https://github.com/remindmodel/remind/pull/1663) + origins in FE carriers for performance reasons [[#1659](https://github.com/remindmodel/remind/pull/1659)] +- **37_industry** make process-based steel production model the default over the ces-based model [[#1663](https://github.com/remindmodel/remind/pull/1663)] - **37_industry** fixed incineration of plastic and non-plastic waste causing non-zero emissions for biomass and synfuels - [[#1682]](https://github.com/remindmodel/remind/pull/1682) -- **core** another change of preference parameters and associated computation of interest rates/mark ups [[#1663]](https://github.com/remindmodel/remind/pull/1663) + [[#1682](https://github.com/remindmodel/remind/pull/1682)] +- **core** another change of preference parameters and associated computation of interest rates/mark ups [[#1663](https://github.com/remindmodel/remind/pull/1663)] - **scripts** do not check anymore that MAgPIE uses renv [[1646](https://github.com/remindmodel/remind/pull/1646)] -- **scripts** adjust function calls after moving functionality from `remind2` [[#578]]](https://github.com/pik-piam/remind2/pull/578) to `piamPlotComparison` and `piamutils` [[#1661](https://github.com/remindmodel/remind/pull/1661) -- **scripts** enhance output script `reportCEScalib` to include additional plot formats [[#1671](https://github.com/remindmodel/remind/pull/1671) +- **scripts** adjust function calls after moving functionality from `remind2` + [[#578](https://github.com/pik-piam/remind2/pull/578)] to `piamPlotComparison` and `piamutils` [[#1661](https://github.com/remindmodel/remind/pull/1661)] +- **scripts** enhance output script `reportCEScalib` to include additional plot formats [[#1671](https://github.com/remindmodel/remind/pull/1671)] ### added - **24_trade** add optinal trade scenario for EUR hydrogen and e-liquids imports [[#1666](https://github.com/remindmodel/remind/pull/1666)] @@ -48,7 +61,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### changed - **37_industry** changed industry to have subsector-specific shares of SE - origins in FE carriers [[#1620]](https://github.com/remindmodel/remind/pull/1620) + origins in FE carriers [[#1620](https://github.com/remindmodel/remind/pull/1620)] ### added - **config** regex tests for many parameters [[#1356](https://github.com/remindmodel/remind/pull/1356)] diff --git a/scripts/start/readCheckScenarioConfig.R b/scripts/start/readCheckScenarioConfig.R index d63a1de25..fac321488 100644 --- a/scripts/start/readCheckScenarioConfig.R +++ b/scripts/start/readCheckScenarioConfig.R @@ -26,6 +26,10 @@ readCheckScenarioConfig <- function(filename, remindPath = ".", testmode = FALSE scenConf <- scenConf[! is.na(scenConf[1]), ] rownames(scenConf) <- scenConf[, 1] scenConf[1] <- NULL + colduplicates <- grep("\\.[1-9]$", colnames(scenConf), value = TRUE) + if (length(colduplicates) > 0) { + warning("These colnames are signs of duplicated columns: ", paste(colduplicates, collapse = ", ")) + } toolong <- nchar(rownames(scenConf)) > 75 if (any(toolong)) { warning("These titles are too long: ", @@ -120,7 +124,7 @@ readCheckScenarioConfig <- function(filename, remindPath = ".", testmode = FALSE scenConf[, names(path_gdx_list)[! names(path_gdx_list) %in% names(scenConf)]] <- NA # collect errors - errorsfound <- sum(toolong) + sum(regionname) + sum(nameisNA) + sum(illegalchars) + whitespaceErrors + copyConfigFromErrors + pathgdxerrors + missingRealizations + errorsfound <- length(colduplicates) + sum(toolong) + sum(regionname) + sum(nameisNA) + sum(illegalchars) + whitespaceErrors + copyConfigFromErrors + pathgdxerrors + missingRealizations # check column names knownColumnNames <- c(names(cfg$gms), setdiff(names(cfg), "gms"), names(path_gdx_list), From 3ec4b255fc9523446523b23f72a0599e3eecec88 Mon Sep 17 00:00:00 2001 From: orichters Date: Fri, 12 Jul 2024 13:48:11 +0200 Subject: [PATCH 2/2] add link to debugging tutorial --- tutorials/10_DebuggingREMIND.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tutorials/10_DebuggingREMIND.md b/tutorials/10_DebuggingREMIND.md index b41c60cd8..968d67666 100644 --- a/tutorials/10_DebuggingREMIND.md +++ b/tutorials/10_DebuggingREMIND.md @@ -124,9 +124,9 @@ An explanation of the modelstat and solvestat numbers can be found in the tables |Modelstat = 6 | Intermediate Infeasible| |Modelstat = 7 | Intermediate Nonoptimal| | | | -|Solvestat = 1 | Normal Completion| -|Solvestat = 2 | Iteration Interrupt| -|Solvestat = 3 | Resource Interrupt| +|Solvestat = 1 | Normal Completion | +|Solvestat = 2 | Iteration Interrupt | +|Solvestat = 3 | Resource Interrupt | |Solvestat = 4 | Terminated by Solver| |Desirable Status in REMIND| @@ -134,6 +134,7 @@ An explanation of the modelstat and solvestat numbers can be found in the tables |Solve + Model stat = 1 + 2 | solution found| |Solve + Model stat = 4 + 7 | feasible but slow convergence| +Further possible error codes can be found in [the GAMS documentation](https://www.gams.com/latest/docs/UG_GAMSOutput.html#UG_GAMSOutput_SolverStatus). If infeasibilities show up already in the first iteration, it may be related to a wrong `input.gdx` (specified with `path_gdx` in the `scenario_config_XYZ.csv`) or some general error in the GAMS code. Via the international trade, infeasibilities in one region may propagate to other regions in later iterations, but then it is worth knowing where it started. There are different types of solver infeasibilities: pre-triangular and optimization infeasibilities. In pre-triangular infeasibilities, GAMS shows you in the solution report the equations that are incompatible with each other. For optimization infeasibilies, the CONOPT solver tries to reduce the infeasibility to the thing that less affects the objective function. It does not show all affected equations, as it is not a simple problem as a non-square system of equations like in the pre-triangular case. You need to check if it is bound-related: the variables bounds, and the equation bounds starting from the infeasibility and going through the variables that have relation with it. You can always force in another run the variable that is infeasible to a feasible value to see what else is affected by it. But this is usually not necessary as just checking the logic behind the equation and the infeasible variable is usually sufficient to find the limitation.