diff --git a/README.md b/README.md index 479f066d..40aa117f 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ SDEverywhere allows for unit testing, continuous integration, the ability to compare model runs, and a toolchain to transform a model into C, JavaScript, and WebAssembly code. Using SDEverywhere, you can deploy interactive System Dynamics models on mobile, desktop, and the web for policymakers and the public. -Or you could perform model analysis using general-purpose languages, running the model as high-performance C code. +Or you could perform model analysis using general-purpose languages, running the model as high-performance JavaScript or C code. By following the ["Quick Start"](#quick-start) instructions below, within minutes you can turn a Vensim model like this: @@ -20,7 +20,7 @@ By following the ["Quick Start"](#quick-start) instructions below, within minute ## Supported Platforms The SDEverywhere tools can be used on on any computer running macOS, Windows, or Linux. -The libraries and code generated by SDEverywhere can be run in web browsers on almost any device (including phones and tablets) as well as on the server side (using C/C++ or WebAssembly). +The libraries and code generated by SDEverywhere can be run in web browsers on almost any device (including phones and tablets) as well as on the server side (using C/C++, JavaScript, and/or WebAssembly). ## Supported File Formats @@ -46,7 +46,7 @@ https://github.com/climateinteractive/SDEverywhere/assets/438425/cd495776-5744-4 ### Quality Checks and Comparisons -SDEverywhere includes extensive QA (quality assurance) tools that you can run locally on your machine as you develop your model or in the cloud in a continuous integration environment. +SDEverywhere includes extensive QA (quality assurance) tools that you can run as you develop your model, either locally on your machine or in the cloud in a continuous integration environment (or both). There is support for two different kinds of tests: - **_Checks:_** Check tests can be used to verify that your model behaves according to some guidelines (or conforms to your mental models). For example, you can have checks that verify that stocks are always positive, or that some output variable produces values that are close to historical reference data. @@ -63,17 +63,17 @@ https://github.com/climateinteractive/SDEverywhere/assets/438425/b6f05b3f-f18a-4 ## Core Functionality -At its core, SDEverywhere includes a [transpiler](https://en.wikipedia.org/wiki/Source-to-source_compiler) that can read a [Vensim](https://www.vensim.com/documentation/index.html?ref_language.htm) model and generate a high-performance version of that model in the C programming language. +At its core, SDEverywhere includes a [transpiler](https://en.wikipedia.org/wiki/Source-to-source_compiler) that can read a [Vensim](https://www.vensim.com/documentation/index.html?ref_language.htm) model and generate a high-performance version of that model as JavaScript or C code. -The [`sde`](./packages/cli) command line tool — in addition to generating C code — provides a plugin-based build system for extended functionality: +The [`sde`](./packages/cli) command line tool — in addition to generating JavaScript or C code — provides a plugin-based build system for extended functionality: - [plugin-config](./packages/plugin-config) allows you to configure your model/library/app using CSV files -- [plugin-wasm](./packages/plugin-wasm) converts the generated C model into a fast WebAssembly module for use in a web browser or Node.js application -- [plugin-worker](./packages/plugin-worker) generates a Web Worker that can be used to run the WebAssembly model in a separate thread for improved user experience +- [plugin-wasm](./packages/plugin-wasm) converts a generated C model into a fast WebAssembly module for use in a web browser or Node.js application +- [plugin-worker](./packages/plugin-worker) generates a worker module that can be used to run your model in a separate thread for improved user experience - [plugin-vite](./packages/plugin-vite) provides integration with [Vite](https://github.com/vitejs/vite) for developing a library or application around your generated model - [plugin-check](./packages/plugin-check) allows for running QA checks and comparison tests on your model every time you make a change -Additionally, the [runtime](./packages/runtime) and [runtime-async](./packages/runtime-async) packages make it easy to interact with your generated WebAssembly model in a JavaScript- or TypeScript-based application. +Additionally, the [runtime](./packages/runtime) and [runtime-async](./packages/runtime-async) packages make it easy to interact with your generated model in a JavaScript- or TypeScript-based application. For more details on all of these packages, refer to the ["Packages"](#packages) section below. @@ -296,6 +296,15 @@ Most users won't need to interact with these implementation packages directly, b Changelog + + @sdeverywhere/parse * + + + Source |  + Docs |  + Changelog + + `model-check` implementation @@ -328,7 +337,7 @@ There is still much to contribute, for example: - Enhance the C code generator to produce code for new language features now that you can parse them. - Implement more Vensim functions. This is the easiest way to help out. - Add support for the XMILE file format. -- Target languages other than C, such as R or Rust. (If you want Python, check out the excellent [PySD](https://github.com/JamesPHoughton/pysd)). +- Target languages other than JavaScript and C, such as R or Rust. (If you want Python, check out the excellent [PySD](https://github.com/JamesPHoughton/pysd)). For more guidance on contributing to SDEverywhere, please consult the [wiki](https://github.com/climateinteractive/SDEverywhere/wiki). diff --git a/packages/cli/src/c/vensim.c b/packages/cli/src/c/vensim.c index 341b4b90..51339260 100644 --- a/packages/cli/src/c/vensim.c +++ b/packages/cli/src/c/vensim.c @@ -94,7 +94,7 @@ double __lookup(Lookup* lookup, double input, bool use_inverted_data, LookupMode // Interpolate the y value from an array of (x,y) pairs. // NOTE: The x values are assumed to be monotonically increasing. - if (lookup == NULL) { + if (lookup == NULL || lookup->n == 0) { return _NA_; } @@ -162,9 +162,16 @@ double __lookup(Lookup* lookup, double input, bool use_inverted_data, LookupMode // This function is similar to `__lookup` in concept, but Vensim produces results for // the GET DATA BETWEEN TIMES function that differ in unexpected ways from normal lookup // behavior, so we implement it as a separate function here. -double __get_data_between_times(double* data, size_t n, double input, LookupMode mode) { +double __get_data_between_times(Lookup* lookup, double input, LookupMode mode) { // Interpolate the y value from an array of (x,y) pairs. // NOTE: The x values are assumed to be monotonically increasing. + + if (lookup == NULL || lookup->n == 0) { + return _NA_; + } + + const double* data = lookup->data; + const size_t n = lookup->n; const size_t max = n * 2; switch (mode) { diff --git a/packages/cli/src/c/vensim.h b/packages/cli/src/c/vensim.h index 51e4e268..84f8b970 100644 --- a/packages/cli/src/c/vensim.h +++ b/packages/cli/src/c/vensim.h @@ -74,9 +74,9 @@ double __lookup(Lookup *lookup, double input, bool use_inverted_data, LookupMode #define _WITH_LOOKUP(x, lookup) __lookup(lookup, x, false, Interpolate) double _LOOKUP_INVERT(Lookup* lookup, double y); -double __get_data_between_times(double *data, size_t n, double input, LookupMode mode); +double __get_data_between_times(Lookup* lookup, double input, LookupMode mode); #define _GET_DATA_MODE_TO_LOOKUP_MODE(mode) ((mode) >= 1) ? Forward : (((mode) <= -1) ? Backward : Interpolate) -#define _GET_DATA_BETWEEN_TIMES(lookup, x, mode) __get_data_between_times((lookup)->data, (lookup)->n, x, _GET_DATA_MODE_TO_LOOKUP_MODE(mode)) +#define _GET_DATA_BETWEEN_TIMES(lookup, x, mode) __get_data_between_times(lookup, x, _GET_DATA_MODE_TO_LOOKUP_MODE(mode)) // // DELAY FIXED diff --git a/packages/compile/src/generate/gen-code-c.js b/packages/compile/src/generate/gen-code-c.js index 346491ed..d5dae23a 100644 --- a/packages/compile/src/generate/gen-code-c.js +++ b/packages/compile/src/generate/gen-code-c.js @@ -81,51 +81,54 @@ ${section(Model.dataVars())} // function emitInitLookupsCode() { mode = 'init-lookups' - let code = `// Internal state + let code = `\ +// Internal state bool lookups_initialized = false; bool data_initialized = false; + ` code += chunkedFunctions( 'initLookups', Model.lookupVars(), - ` // Initialize lookups. - if (!lookups_initialized) { -`, - ` lookups_initialized = true; - } -` + `\ + // Initialize lookups. + if (lookups_initialized) { + return; + }`, + ' lookups_initialized = true;' ) code += chunkedFunctions( 'initData', Model.dataVars(), - ` // Initialize data. - if (!data_initialized) { -`, - ` data_initialized = true; - } -` + `\ + // Initialize data. + if (data_initialized) { + return; + }`, + ' data_initialized = true;' ) return code } function emitInitConstantsCode() { mode = 'init-constants' - return ` -${chunkedFunctions('initConstants', Model.constVars(), ' // Initialize constants.', ' initLookups();\n initData();')} -` + return chunkedFunctions( + 'initConstants', + Model.constVars(), + ' // Initialize constants.', + ' initLookups();\n initData();' + ) } function emitInitLevelsCode() { mode = 'init-levels' - return ` -${chunkedFunctions( - 'initLevels', - Model.initVars(), - ` + return chunkedFunctions( + 'initLevels', + Model.initVars(), + `\ // Initialize variables with initialization values, such as levels, and the variables they depend on. _time = _initial_time;` -)} -` + ) } // @@ -134,11 +137,9 @@ ${chunkedFunctions( function emitEvalCode() { mode = 'eval' - return ` -${chunkedFunctions('evalAux', Model.auxVars(), ' // Evaluate auxiliaries in order from the bottom up.')} - -${chunkedFunctions('evalLevels', Model.levelVars(), ' // Evaluate levels.')} -` + return `\ +${chunkedFunctions('evalAux', Model.auxVars(), ' // Evaluate auxiliaries in order from the bottom up.')}\ +${chunkedFunctions('evalLevels', Model.levelVars(), ' // Evaluate levels.')}` } // @@ -148,9 +149,35 @@ ${chunkedFunctions('evalLevels', Model.levelVars(), ' // Evaluate levels.')} let headerVarNames = outputAllVars ? expandedVarNames(true) : spec.outputVarNames let outputVarIds = outputAllVars ? expandedVarNames() : spec.outputVars mode = 'io' - return `void setInputs(const char* inputData) {${inputsFromStringImpl()}} + return `\ +void setInputs(const char* inputData) { +${inputsFromStringImpl()} +} -void setInputsFromBuffer(double* inputData) {${inputsFromBufferImpl()}} +void setInputsFromBuffer(double* inputData) { +${inputsFromBufferImpl()} +} + +void replaceLookup(Lookup** lookup, double* points, size_t numPoints) { + if (lookup == NULL) { + return; + } + if (*lookup != NULL) { + __delete_lookup(*lookup); + *lookup = NULL; + } + if (points != NULL) { + *lookup = __new_lookup(numPoints, /*copy=*/true, points); + } +} + +void setLookup(size_t varIndex, size_t* subIndices, double* points, size_t numPoints) { + switch (varIndex) { +${setLookupImpl(Model.varIndexInfo())} + default: + break; + } +} const char* getHeader() { return "${R.map(varName => varName.replace(/"/g, '\\"'), headerVarNames).join('\\t')}"; @@ -178,9 +205,9 @@ ${fullOutputSection(Model.varIndexInfo())} function chunkedFunctions(name, vars, preStep, postStep) { // Emit one function for each chunk let func = (chunk, idx) => { - return ` + return `\ void ${name}${idx}() { - ${section(chunk)} +${section(chunk)} } ` } @@ -208,22 +235,25 @@ void ${name}${idx}() { chunks = [vars] } - if (!preStep) { - preStep = '' + let funcsPart = funcs(chunks) + let callsPart = funcCalls(chunks) + + let out = '' + if (funcsPart.length > 0) { + out += funcsPart + '\n' } - if (!postStep) { - postStep = '' + out += `void ${name}() {\n` + if (preStep) { + out += preStep + '\n' } - - return ` -${funcs(chunks)} - -void ${name}() { -${preStep} -${funcCalls(chunks)} -${postStep} -} - ` + if (callsPart.length > 0) { + out += callsPart + '\n' + } + if (postStep) { + out += postStep + '\n' + } + out += '}\n\n' + return out } // @@ -314,7 +344,10 @@ ${postStep} return section(varNames) } function fullOutputSection(varIndexInfo) { - // Emit output calls for all variables. + // Emit `storeValue` calls for all variables that can be accessed as an output. + // This excludes data and lookup variables; at this time, the data for these + // cannot be output like for other types of variables. + const outputVars = R.filter(info => info.varType !== 'lookup' && info.varType !== 'data') const code = R.map(info => { let varAccess = info.varName if (info.subscriptCount > 0) { @@ -326,13 +359,12 @@ ${postStep} if (info.subscriptCount > 2) { varAccess += '[subIndex2]' } - let c = '' - c += ` case ${info.varIndex}:\n` - c += ` outputVar(${varAccess});\n` - c += ` break;` - return c + return `\ + case ${info.varIndex}: + outputVar(${varAccess}); + break;` }) - const section = R.pipe(code, lines) + const section = R.pipe(outputVars, code, lines) return section(varIndexInfo) } function inputsFromStringImpl() { @@ -341,7 +373,7 @@ ${postStep} let inputVars = '' if (spec.inputVars && spec.inputVars.length > 0) { let inputVarPtrs = R.reduce((a, inputVar) => R.concat(a, ` &${inputVar},\n`), '', spec.inputVars) - inputVars = ` + inputVars = `\ static double* inputVarPtrs[] = {\n${inputVarPtrs} }; char* inputs = (char*)inputData; char* token = strtok(inputs, " "); @@ -354,21 +386,37 @@ ${postStep} *inputVarPtrs[modelVarIndex] = value; } token = strtok(NULL, " "); - } -` + }` } return inputVars } function inputsFromBufferImpl() { - let inputVars = '' + let inputVars = [] if (spec.inputVars && spec.inputVars.length > 0) { - inputVars += '\n' for (let i = 0; i < spec.inputVars.length; i++) { const inputVar = spec.inputVars[i] - inputVars += ` ${inputVar} = inputData[${i}];\n` + inputVars.push(` ${inputVar} = inputData[${i}];`) } } - return inputVars + return inputVars.join('\n') + } + function setLookupImpl(varIndexInfo) { + // Emit `createLookup` calls for all lookups and data variables that can be overridden + // at runtime + const lookupAndDataVars = R.filter(info => info.varType === 'lookup' || info.varType === 'data') + const code = R.map(info => { + let lookupVar = info.varName + for (let i = 0; i < info.subscriptCount; i++) { + lookupVar += `[subIndices[${i}]]` + } + let c = '' + c += ` case ${info.varIndex}:\n` + c += ` replaceLookup(&${lookupVar}, points, numPoints);\n` + c += ` break;` + return c + }) + const section = R.pipe(lookupAndDataVars, code, lines) + return section(varIndexInfo) } return { diff --git a/packages/compile/src/generate/gen-code-c.spec.ts b/packages/compile/src/generate/gen-code-c.spec.ts index 30e97ee6..5d78d5ba 100644 --- a/packages/compile/src/generate/gen-code-c.spec.ts +++ b/packages/compile/src/generate/gen-code-c.spec.ts @@ -57,18 +57,296 @@ function readInlineModelAndGenerateC( } describe('generateC (Vensim -> C)', () => { - it('should work for simple model', () => { + it('should generate code for a simple model', () => { + const extData: ExtData = new Map() + function addData(varId: string) { + extData.set( + varId, + new Map([ + [0, 0], + [1, 2], + [2, 5] + ]) + ) + } + addData('_a_data[_a1]') + addData('_a_data[_a2]') + addData('_b_data[_a1,_b1]') + addData('_b_data[_a1,_b2]') + addData('_b_data[_a2,_b1]') + addData('_b_data[_a2,_b2]') + addData('_c_data') const mdl = ` - x = 1 ~~| + DimA: A1, A2 ~~| + DimB: B1, B2 ~~| + input = 1 ~~| + x = input ~~| y = :NOT: x ~~| + z = ABS(y) ~~| + a data[DimA] ~~| + a[DimA] = a data[DimA] ~~| + b data[DimA, DimB] ~~| + b[DimA, DimB] = b data[DimA, DimB] ~~| + c data ~~| + c = c data ~~| + w = WITH LOOKUP(x, ( [(0,0)-(2,2)], (0,0),(0.1,0.01),(0.5,0.7),(1,1),(1.5,1.2),(2,1.3) )) ~~| + INITIAL TIME = 0 ~~| + FINAL TIME = 2 ~~| + TIME STEP = 1 ~~| + SAVEPER = 1 ~~| ` - const code = readInlineModelAndGenerateC(mdl) - expect(code).toMatch(`\ + const code = readInlineModelAndGenerateC(mdl, { + inputVarNames: ['input'], + outputVarNames: ['x', 'y', 'z', 'a[A1]', 'b[A2,B1]', 'c', 'w'], + extData + }) + expect(code).toEqual(`\ #include "sde.h" // Model variables +Lookup* __lookup1; +Lookup* _a_data[2]; +Lookup* _b_data[2][2]; +Lookup* _c_data; +double _a[2]; +double _b[2][2]; +double _c; +double _final_time; +double _initial_time; +double _input; +double _saveper; +double _time_step; +double _w; double _x; -double _y;`) +double _y; +double _z; + +// Internal variables +const int numOutputs = 7; +#define SDE_USE_OUTPUT_INDICES 0 +#define SDE_MAX_OUTPUT_INDICES 1000 +const int maxOutputIndices = SDE_USE_OUTPUT_INDICES ? SDE_MAX_OUTPUT_INDICES : 0; + +// Array dimensions +const size_t _dima[2] = { 0, 1 }; +const size_t _dimb[2] = { 0, 1 }; + +// Dimension mappings + + +// Lookup data arrays +double __lookup1_data_[12] = { 0.0, 0.0, 0.1, 0.01, 0.5, 0.7, 1.0, 1.0, 1.5, 1.2, 2.0, 1.3 }; +double _a_data_data__0_[6] = { 0.0, 0.0, 1.0, 2.0, 2.0, 5.0 }; +double _a_data_data__1_[6] = { 0.0, 0.0, 1.0, 2.0, 2.0, 5.0 }; +double _b_data_data__0__0_[6] = { 0.0, 0.0, 1.0, 2.0, 2.0, 5.0 }; +double _b_data_data__0__1_[6] = { 0.0, 0.0, 1.0, 2.0, 2.0, 5.0 }; +double _b_data_data__1__0_[6] = { 0.0, 0.0, 1.0, 2.0, 2.0, 5.0 }; +double _b_data_data__1__1_[6] = { 0.0, 0.0, 1.0, 2.0, 2.0, 5.0 }; +double _c_data_data_[6] = { 0.0, 0.0, 1.0, 2.0, 2.0, 5.0 }; + +// Internal state +bool lookups_initialized = false; +bool data_initialized = false; + +void initLookups0() { + __lookup1 = __new_lookup(6, /*copy=*/false, __lookup1_data_); +} + +void initLookups() { + // Initialize lookups. + if (lookups_initialized) { + return; + } + initLookups0(); + lookups_initialized = true; +} + +void initData0() { + _a_data[0] = __new_lookup(3, /*copy=*/false, _a_data_data__0_); + _a_data[1] = __new_lookup(3, /*copy=*/false, _a_data_data__1_); + _b_data[0][0] = __new_lookup(3, /*copy=*/false, _b_data_data__0__0_); + _b_data[0][1] = __new_lookup(3, /*copy=*/false, _b_data_data__0__1_); + _b_data[1][0] = __new_lookup(3, /*copy=*/false, _b_data_data__1__0_); + _b_data[1][1] = __new_lookup(3, /*copy=*/false, _b_data_data__1__1_); + _c_data = __new_lookup(3, /*copy=*/false, _c_data_data_); +} + +void initData() { + // Initialize data. + if (data_initialized) { + return; + } + initData0(); + data_initialized = true; +} + +void initConstants0() { + // FINAL TIME = 2 + _final_time = 2.0; + // INITIAL TIME = 0 + _initial_time = 0.0; + // SAVEPER = 1 + _saveper = 1.0; + // TIME STEP = 1 + _time_step = 1.0; + // input = 1 + _input = 1.0; +} + +void initConstants() { + // Initialize constants. + initConstants0(); + initLookups(); + initData(); +} + +void initLevels() { + // Initialize variables with initialization values, such as levels, and the variables they depend on. + _time = _initial_time; +} + +void evalAux0() { + // a[DimA] = a data[DimA] + for (size_t i = 0; i < 2; i++) { + _a[i] = _LOOKUP(_a_data[i], _time); + } + // b[DimA,DimB] = b data[DimA,DimB] + for (size_t i = 0; i < 2; i++) { + for (size_t j = 0; j < 2; j++) { + _b[i][j] = _LOOKUP(_b_data[i][j], _time); + } + } + // c = c data + _c = _LOOKUP(_c_data, _time); + // x = input + _x = _input; + // w = WITH LOOKUP(x,([(0,0)-(2,2)],(0,0),(0.1,0.01),(0.5,0.7),(1,1),(1.5,1.2),(2,1.3))) + _w = _WITH_LOOKUP(_x, __lookup1); + // y = :NOT: x + _y = !_x; + // z = ABS(y) + _z = _ABS(_y); +} + +void evalAux() { + // Evaluate auxiliaries in order from the bottom up. + evalAux0(); +} + +void evalLevels() { + // Evaluate levels. +} + +void setInputs(const char* inputData) { + static double* inputVarPtrs[] = { + &_input, + }; + char* inputs = (char*)inputData; + char* token = strtok(inputs, " "); + while (token) { + char* p = strchr(token, ':'); + if (p) { + *p = '\\0'; + int modelVarIndex = atoi(token); + double value = atof(p+1); + *inputVarPtrs[modelVarIndex] = value; + } + token = strtok(NULL, " "); + } +} + +void setInputsFromBuffer(double* inputData) { + _input = inputData[0]; +} + +void replaceLookup(Lookup** lookup, double* points, size_t numPoints) { + if (lookup == NULL) { + return; + } + if (*lookup != NULL) { + __delete_lookup(*lookup); + *lookup = NULL; + } + if (points != NULL) { + *lookup = __new_lookup(numPoints, /*copy=*/true, points); + } +} + +void setLookup(size_t varIndex, size_t* subIndices, double* points, size_t numPoints) { + switch (varIndex) { + case 6: + replaceLookup(&_a_data[subIndices[0]], points, numPoints); + break; + case 7: + replaceLookup(&_b_data[subIndices[0]][subIndices[1]], points, numPoints); + break; + case 8: + replaceLookup(&_c_data, points, numPoints); + break; + default: + break; + } +} + +const char* getHeader() { + return "x\\ty\\tz\\ta[A1]\\tb[A2,B1]\\tc\\tw"; +} + +void storeOutputData() { + outputVar(_x); + outputVar(_y); + outputVar(_z); + outputVar(_a[0]); + outputVar(_b[1][0]); + outputVar(_c); + outputVar(_w); +} + +void storeOutput(size_t varIndex, size_t subIndex0, size_t subIndex1, size_t subIndex2) { +#if SDE_USE_OUTPUT_INDICES + switch (varIndex) { + case 1: + outputVar(_final_time); + break; + case 2: + outputVar(_initial_time); + break; + case 3: + outputVar(_saveper); + break; + case 4: + outputVar(_time_step); + break; + case 5: + outputVar(_input); + break; + case 9: + outputVar(_a[subIndex0]); + break; + case 10: + outputVar(_b[subIndex0][subIndex1]); + break; + case 11: + outputVar(_c); + break; + case 12: + outputVar(_x); + break; + case 13: + outputVar(_w); + break; + case 14: + outputVar(_y); + break; + case 15: + outputVar(_z); + break; + default: + break; + } +#endif +} +`) }) it('should work when valid input variable name without subscript is provided in spec file', () => { diff --git a/packages/compile/src/generate/gen-code-js.js b/packages/compile/src/generate/gen-code-js.js index c3f75686..bab2ae7d 100644 --- a/packages/compile/src/generate/gen-code-js.js +++ b/packages/compile/src/generate/gen-code-js.js @@ -263,6 +263,19 @@ ${chunkedFunctions('evalLevels', true, Model.levelVars(), ' // Evaluate levels' return `\ /*export*/ function setInputs(valueAtIndex /*: (index: number) => number*/) {${inputsFromBufferImpl()}} +/*export*/ function setLookup(varSpec /*: VarSpec*/, points /*: Float64Array*/) { + if (!varSpec) { + throw new Error('Got undefined varSpec in setLookup'); + } + const varIndex = varSpec.varIndex; + const subs = varSpec.subscriptIndices; + switch (varIndex) { +${setLookupImpl(Model.varIndexInfo())} + default: + break; + } +} + /*export*/ const outputVarIds = [ ${outputVarIdElems} ]; @@ -276,7 +289,12 @@ ${specOutputSection(outputVarAccesses)} } /*export*/ function storeOutput(varSpec /*: VarSpec*/, storeValue /*: (value: number) => void*/) { - switch (varSpec.varIndex) { + if (!varSpec) { + throw new Error('Got undefined varSpec in storeOutput'); + } + const varIndex = varSpec.varIndex; + const subs = varSpec.subscriptIndices; + switch (varIndex) { ${fullOutputSection(Model.varIndexInfo())} default: break; @@ -440,17 +458,14 @@ ${section(chunk)} return section(varNames) } function fullOutputSection(varIndexInfo) { - // Emit output calls for all variables. + // Emit `storeValue` calls for all variables that can be accessed as an output. + // This excludes data and lookup variables; at this time, the data for these + // cannot be output like for other types of variables. + const outputVars = R.filter(info => info.varType !== 'lookup' && info.varType !== 'data') const code = R.map(info => { let varAccess = info.varName - if (info.subscriptCount > 0) { - varAccess += '[varSpec.subscriptIndices[0]]' - } - if (info.subscriptCount > 1) { - varAccess += '[varSpec.subscriptIndices[1]]' - } - if (info.subscriptCount > 2) { - varAccess += '[varSpec.subscriptIndices[2]]' + for (let i = 0; i < info.subscriptCount; i++) { + varAccess += `[subs[${i}]]` } let c = '' c += ` case ${info.varIndex}:\n` @@ -458,7 +473,7 @@ ${section(chunk)} c += ` break;` return c }) - const section = R.pipe(code, lines) + const section = R.pipe(outputVars, code, lines) return section(varIndexInfo) } function inputsFromBufferImpl() { @@ -472,6 +487,24 @@ ${section(chunk)} } return inputVars } + function setLookupImpl(varIndexInfo) { + // Emit `createLookup` calls for all lookups and data variables that can be overridden + // at runtime + const lookupAndDataVars = R.filter(info => info.varType === 'lookup' || info.varType === 'data') + const code = R.map(info => { + let lookupVar = info.varName + for (let i = 0; i < info.subscriptCount; i++) { + lookupVar += `[subs[${i}]]` + } + let c = '' + c += ` case ${info.varIndex}:\n` + c += ` ${lookupVar} = fns.createLookup(points.length / 2, points);\n` + c += ` break;` + return c + }) + const section = R.pipe(lookupAndDataVars, code, lines) + return section(varIndexInfo) + } return { generate: generate @@ -508,6 +541,7 @@ export default async function () { setTime, setInputs, + setLookup, storeOutputs, storeOutput, diff --git a/packages/compile/src/generate/gen-code-js.spec.ts b/packages/compile/src/generate/gen-code-js.spec.ts index bb1128ee..24a4e7b3 100644 --- a/packages/compile/src/generate/gen-code-js.spec.ts +++ b/packages/compile/src/generate/gen-code-js.spec.ts @@ -74,6 +74,7 @@ interface JsModel { setTime(time: number): void setInputs(inputValue: (index: number) => number): void + setLookup(varSpec: /*VarSpec*/ any, lookup: /*Lookup*/ any): void storeOutputs(storeValue: (value: number) => void): void storeOutput(varSpec: /*VarSpec*/ any, storeValue: (value: number) => void): void @@ -163,6 +164,24 @@ function runJsModel(model: JsModel, inputs: number[], outputs: number[]) { describe('generateJS (Vensim -> JS)', () => { it('should generate code for a simple model', () => { + const extData: ExtData = new Map() + function addData(varId: string) { + extData.set( + varId, + new Map([ + [0, 0], + [1, 2], + [2, 5] + ]) + ) + } + addData('_a_data[_a1]') + addData('_a_data[_a2]') + addData('_b_data[_a1,_b1]') + addData('_b_data[_a1,_b2]') + addData('_b_data[_a2,_b1]') + addData('_b_data[_a2,_b2]') + addData('_c_data') const mdl = ` DimA: A1, A2 ~~| DimB: B1, B2 ~~| @@ -170,20 +189,37 @@ describe('generateJS (Vensim -> JS)', () => { x = input ~~| y = :NOT: x ~~| z = ABS(y) ~~| + a data[DimA] ~~| + a[DimA] = a data[DimA] ~~| + b data[DimA, DimB] ~~| + b[DimA, DimB] = b data[DimA, DimB] ~~| + c data ~~| + c = c data ~~| w = WITH LOOKUP(x, ( [(0,0)-(2,2)], (0,0),(0.1,0.01),(0.5,0.7),(1,1),(1.5,1.2),(2,1.3) )) ~~| - a[DimA] = 0, 1 ~~| - b[DimA, DimB] = 5 ~~| + INITIAL TIME = 0 ~~| + FINAL TIME = 2 ~~| + TIME STEP = 1 ~~| + SAVEPER = 1 ~~| ` const code = readInlineModelAndGenerateJS(mdl, { inputVarNames: ['input'], - outputVarNames: ['a[A1]', 'b[A2,B1]', 'x', 'y', 'z', 'w'] + outputVarNames: ['x', 'y', 'z', 'a[A1]', 'b[A2,B1]', 'c', 'w'], + extData }) expect(code).toEqual(`\ // Model variables let __lookup1; let _a = multiDimArray([2]); +let _a_data = multiDimArray([2]); let _b = multiDimArray([2, 2]); +let _b_data = multiDimArray([2, 2]); +let _c; +let _c_data; +let _final_time; +let _initial_time; let _input; +let _saveper; +let _time_step; let _w; let _x; let _y; @@ -198,7 +234,13 @@ const _dimb = [0, 1]; // Lookup data arrays const __lookup1_data_ = [0.0, 0.0, 0.1, 0.01, 0.5, 0.7, 1.0, 1.0, 1.5, 1.2, 2.0, 1.3]; - +const _a_data_data__0_ = [0.0, 0.0, 1.0, 2.0, 2.0, 5.0]; +const _a_data_data__1_ = [0.0, 0.0, 1.0, 2.0, 2.0, 5.0]; +const _b_data_data__0__0_ = [0.0, 0.0, 1.0, 2.0, 2.0, 5.0]; +const _b_data_data__0__1_ = [0.0, 0.0, 1.0, 2.0, 2.0, 5.0]; +const _b_data_data__1__0_ = [0.0, 0.0, 1.0, 2.0, 2.0, 5.0]; +const _b_data_data__1__1_ = [0.0, 0.0, 1.0, 2.0, 2.0, 5.0]; +const _c_data_data_ = [0.0, 0.0, 1.0, 2.0, 2.0, 5.0]; // Time variable let _time; @@ -314,24 +356,33 @@ function initLookups() { } } +function initData0() { + _a_data[0] = fns.createLookup(3, _a_data_data__0_); + _a_data[1] = fns.createLookup(3, _a_data_data__1_); + _b_data[0][0] = fns.createLookup(3, _b_data_data__0__0_); + _b_data[0][1] = fns.createLookup(3, _b_data_data__0__1_); + _b_data[1][0] = fns.createLookup(3, _b_data_data__1__0_); + _b_data[1][1] = fns.createLookup(3, _b_data_data__1__1_); + _c_data = fns.createLookup(3, _c_data_data_); +} + function initData() { // Initialize data if (!data_initialized) { + initData0(); data_initialized = true; } } function initConstants0() { - // a[DimA] = 0,1 - _a[0] = 0.0; - // a[DimA] = 0,1 - _a[1] = 1.0; - // b[DimA,DimB] = 5 - for (let i = 0; i < 2; i++) { - for (let j = 0; j < 2; j++) { - _b[i][j] = 5.0; - } - } + // FINAL TIME = 2 + _final_time = 2.0; + // INITIAL TIME = 0 + _initial_time = 0.0; + // SAVEPER = 1 + _saveper = 1.0; + // TIME STEP = 1 + _time_step = 1.0; // input = 1 _input = 1.0; } @@ -348,6 +399,18 @@ function initConstants0() { } function evalAux0() { + // a[DimA] = a data[DimA] + for (let i = 0; i < 2; i++) { + _a[i] = fns.LOOKUP(_a_data[i], _time); + } + // b[DimA,DimB] = b data[DimA,DimB] + for (let i = 0; i < 2; i++) { + for (let j = 0; j < 2; j++) { + _b[i][j] = fns.LOOKUP(_b_data[i][j], _time); + } + } + // c = c data + _c = fns.LOOKUP(_c_data, _time); // x = input _x = _input; // w = WITH LOOKUP(x,([(0,0)-(2,2)],(0,0),(0.1,0.01),(0.5,0.7),(1,1),(1.5,1.2),(2,1.3))) @@ -371,54 +434,98 @@ function evalAux0() { _input = valueAtIndex(0); } +/*export*/ function setLookup(varSpec /*: VarSpec*/, points /*: Float64Array*/) { + if (!varSpec) { + throw new Error('Got undefined varSpec in setLookup'); + } + const varIndex = varSpec.varIndex; + const subs = varSpec.subscriptIndices; + switch (varIndex) { + case 6: + _a_data[subs[0]] = fns.createLookup(points.length / 2, points); + break; + case 7: + _b_data[subs[0]][subs[1]] = fns.createLookup(points.length / 2, points); + break; + case 8: + _c_data = fns.createLookup(points.length / 2, points); + break; + default: + break; + } +} + /*export*/ const outputVarIds = [ - '_a[_a1]', - '_b[_a2,_b1]', '_x', '_y', '_z', + '_a[_a1]', + '_b[_a2,_b1]', + '_c', '_w' ]; /*export*/ const outputVarNames = [ - 'a[A1]', - 'b[A2,B1]', 'x', 'y', 'z', + 'a[A1]', + 'b[A2,B1]', + 'c', 'w' ]; /*export*/ function storeOutputs(storeValue /*: (value: number) => void*/) { - storeValue(_a[0]); - storeValue(_b[1][0]); storeValue(_x); storeValue(_y); storeValue(_z); + storeValue(_a[0]); + storeValue(_b[1][0]); + storeValue(_c); storeValue(_w); } /*export*/ function storeOutput(varSpec /*: VarSpec*/, storeValue /*: (value: number) => void*/) { - switch (varSpec.varIndex) { + if (!varSpec) { + throw new Error('Got undefined varSpec in storeOutput'); + } + const varIndex = varSpec.varIndex; + const subs = varSpec.subscriptIndices; + switch (varIndex) { case 1: - storeValue(_a[varSpec.subscriptIndices[0]]); + storeValue(_final_time); break; case 2: - storeValue(_b[varSpec.subscriptIndices[0]][varSpec.subscriptIndices[1]]); + storeValue(_initial_time); break; case 3: - storeValue(_input); + storeValue(_saveper); break; case 4: - storeValue(_x); + storeValue(_time_step); break; case 5: + storeValue(_input); + break; + case 9: + storeValue(_a[subs[0]]); + break; + case 10: + storeValue(_b[subs[0]][subs[1]]); + break; + case 11: + storeValue(_c); + break; + case 12: + storeValue(_x); + break; + case 13: storeValue(_w); break; - case 6: + case 14: storeValue(_y); break; - case 7: + case 15: storeValue(_z); break; default: @@ -442,6 +549,7 @@ export default async function () { setTime, setInputs, + setLookup, storeOutputs, storeOutput, diff --git a/packages/compile/src/model/model.js b/packages/compile/src/model/model.js index e13dc99d..f6aeb9b0 100644 --- a/packages/compile/src/model/model.js +++ b/packages/compile/src/model/model.js @@ -1062,13 +1062,12 @@ function allListedVars() { if (timeVar) { vars.push(timeVar) } - vars.push(...initVars()) vars.push(...auxVars()) - // TODO: Also levelVars not covered by initVars? + vars.push(...levelVars()) - // Filter out data/lookup variables and variables that are generated/used internally + // Filter out variables that are generated/used internally const isInternal = v => { - return v.refId.startsWith('__level') || v.refId.startsWith('__aux') + return v.includeInOutput === false } return R.filter(v => !isInternal(v), vars) @@ -1083,6 +1082,7 @@ function filteredListedVars() { function varIndexInfoMap() { // Return a map containing information for each listed variable: // varName + // varType // varIndex // subscriptCount @@ -1090,21 +1090,17 @@ function varIndexInfoMap() { // generated model const sortedVars = filteredListedVars() - // Get the set of unique variable names, and assign a 1-based index - // to each; this matches the index number used in `storeOutput()` - // in the generated C code + // Get the set of unique variable names, and assign a 1-based index to each. + // This matches the index number used in `storeOutput` and `setLookup` in the + // generated C/JS code const infoMap = new Map() let varIndex = 1 for (const v of sortedVars) { - if (v.varType === 'data' || v.varType === 'lookup') { - // Omit the index for data and lookup variables; at this time, the data for these - // cannot be output like for other types of variables - continue - } const varName = v.varName if (!infoMap.get(varName)) { infoMap.set(varName, { varName, + varType: v.varType, varIndex, subscriptCount: v.families ? v.families.length : 0 }) @@ -1119,6 +1115,7 @@ function varIndexInfo() { // Return an array, sorted by `varName`, containing information for each // listed variable: // varName + // varType // varIndex // subscriptCount return Array.from(varIndexInfoMap().values()) @@ -1136,7 +1133,8 @@ function jsonList() { const sortedVars = filteredListedVars() // Assign a 1-based index for each variable that has data that can be accessed. - // This matches the index number used in `storeOutput()` in the generated C code. + // This matches the index number used in `storeOutput` and `setLookup` in the + // generated C/JS code const infoMap = varIndexInfoMap() for (const v of sortedVars) { const varInfo = infoMap.get(v.varName) diff --git a/packages/compile/src/model/model.spec.ts b/packages/compile/src/model/model.spec.ts new file mode 100644 index 00000000..48d80f6d --- /dev/null +++ b/packages/compile/src/model/model.spec.ts @@ -0,0 +1,326 @@ +import { describe, expect, it } from 'vitest' + +import { resetHelperState } from '../_shared/helpers' +import { resetSubscriptsAndDimensions } from '../_shared/subscript' + +import Model from './model' + +import type { ParsedModel } from '../_tests/test-support' +import { parseInlineVensimModel, parseVensimModel, sampleModelDir } from '../_tests/test-support' + +/** + * This is a shorthand for the following steps to read equations: + * - parseVensimModel + * - readSubscriptRanges + * - resolveSubscriptRanges + * - readVariables + * - analyze (this includes readEquations) + */ +function readSubscriptsAndEquationsFromSource( + source: { + modelText?: string + modelName?: string + modelDir?: string + }, + opts?: { + specialSeparationDims?: { [key: string]: string } + } +): any { + // XXX: These steps are needed due to subs/dims and variables being in module-level storage + resetHelperState() + resetSubscriptsAndDimensions() + Model.resetModelState() + + let parsedModel: ParsedModel + if (source.modelText) { + parsedModel = parseInlineVensimModel(source.modelText) + } else { + parsedModel = parseVensimModel(source.modelName) + } + + const spec = { + specialSeparationDims: opts?.specialSeparationDims + } + + let modelDir = source.modelDir + if (modelDir === undefined) { + if (source.modelName) { + modelDir = sampleModelDir(source.modelName) + } + } + + Model.read(parsedModel, spec, /*extData=*/ undefined, /*directData=*/ undefined, modelDir, { + reduceVariables: false, + stopAfterAnalyze: true + }) + + return JSON.parse(Model.jsonList()) +} + +function readInlineModel( + modelText: string, + modelDir?: string, + opts?: { + specialSeparationDims?: { [key: string]: string } + } +): any { + return readSubscriptsAndEquationsFromSource({ modelText, modelDir }, opts) +} + +describe('Model', () => { + describe('jsonList', () => { + it('should expose accessible variables', () => { + const json = readInlineModel(` + DimA: A1, A2 ~~| + DimB: B1, B2 ~~| + input = 1 ~~| + x = input ~~| + y = :NOT: x ~~| + z = ABS(y) ~~| + a data[DimA] ~~| + a[DimA] = a data[DimA] ~~| + b data[DimA, DimB] ~~| + b[DimA, DimB] = b data[DimA, DimB] ~~| + c data ~~| + c = c data ~~| + d[DimA] = 10, 11 ~~| + level init = 5 ~~| + level = INTEG(x, level init) ~~| + w = WITH LOOKUP(x, ( [(0,0)-(2,2)], (0,0),(0.1,0.01),(0.5,0.7),(1,1),(1.5,1.2),(2,1.3) )) ~~| + INITIAL TIME = 0 ~~| + FINAL TIME = 2 ~~| + TIME STEP = 1 ~~| + SAVEPER = 1 ~~| + `) + expect(json).toEqual({ + dimensions: [ + { + modelName: 'DimA', + modelValue: ['A1', 'A2'], + modelMappings: [], + name: '_dima', + value: ['_a1', '_a2'], + size: 2, + family: '_dima', + mappings: {} + }, + { + modelName: 'DimB', + modelValue: ['B1', 'B2'], + modelMappings: [], + name: '_dimb', + value: ['_b1', '_b2'], + size: 2, + family: '_dimb', + mappings: {} + } + ], + variables: [ + { + refId: '_final_time', + varName: '_final_time', + hasInitValue: false, + varType: 'const', + modelLHS: 'FINAL TIME', + modelFormula: '2', + varIndex: 1 + }, + { + refId: '_initial_time', + varName: '_initial_time', + hasInitValue: false, + varType: 'const', + modelLHS: 'INITIAL TIME', + modelFormula: '0', + varIndex: 2 + }, + { + refId: '_saveper', + varName: '_saveper', + hasInitValue: false, + varType: 'const', + modelLHS: 'SAVEPER', + modelFormula: '1', + varIndex: 3 + }, + { + refId: '_time_step', + varName: '_time_step', + hasInitValue: false, + varType: 'const', + modelLHS: 'TIME STEP', + modelFormula: '1', + varIndex: 4 + }, + { + refId: '_d[_a1]', + varName: '_d', + subscripts: ['_a1'], + families: ['_dima'], + hasInitValue: false, + varType: 'const', + separationDims: ['_dima'], + modelLHS: 'd[DimA]', + modelFormula: '10,11', + varIndex: 5 + }, + { + refId: '_d[_a2]', + varName: '_d', + subscripts: ['_a2'], + families: ['_dima'], + hasInitValue: false, + varType: 'const', + separationDims: ['_dima'], + modelLHS: 'd[DimA]', + modelFormula: '10,11', + varIndex: 5 + }, + { + refId: '_input', + varName: '_input', + hasInitValue: false, + varType: 'const', + modelLHS: 'input', + modelFormula: '1', + varIndex: 6 + }, + { + refId: '_level_init', + varName: '_level_init', + hasInitValue: false, + varType: 'const', + modelLHS: 'level init', + modelFormula: '5', + varIndex: 7 + }, + { + refId: '_a_data', + varName: '_a_data', + subscripts: ['_dima'], + families: ['_dima'], + hasInitValue: false, + varType: 'data', + modelLHS: 'a data[DimA]', + modelFormula: '', + varIndex: 8 + }, + { + refId: '_b_data', + varName: '_b_data', + subscripts: ['_dima', '_dimb'], + families: ['_dima', '_dimb'], + hasInitValue: false, + varType: 'data', + modelLHS: 'b data[DimA,DimB]', + modelFormula: '', + varIndex: 9 + }, + { + refId: '_c_data', + varName: '_c_data', + hasInitValue: false, + varType: 'data', + modelLHS: 'c data', + modelFormula: '', + varIndex: 10 + }, + { + refId: '_time', + varName: '_time', + hasInitValue: false, + varType: 'const', + modelLHS: 'Time', + modelFormula: '', + varIndex: 11 + }, + { + refId: '_a', + varName: '_a', + subscripts: ['_dima'], + families: ['_dima'], + references: ['_a_data'], + hasInitValue: false, + varType: 'aux', + modelLHS: 'a[DimA]', + modelFormula: 'a data[DimA]', + varIndex: 12 + }, + { + refId: '_b', + varName: '_b', + subscripts: ['_dima', '_dimb'], + families: ['_dima', '_dimb'], + references: ['_b_data'], + hasInitValue: false, + varType: 'aux', + modelLHS: 'b[DimA,DimB]', + modelFormula: 'b data[DimA,DimB]', + varIndex: 13 + }, + { + refId: '_c', + varName: '_c', + references: ['_c_data'], + hasInitValue: false, + varType: 'aux', + modelLHS: 'c', + modelFormula: 'c data', + varIndex: 14 + }, + { + refId: '_x', + varName: '_x', + references: ['_input'], + hasInitValue: false, + varType: 'aux', + modelLHS: 'x', + modelFormula: 'input', + varIndex: 15 + }, + { + refId: '_w', + varName: '_w', + references: ['_x'], + hasInitValue: false, + varType: 'aux', + modelLHS: 'w', + modelFormula: 'WITH LOOKUP(x,([(0,0)-(2,2)],(0,0),(0.1,0.01),(0.5,0.7),(1,1),(1.5,1.2),(2,1.3)))', + varIndex: 16 + }, + { + refId: '_y', + varName: '_y', + references: ['_x'], + hasInitValue: false, + varType: 'aux', + modelLHS: 'y', + modelFormula: ':NOT: x', + varIndex: 17 + }, + { + refId: '_z', + varName: '_z', + references: ['_y'], + hasInitValue: false, + varType: 'aux', + modelLHS: 'z', + modelFormula: 'ABS(y)', + varIndex: 18 + }, + { + refId: '_level', + varName: '_level', + references: ['_x'], + hasInitValue: true, + initReferences: ['_level_init'], + varType: 'level', + modelLHS: 'level', + modelFormula: 'INTEG(x,level init)', + varIndex: 19 + } + ] + }) + }) + }) +}) diff --git a/packages/plugin-wasm/docs/interfaces/WasmPluginOptions.md b/packages/plugin-wasm/docs/interfaces/WasmPluginOptions.md index b1edc770..893b8baa 100644 --- a/packages/plugin-wasm/docs/interfaces/WasmPluginOptions.md +++ b/packages/plugin-wasm/docs/interfaces/WasmPluginOptions.md @@ -31,7 +31,7 @@ with) Emscripten versions 2.0.34 and 3.1.46, among others. -s EXPORT_ES6=1 -s USE_ES6_IMPORT_META=0 -s ENVIRONMENT='web,webview,worker' - -s EXPORTED_FUNCTIONS=['_malloc','_free','_getMaxOutputIndices','_getInitialTime','_getFinalTime','_getSaveper','_runModelWithBuffers'] + -s EXPORTED_FUNCTIONS=['_malloc','_free','_getMaxOutputIndices','_getInitialTime','_getFinalTime','_getSaveper','_setLookup','_runModelWithBuffers'] -s EXPORTED_RUNTIME_METHODS=['cwrap'] ``` diff --git a/packages/plugin-wasm/src/options.ts b/packages/plugin-wasm/src/options.ts index 4b97f483..72fb80e1 100644 --- a/packages/plugin-wasm/src/options.ts +++ b/packages/plugin-wasm/src/options.ts @@ -22,7 +22,7 @@ export interface WasmPluginOptions { * -s EXPORT_ES6=1 * -s USE_ES6_IMPORT_META=0 * -s ENVIRONMENT='web,webview,worker' - * -s EXPORTED_FUNCTIONS=['_malloc','_free','_getMaxOutputIndices','_getInitialTime','_getFinalTime','_getSaveper','_runModelWithBuffers'] + * -s EXPORTED_FUNCTIONS=['_malloc','_free','_getMaxOutputIndices','_getInitialTime','_getFinalTime','_getSaveper','_setLookup','_runModelWithBuffers'] * -s EXPORTED_RUNTIME_METHODS=['cwrap'] * ``` */ diff --git a/packages/plugin-wasm/src/plugin.ts b/packages/plugin-wasm/src/plugin.ts index 6eac342e..d752b890 100644 --- a/packages/plugin-wasm/src/plugin.ts +++ b/packages/plugin-wasm/src/plugin.ts @@ -154,7 +154,7 @@ async function buildWasm( // and Node.js contexts (tested in Emscripten 2.0.34 and 3.1.46). addFlag(`ENVIRONMENT='web,webview,worker'`) addFlag( - `EXPORTED_FUNCTIONS=['_malloc','_free','_getMaxOutputIndices','_getInitialTime','_getFinalTime','_getSaveper','_runModelWithBuffers']` + `EXPORTED_FUNCTIONS=['_malloc','_free','_getMaxOutputIndices','_getInitialTime','_getFinalTime','_getSaveper','_setLookup','_runModelWithBuffers']` ) addFlag(`EXPORTED_RUNTIME_METHODS=['cwrap']`) } diff --git a/packages/runtime-async/docs/functions/spawnAsyncModelRunner.md b/packages/runtime-async/docs/functions/spawnAsyncModelRunner.md index 4eac3ad4..b3029082 100644 --- a/packages/runtime-async/docs/functions/spawnAsyncModelRunner.md +++ b/packages/runtime-async/docs/functions/spawnAsyncModelRunner.md @@ -4,10 +4,12 @@ **spawnAsyncModelRunner**(`workerSpec`): `Promise`<[`ModelRunner`](../../../runtime/docs/interfaces/ModelRunner.md)\> -Initialize a [`ModelRunner`](../../../runtime/docs/interfaces/ModelRunner.md) that runs the model asynchronously in a worker thread. +Initialize a [`ModelRunner`](../../../runtime/docs/interfaces/ModelRunner.md) that runs the model asynchronously in a worker +(a Web Worker when running in a browser environment, or a worker thread +when running in a Node.js environment). -In your app project, define a JavaScript file, called `worker.js` for example, that -initializes the generated model in the context of the Web Worker: +In your app project, define a JavaScript file, called `worker.js` for example, +that initializes the generated model in the context of a worker thread: ```js import { exposeModelWorker } from '@sdeverywhere/runtime-async/worker' @@ -17,7 +19,7 @@ exposeModelWorker(loadGeneratedModel) ``` Then, in your web app, call the `spawnAsyncModelRunner` function, which -will spawn the Web Worker and initialize the [`ModelRunner`](../../../runtime/docs/interfaces/ModelRunner.md) that communicates +will spawn the worker thread and initialize the [`ModelRunner`](../../../runtime/docs/interfaces/ModelRunner.md) that communicates with the worker: ```js diff --git a/packages/runtime-async/src/runner.ts b/packages/runtime-async/src/runner.ts index 7b24a85e..cfc247e8 100644 --- a/packages/runtime-async/src/runner.ts +++ b/packages/runtime-async/src/runner.ts @@ -6,10 +6,12 @@ import type { ModelRunner } from '@sdeverywhere/runtime' import { BufferedRunModelParams, Outputs } from '@sdeverywhere/runtime' /** - * Initialize a `ModelRunner` that runs the model asynchronously in a worker thread. + * Initialize a `ModelRunner` that runs the model asynchronously in a worker + * (a Web Worker when running in a browser environment, or a worker thread + * when running in a Node.js environment). * - * In your app project, define a JavaScript file, called `worker.js` for example, that - * initializes the generated model in the context of the Web Worker: + * In your app project, define a JavaScript file, called `worker.js` for example, + * that initializes the generated model in the context of a worker thread: * * ```js * import { exposeModelWorker } from '@sdeverywhere/runtime-async/worker' @@ -19,7 +21,7 @@ import { BufferedRunModelParams, Outputs } from '@sdeverywhere/runtime' * ``` * * Then, in your web app, call the `spawnAsyncModelRunner` function, which - * will spawn the Web Worker and initialize the `ModelRunner` that communicates + * will spawn the worker thread and initialize the `ModelRunner` that communicates * with the worker: * * ```js @@ -67,7 +69,7 @@ async function spawnAsyncModelRunnerWithWorker(worker: Worker): Promise { + runModel: async (inputs, outputs, options) => { if (terminated) { throw new Error('Async model runner has already been terminated') } else if (running) { @@ -77,7 +79,7 @@ async function spawnAsyncModelRunnerWithWorker(worker: Worker): Promise { - switch (varSpec.varIndex) { - case 1: return '_output_1' - case 2: return '_output_2' - case 3: return '_x' - default: throw new Error('No var id found for index') - } - }, - onEvalAux: (vars /*, lookups*/) => { + listing: new ModelListing(\`${json}\`), + onEvalAux: (vars, lookups) => { const time = vars.get('_time') - // if (lookups.size > 0) { - // const lookup1 = lookups.get('_output_1_data') - // const lookup2 = lookups.get('_output_2_data') - // expect(lookup1).toBeDefined() - // expect(lookup2).toBeDefined() - // vars.set('_output_1', lookup1.getValueForX(time, 'interpolate')) - // vars.set('_output_2', lookup2.getValueForX(time, 'interpolate')) - // } else { - vars.set('_output_1', time - startTime + 1) - vars.set('_output_2', time - startTime + 4) - vars.set('_x', time - startTime + 7) - // } + if (lookups.size > 0) { + const lookup1 = lookups.get('_output_1_data') + const lookup2 = lookups.get('_output_2_data') + // expect(lookup1).toBeDefined() + // expect(lookup2).toBeDefined() + vars.set('_output_1', lookup1.getValueForX(time, 'interpolate')) + vars.set('_output_2', lookup2.getValueForX(time, 'interpolate')) + } else { + vars.set('_output_1', time - startTime + 1) + vars.set('_output_2', time - startTime + 4) + vars.set('_x', time - startTime + 7) + } } }) } @@ -59,7 +86,7 @@ exposeModelWorker(createMockJsModel) const workerWithMockWasmModule = `\ const path = require('path') -const { MockWasmModule } = require('@sdeverywhere/runtime') +const { MockWasmModule, ModelListing } = require('@sdeverywhere/runtime') const { exposeModelWorker } = require('@sdeverywhere/runtime-async') const startTime = 2000 @@ -70,26 +97,27 @@ async function createMockWasmModule() { initialTime: startTime, finalTime: endTime, outputVarIds: ['_output_1', '_output_2'], - onRunModel: (inputs, outputs, /*lookups,*/ outputIndices) => { - // if (lookups.size > 0) { - // // Pretend that outputs are derived from lookup data - // const lookup1 = lookups.get('_output_1_data') - // const lookup2 = lookups.get('_output_2_data') - // expect(lookup1).toBeDefined() - // expect(lookup2).toBeDefined() - // for (let i = 0; i < 3; i++) { - // outputs[i] = lookup1.getValueForX(2000 + i, 'interpolate') - // outputs[i + 3] = lookup2.getValueForX(2000 + i, 'interpolate') - // } - // } else { - if (outputIndices === undefined) { - // Store 3 values for the _output_1, and 3 for _output_2 - outputs.set([1, 2, 3, 4, 5, 6]) + listing: new ModelListing(\`${json}\`), + onRunModel: (inputs, outputs, lookups, outputIndices) => { + if (lookups.size > 0) { + // Pretend that outputs are derived from lookup data + const lookup1 = lookups.get('_output_1_data') + const lookup2 = lookups.get('_output_2_data') + // expect(lookup1).toBeDefined() + // expect(lookup2).toBeDefined() + for (let i = 0; i < 3; i++) { + outputs[i] = lookup1.getValueForX(2000 + i, 'interpolate') + outputs[i + 3] = lookup2.getValueForX(2000 + i, 'interpolate') + } } else { - // Store 3 values for each of the three variables - outputs.set([7, 8, 9, 4, 5, 6, 1, 2, 3]) + if (outputIndices === undefined) { + // Store 3 values for the _output_1, and 3 for _output_2 + outputs.set([1, 2, 3, 4, 5, 6]) + } else { + // Store 3 values for each of the three variables + outputs.set([7, 8, 9, 4, 5, 6, 1, 2, 3]) + } } - // } } }) } @@ -144,31 +172,42 @@ describe.each([ expect(outOutputs.getSeriesForVar('_output_2')!.points).toEqual([p(2000, 4), p(2001, 5), p(2002, 6)]) }) - it('should run the model in a worker (when output var specs are included)', async () => { - const json = ` -{ - "dimensions": [ - ], - "variables": [ - { - "refId": "_output_1", - "varName": "_output_1", - "varIndex": 1 - }, - { - "refId": "_output_2", - "varName": "_output_2", - "varIndex": 2 - }, - { - "refId": "_x", - "varName": "_x", - "varIndex": 3 - } - ] -} -` + it('should run the model (with lookup overrides)', async () => { + const listing = new ModelListing(json) + const inputs = [createInputValue('_input_1', 7), createInputValue('_input_2', 8), createInputValue('_input_3', 9)] + let outputs = runner.createOutputs() + + // Run once without lookup overrides + outputs = await runner.runModel(inputs, outputs) + + // Verify that outputs contain the original values + expect(outputs.getSeriesForVar('_output_1')!.points).toEqual([p(2000, 1), p(2001, 2), p(2002, 3)]) + expect(outputs.getSeriesForVar('_output_2')!.points).toEqual([p(2000, 4), p(2001, 5), p(2002, 6)]) + + // Run again, this time with lookup overrides + const lookup1Points = [p(2000, 101), p(2001, 102), p(2002, 103)] + const lookup2Points = [p(2000, 104), p(2001, 105), p(2002, 106)] + outputs = await runner.runModel(inputs, outputs, { + lookups: [ + createLookupDef(listing.varSpecs.get('_output_1_data')!, lookup1Points), + createLookupDef(listing.varSpecs.get('_output_2_data')!, lookup2Points) + ] + }) + + // Verify that outputs contain the values from the overridden lookups + expect(outputs.getSeriesForVar('_output_1')!.points).toEqual(lookup1Points) + expect(outputs.getSeriesForVar('_output_2')!.points).toEqual(lookup2Points) + + // Run again without lookup overrides + outputs = await runner.runModel(inputs, outputs) + + // Verify that the lookup overrides are still in effect from the previous run + expect(outputs.getSeriesForVar('_output_1')!.points).toEqual(lookup1Points) + expect(outputs.getSeriesForVar('_output_2')!.points).toEqual(lookup2Points) + }) + + it('should run the model in a worker (when output var specs are included)', async () => { const listing = new ModelListing(json) const inputs = [7, 8, 9] const normalOutputs = runner.createOutputs() diff --git a/packages/runtime/README.md b/packages/runtime/README.md index 588ab7b4..ca59d633 100644 --- a/packages/runtime/README.md +++ b/packages/runtime/README.md @@ -153,7 +153,7 @@ $ emcc \ build/.c build/macros.c build/model.c build/vensim.c \ -Ibuild -o ./output/.js -Wall -Os \ -s STRICT=1 -s MALLOC=emmalloc -s FILESYSTEM=0 -s MODULARIZE=1 \ --s EXPORTED_FUNCTIONS="['_malloc','_free','_getInitialTime','_getFinalTime','_getSaveper','_runModelWithBuffers']" \ +-s EXPORTED_FUNCTIONS="['_malloc','_free','_getInitialTime','_getFinalTime','_getSaveper','_setLookup','_runModelWithBuffers']" \ -s EXPORTED_RUNTIME_METHODS="['cwrap']" ``` @@ -164,6 +164,7 @@ Note that the generated module must export the following functions at minimum: - `_getInitialTime` - `_getFinalTime` - `_getSaveper` +- `_setLookup` - `_runModelWithBuffers` - `cwrap` diff --git a/packages/runtime/docs/functions/createLookupDef.md b/packages/runtime/docs/functions/createLookupDef.md new file mode 100644 index 00000000..c87358bf --- /dev/null +++ b/packages/runtime/docs/functions/createLookupDef.md @@ -0,0 +1,18 @@ +[@sdeverywhere/runtime](../index.md) / createLookupDef + +# Function: createLookupDef + +**createLookupDef**(`varSpec`, `points`): [`LookupDef`](../interfaces/LookupDef.md) + +Create a `LookupDef` instance from the given array of `Point` objects. + +#### Parameters + +| Name | Type | Description | +| :------ | :------ | :------ | +| `varSpec` | [`VarSpec`](../interfaces/VarSpec.md) | The spec for the lookup or data variable to be modified. | +| `points` | [`Point`](../interfaces/Point.md)[] | The lookup data as an array of `Point` objects. | + +#### Returns + +[`LookupDef`](../interfaces/LookupDef.md) diff --git a/packages/runtime/docs/interfaces/JsModel.md b/packages/runtime/docs/interfaces/JsModel.md index 6a72b1e4..cd160119 100644 --- a/packages/runtime/docs/interfaces/JsModel.md +++ b/packages/runtime/docs/interfaces/JsModel.md @@ -108,6 +108,23 @@ ___ ___ +### setLookup + +**setLookup**(`varSpec`, `points`): `void` + +#### Parameters + +| Name | Type | +| :------ | :------ | +| `varSpec` | [`VarSpec`](VarSpec.md) | +| `points` | `Float64Array` | + +#### Returns + +`void` + +___ + ### storeOutputs **storeOutputs**(`storeValue`): `void` @@ -124,6 +141,23 @@ ___ ___ +### storeOutput + +**storeOutput**(`varSpec`, `storeValue`): `void` + +#### Parameters + +| Name | Type | +| :------ | :------ | +| `varSpec` | [`VarSpec`](VarSpec.md) | +| `storeValue` | (`value`: `number`) => `void` | + +#### Returns + +`void` + +___ + ### initConstants **initConstants**(): `void` diff --git a/packages/runtime/docs/interfaces/LookupDef.md b/packages/runtime/docs/interfaces/LookupDef.md new file mode 100644 index 00000000..f05ae059 --- /dev/null +++ b/packages/runtime/docs/interfaces/LookupDef.md @@ -0,0 +1,21 @@ +[@sdeverywhere/runtime](../index.md) / LookupDef + +# Interface: LookupDef + +Specifies the data that will be used to set or override a lookup definition. + +## Properties + +### varSpec + + **varSpec**: [`VarSpec`](VarSpec.md) + +The spec for the lookup or data variable to be modified. + +___ + +### points + + **points**: `Float64Array` + +The lookup data as a flat array of (x,y) pairs. diff --git a/packages/runtime/docs/interfaces/ModelRunner.md b/packages/runtime/docs/interfaces/ModelRunner.md index 8baa379f..075f6703 100644 --- a/packages/runtime/docs/interfaces/ModelRunner.md +++ b/packages/runtime/docs/interfaces/ModelRunner.md @@ -24,7 +24,7 @@ ___ ### runModel -**runModel**(`inputs`, `outputs`): `Promise`<[`Outputs`](../classes/Outputs.md)\> +**runModel**(`inputs`, `outputs`, `options?`): `Promise`<[`Outputs`](../classes/Outputs.md)\> Run the model. @@ -34,6 +34,7 @@ Run the model. | :------ | :------ | :------ | | `inputs` | (`number` \| [`InputValue`](InputValue.md))[] | The model input values (must be in the same order as in the spec file). | | `outputs` | [`Outputs`](../classes/Outputs.md) | The structure into which the model outputs will be stored. | +| `options?` | [`RunModelOptions`](RunModelOptions.md) | Additional options that influence the model run. | #### Returns diff --git a/packages/runtime/docs/interfaces/RunModelOptions.md b/packages/runtime/docs/interfaces/RunModelOptions.md new file mode 100644 index 00000000..dd5369ae --- /dev/null +++ b/packages/runtime/docs/interfaces/RunModelOptions.md @@ -0,0 +1,22 @@ +[@sdeverywhere/runtime](../index.md) / RunModelOptions + +# Interface: RunModelOptions + +Additional options that can be passed to a `runModel` call to influence the model run. + +## Properties + +### lookups + + `Optional` **lookups**: [`LookupDef`](LookupDef.md)[] + +If defined, override the data for the specified lookups and/or data variables. + +If data was already defined in the generated model, the data provided in a +`LookupDef` here will override the default data in the generated model. + +Note that unlike the `inputs` parameter for `runModel` (which must be provided +with each call), the data overrides provided here persist after the `runModel` +call. If you pass `lookups` in your Nth `runModel` call, that lookup data will +still be in effect for the (N+1)th call. In other words, if your lookup data +is not changing, you do not need to supply it with every `runModel` call. diff --git a/packages/runtime/docs/interfaces/VarSpec.md b/packages/runtime/docs/interfaces/VarSpec.md new file mode 100644 index 00000000..bc0b406e --- /dev/null +++ b/packages/runtime/docs/interfaces/VarSpec.md @@ -0,0 +1,22 @@ +[@sdeverywhere/runtime](../index.md) / VarSpec + +# Interface: VarSpec + +The variable index metadata that is used to identify a specific instance of a +variable in a generated model. + +## Properties + +### varIndex + + **varIndex**: `number` + +The variable index as used in the generated C/JS code. + +___ + +### subscriptIndices + + `Optional` **subscriptIndices**: `number`[] + +The subscript index values as used in the generated C/JS code. diff --git a/packages/runtime/package.json b/packages/runtime/package.json index 4c5d74b6..08eb3191 100644 --- a/packages/runtime/package.json +++ b/packages/runtime/package.json @@ -24,7 +24,7 @@ "test": "vitest run", "test:watch": "vitest", "test:ci": "vitest run", - "type-check": "tsc --noEmit -p tsconfig-build.json", + "type-check": "tsc --noEmit -p tsconfig-test.json", "build": "tsup", "docs": "../../scripts/gen-docs.js", "ci:build": "run-s clean lint prettier:check test:ci type-check build docs" diff --git a/packages/runtime/src/_shared/index.ts b/packages/runtime/src/_shared/index.ts index 289aaa63..fa767fb9 100644 --- a/packages/runtime/src/_shared/index.ts +++ b/packages/runtime/src/_shared/index.ts @@ -4,3 +4,4 @@ export * from './types' export * from './inputs' export * from './outputs' export * from './var-indices' +export * from './lookup-def' diff --git a/packages/runtime/src/_shared/lookup-def.ts b/packages/runtime/src/_shared/lookup-def.ts new file mode 100644 index 00000000..f1bf193c --- /dev/null +++ b/packages/runtime/src/_shared/lookup-def.ts @@ -0,0 +1,37 @@ +// Copyright (c) 2020-2022 Climate Interactive / New Venture Fund + +import type { Point } from './types' +import type { VarSpec } from './var-indices' + +/** + * Specifies the data that will be used to set or override a lookup definition. + */ +export interface LookupDef { + /** The spec for the lookup or data variable to be modified. */ + varSpec: VarSpec + /** The lookup data as a flat array of (x,y) pairs. */ + points: Float64Array +} + +/** + * Create a `LookupDef` instance from the given array of `Point` objects. + * + * @param varSpec The spec for the lookup or data variable to be modified. + * @param points The lookup data as an array of `Point` objects. + */ +export function createLookupDef(varSpec: VarSpec, points: Point[]): LookupDef { + if (varSpec === undefined) { + throw new Error('Got undefined varSpec in createLookupDef') + } + + const flatPoints = new Float64Array(points.length * 2) + let i = 0 + for (const p of points) { + flatPoints[i++] = p.x + flatPoints[i++] = p.y + } + return { + varSpec, + points: flatPoints + } +} diff --git a/packages/runtime/src/_shared/outputs.spec.ts b/packages/runtime/src/_shared/outputs.spec.ts index 5bae035c..130d6e84 100644 --- a/packages/runtime/src/_shared/outputs.spec.ts +++ b/packages/runtime/src/_shared/outputs.spec.ts @@ -2,8 +2,8 @@ import { describe, expect, it } from 'vitest' -import type { Point } from './outputs' import { Outputs, Series } from './outputs' +import type { Point } from './types' function p(x: number, y: number): Point { return { x, y } diff --git a/packages/runtime/src/_shared/outputs.ts b/packages/runtime/src/_shared/outputs.ts index 513d0ed1..352bde55 100644 --- a/packages/runtime/src/_shared/outputs.ts +++ b/packages/runtime/src/_shared/outputs.ts @@ -3,20 +3,12 @@ import type { Result } from 'neverthrow' import { ok, err } from 'neverthrow' -import type { OutputVarId } from './types' +import type { OutputVarId, Point } from './types' import type { VarSpec } from './var-indices' /** Indicates the type of error encountered when parsing an outputs buffer. */ export type ParseError = 'invalid-point-count' -/** A data point. */ -export interface Point { - /** The x value (typically a time value). */ - x: number - /** The y value. */ - y: number -} - /** * A time series of data points for an output variable. */ diff --git a/packages/runtime/src/_shared/types.ts b/packages/runtime/src/_shared/types.ts index eac1a994..a09529da 100644 --- a/packages/runtime/src/_shared/types.ts +++ b/packages/runtime/src/_shared/types.ts @@ -8,3 +8,11 @@ export type InputVarId = string /** An output variable identifier string, as used in SDEverywhere. */ export type OutputVarId = string + +/** A data point. */ +export interface Point { + /** The x value (typically a time value). */ + x: number + /** The y value. */ + y: number +} diff --git a/packages/runtime/src/_shared/var-indices.ts b/packages/runtime/src/_shared/var-indices.ts index 2a83b90a..a8227d0d 100644 --- a/packages/runtime/src/_shared/var-indices.ts +++ b/packages/runtime/src/_shared/var-indices.ts @@ -12,13 +12,13 @@ export const indicesPerVariable = 4 /** - * The variable index values for use with the optional input/output indices buffer. - * @hidden This is not yet part of the public API; it is exposed here for use in testing tools. + * The variable index metadata that is used to identify a specific instance of a + * variable in a generated model. */ export interface VarSpec { - /** The variable index as used in the generated C code. */ + /** The variable index as used in the generated C/JS code. */ varIndex: number - /** The subscript index values as used in the generated C code. */ + /** The subscript index values as used in the generated C/JS code. */ subscriptIndices?: number[] } diff --git a/packages/runtime/src/js-model/_mocks/mock-js-model.ts b/packages/runtime/src/js-model/_mocks/mock-js-model.ts index 598e8b99..21c23272 100644 --- a/packages/runtime/src/js-model/_mocks/mock-js-model.ts +++ b/packages/runtime/src/js-model/_mocks/mock-js-model.ts @@ -4,18 +4,13 @@ import type { OutputVarId, VarId, VarSpec } from '../../_shared' import type { ModelListing } from '../../model-listing' import type { JsModel } from '../js-model' import type { JsModelFunctions } from '../js-model-functions' +import { JsModelLookup } from '../js-model-lookup' /** * @hidden This type is not part of the public API; it is exposed only for use in * tests in the runtime-async package. */ -export type VarIdForSpec = (varSpec: VarSpec) => VarId | undefined - -/** - * @hidden This type is not part of the public API; it is exposed only for use in - * tests in the runtime-async package. - */ -export type OnEvalAux = (vars: Map /*, lookups: Map*/) => void +export type OnEvalAux = (vars: Map, lookups: Map) => void /** * @hidden This type is not part of the public API; it is exposed only for use in @@ -35,19 +30,18 @@ export class MockJsModel implements JsModel { private readonly finalTime: number private readonly vars: Map = new Map() - // private readonly lookups: Map = new Map() + private readonly lookups: Map = new Map() private fns: JsModelFunctions private listing: ModelListing public readonly onEvalAux: OnEvalAux - public readonly varIdForSpec: VarIdForSpec constructor(options: { initialTime: number finalTime: number outputVarIds: OutputVarId[] - varIdForSpec?: VarIdForSpec + listing?: ModelListing onEvalAux: OnEvalAux }) { this.outputVarIds = options.outputVarIds @@ -55,21 +49,7 @@ export class MockJsModel implements JsModel { this.initialTime = options.initialTime this.finalTime = options.finalTime this.outputVarIds = options.outputVarIds - if (options.varIdForSpec) { - // Use the provided lookup function - this.varIdForSpec = options.varIdForSpec - } else { - // Use a default lookup function that relies on the `ModelListing` - this.varIdForSpec = (varSpec: VarSpec) => { - for (const [listingVarId, listingSpec] of this.listing.varSpecs) { - // TODO: This doesn't compare subscripts yet - if (listingSpec.varIndex === varSpec.varIndex) { - return listingVarId - } - } - return undefined - } - } + this.listing = options.listing this.onEvalAux = options.onEvalAux } @@ -77,6 +57,16 @@ export class MockJsModel implements JsModel { this.listing = listing } + varIdForSpec(varSpec: VarSpec): VarId { + for (const [listingVarId, listingSpec] of this.listing.varSpecs) { + // TODO: This doesn't compare subscripts yet + if (listingSpec.varIndex === varSpec.varIndex) { + return listingVarId + } + } + return undefined + } + // from JsModel interface getInitialTime(): number { return this.initialTime @@ -117,14 +107,14 @@ export class MockJsModel implements JsModel { // TODO } - // // from JsModel interface - // setLookup(varSpec: VarSpec, points: Float64Array): void { - // const varId = this.varIdForSpec(varSpec) - // if (varId === undefined) { - // throw new Error(`No lookup variable found for spec ${varSpec}`) - // } - // this.lookups.set(varId, new JsModelLookup(points.length / 2, points)) - // } + // from JsModel interface + setLookup(varSpec: VarSpec, points: Float64Array): void { + const varId = this.varIdForSpec(varSpec) + if (varId === undefined) { + throw new Error(`No lookup variable found for spec ${varSpec}`) + } + this.lookups.set(varId, new JsModelLookup(points.length / 2, points)) + } // from JsModel interface storeOutputs(storeValue: (value: number) => void): void { @@ -150,7 +140,7 @@ export class MockJsModel implements JsModel { // from JsModel interface evalAux(): void { - this.onEvalAux?.(this.vars /*, this.lookups*/) + this.onEvalAux?.(this.vars, this.lookups) } // from JsModel interface diff --git a/packages/runtime/src/js-model/js-model-constants.ts b/packages/runtime/src/js-model/js-model-constants.ts new file mode 100644 index 00000000..1b3ed8bf --- /dev/null +++ b/packages/runtime/src/js-model/js-model-constants.ts @@ -0,0 +1,6 @@ +// Copyright (c) 2024 Climate Interactive / New Venture Fund + +// This matches Vensim's definition of `:NA:`. It is also defined +// with the same value in the generated `JsModel`, so make sure +// these two values are the same. +export const _NA_ = -Number.MAX_VALUE diff --git a/packages/runtime/src/js-model/js-model-functions.spec.ts b/packages/runtime/src/js-model/js-model-functions.spec.ts index 4c7c3687..827c8720 100644 --- a/packages/runtime/src/js-model/js-model-functions.spec.ts +++ b/packages/runtime/src/js-model/js-model-functions.spec.ts @@ -1,13 +1,13 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund import { beforeEach, describe, expect, it } from 'vitest' + +import { _NA_ } from './js-model-constants' import { getJsModelFunctions, type JsModelFunctionContext, type JsModelFunctions } from './js-model-functions' describe('JsModelFunctions', () => { const fns = getJsModelFunctions() const ctx: JsModelFunctionContext = { - initialTime: 0, - finalTime: 10, timeStep: 1, currentTime: 0 } @@ -172,7 +172,6 @@ describe('JsModelFunctions', () => { // STEP: currentTime + timeStep / 2.0 > stepTime ? height : 0.0 const height = 5 const stepTime = 15 - ctx.finalTime = 20 ctx.timeStep = 2 expect(fnsAtTime(10).STEP(height, stepTime)).toBe(0) expect(fnsAtTime(13).STEP(height, stepTime)).toBe(0) @@ -206,6 +205,13 @@ describe('JsModelFunctions', () => { expect(fns.LOOKUP(lookup, 2)).toBe(4) expect(fns.LOOKUP(lookup, 3)).toBe(6) expect(fns.LOOKUP(lookup, 4)).toBe(6) + + // Verify that it returns _NA_ for an empty lookup + const empty = fns.createLookup(0, []) + expect(fns.LOOKUP(empty, 0)).toBe(_NA_) + + // Verify that it returns _NA_ for an undefined lookup + expect(fns.LOOKUP(undefined, 0)).toBe(_NA_) }) it('should expose LOOKUP_FORWARD', () => { @@ -216,6 +222,13 @@ describe('JsModelFunctions', () => { expect(fns.LOOKUP_FORWARD(lookup, 2)).toBe(6) expect(fns.LOOKUP_FORWARD(lookup, 3)).toBe(6) expect(fns.LOOKUP_FORWARD(lookup, 4)).toBe(6) + + // Verify that it returns _NA_ for an empty lookup + const empty = fns.createLookup(0, []) + expect(fns.LOOKUP_FORWARD(empty, 0)).toBe(_NA_) + + // Verify that it returns _NA_ for an undefined lookup + expect(fns.LOOKUP_FORWARD(undefined, 0)).toBe(_NA_) }) it('should expose LOOKUP_BACKWARD', () => { @@ -226,6 +239,13 @@ describe('JsModelFunctions', () => { expect(fns.LOOKUP_BACKWARD(lookup, 2)).toBe(2) expect(fns.LOOKUP_BACKWARD(lookup, 3)).toBe(6) expect(fns.LOOKUP_BACKWARD(lookup, 4)).toBe(6) + + // Verify that it returns _NA_ for an empty lookup + const empty = fns.createLookup(0, []) + expect(fns.LOOKUP_BACKWARD(empty, 0)).toBe(_NA_) + + // Verify that it returns _NA_ for an undefined lookup + expect(fns.LOOKUP_BACKWARD(undefined, 0)).toBe(_NA_) }) it('should expose LOOKUP_INVERT', () => { @@ -239,6 +259,13 @@ describe('JsModelFunctions', () => { expect(fns.LOOKUP_INVERT(lookup, 5)).toBe(2.5) expect(fns.LOOKUP_INVERT(lookup, 6)).toBe(3) expect(fns.LOOKUP_INVERT(lookup, 7)).toBe(3) + + // Verify that it returns _NA_ for an empty lookup + const empty = fns.createLookup(0, []) + expect(fns.LOOKUP_INVERT(empty, 0)).toBe(_NA_) + + // Verify that it returns _NA_ for an undefined lookup + expect(fns.LOOKUP_INVERT(undefined, 0)).toBe(_NA_) }) it('should expose WITH_LOOKUP', () => { @@ -248,6 +275,13 @@ describe('JsModelFunctions', () => { expect(fns.WITH_LOOKUP(2, lookup)).toBe(4) expect(fns.WITH_LOOKUP(3, lookup)).toBe(6) expect(fns.WITH_LOOKUP(4, lookup)).toBe(6) + + // Verify that it returns _NA_ for an empty lookup + const empty = fns.createLookup(0, []) + expect(fns.WITH_LOOKUP(0, empty)).toBe(_NA_) + + // Verify that it returns _NA_ for an undefined lookup + expect(fns.WITH_LOOKUP(0, undefined)).toBe(_NA_) }) it('should expose GET_DATA_BETWEEN_TIMES', () => { @@ -319,5 +353,12 @@ describe('JsModelFunctions', () => { expect(fns.GET_DATA_BETWEEN_TIMES(lookup, 10.0, -1)).toBe(70) expect(fns.GET_DATA_BETWEEN_TIMES(lookup, 10.5, -1)).toBe(70) expect(fns.GET_DATA_BETWEEN_TIMES(lookup, 11.0, -1)).toBe(70) + + // Verify that it returns _NA_ for an empty lookup + const empty = fns.createLookup(0, []) + expect(fns.GET_DATA_BETWEEN_TIMES(empty, 0, 0)).toBe(_NA_) + + // Verify that it returns _NA_ for an undefined lookup + expect(fns.GET_DATA_BETWEEN_TIMES(undefined, 0, 0)).toBe(_NA_) }) }) diff --git a/packages/runtime/src/js-model/js-model-functions.ts b/packages/runtime/src/js-model/js-model-functions.ts index 21fb41b8..f6edef03 100644 --- a/packages/runtime/src/js-model/js-model-functions.ts +++ b/packages/runtime/src/js-model/js-model-functions.ts @@ -1,16 +1,12 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund +import { _NA_ } from './js-model-constants' import { JsModelLookup, type JsModelLookupMode } from './js-model-lookup' // See XIDZ documentation for an explanation of this value: // https://www.vensim.com/documentation/fn_xidz.html const EPSILON = 1e-6 -// This matches Vensim's definition of `:NA:`. It is also defined -// with the same value in the generated `JsModel`, so make sure -// these two values are the same. -const _NA_ = -Number.MAX_VALUE - /** * Provides access to the minimal set of control parameters that are used in the * implementation of certain model functions. @@ -61,7 +57,7 @@ export interface JsModelFunctions { XIDZ(a: number, b: number, x: number): number ZIDZ(a: number, b: number): number - createLookup(size: number, data: number[]): JsModelLookup + createLookup(size: number, data: number[] | Float64Array): JsModelLookup LOOKUP(lookup: JsModelLookup, x: number): number LOOKUP_FORWARD(lookup: JsModelLookup, x: number): number LOOKUP_BACKWARD(lookup: JsModelLookup, x: number): number @@ -281,7 +277,7 @@ export function getJsModelFunctions(): JsModelFunctions { // Lookup functions // - createLookup(size: number, data: number[]): JsModelLookup { + createLookup(size: number, data: number[] | Float64Array): JsModelLookup { return new JsModelLookup(size, data) }, diff --git a/packages/runtime/src/js-model/js-model-lookup.ts b/packages/runtime/src/js-model/js-model-lookup.ts index 8e29d5cb..206fcd7b 100644 --- a/packages/runtime/src/js-model/js-model-lookup.ts +++ b/packages/runtime/src/js-model/js-model-lookup.ts @@ -1,5 +1,7 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund +import { _NA_ } from './js-model-constants' + export type JsModelLookupMode = 'interpolate' | 'forward' | 'backward' /** @@ -10,10 +12,19 @@ export class JsModelLookup { private lastInput: number private lastHitIndex: number - constructor(private readonly n: number, private readonly data: number[]) { + /** + * @param n The number of (x,y) pairs in the lookup. + * @param data The lookup data, as (x,y) pairs. The length of the array must be + * >= 2*n. Note that the data will be stored by reference, so if there is a chance + * that the array will be reused or modified by other code, be sure to pass in a + * copy of the array. + */ + constructor(private readonly n: number, private readonly data: number[] | Float64Array) { + // Note that we reference the provided data without copying (assumed to be owned elsewhere) if (data.length < n * 2) { throw new Error(`Lookup data array length must be >= 2*size (length=${data.length} size=${n}`) } + this.lastInput = Number.MAX_VALUE this.lastHitIndex = 0 } @@ -42,6 +53,10 @@ export class JsModelLookup { * NOTE: The x values are assumed to be monotonically increasing. */ private getValue(input: number, useInvertedData: boolean, mode: JsModelLookupMode): number { + if (this.n === 0) { + return _NA_ + } + const data = useInvertedData ? this.invertedData : this.data const max = this.n * 2 @@ -110,6 +125,10 @@ export class JsModelLookup { * lookup behavior, so we implement it as a separate method here. */ public getValueBetweenTimes(input: number, mode: JsModelLookupMode): number { + if (this.n === 0) { + return _NA_ + } + const max = this.n * 2 switch (mode) { diff --git a/packages/runtime/src/js-model/js-model.ts b/packages/runtime/src/js-model/js-model.ts index 0b376b33..8f57b7ef 100644 --- a/packages/runtime/src/js-model/js-model.ts +++ b/packages/runtime/src/js-model/js-model.ts @@ -1,6 +1,6 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund -import { indicesPerVariable, type VarSpec } from '../_shared' +import { indicesPerVariable, type LookupDef, type VarSpec } from '../_shared' import type { RunnableModel } from '../runnable-model' import { BaseRunnableModel } from '../runnable-model/base-runnable-model' @@ -38,9 +38,9 @@ export interface JsModel { setTime(time: number): void setInputs(inputValue: (index: number) => number): void + setLookup(varSpec: VarSpec, points: Float64Array): void storeOutputs(storeValue: (value: number) => void): void - /** @hidden */ storeOutput(varSpec: VarSpec, storeValue: (value: number) => void): void initConstants(): void @@ -78,8 +78,20 @@ export function initJsModel(model: JsModel): RunnableModel { saveFreq: saveFreq, numSavePoints, outputVarIds: model.outputVarIds, - onRunModel: (inputs, outputs, outputIndices) => { - runJsModel(model, initialTime, finalTime, timeStep, saveFreq, numSavePoints, inputs, outputs, outputIndices) + onRunModel: (inputs, outputs, options) => { + runJsModel( + model, + initialTime, + finalTime, + timeStep, + saveFreq, + numSavePoints, + inputs, + outputs, + options?.outputIndices, + options?.lookups, + undefined + ) } }) } @@ -93,7 +105,9 @@ function runJsModel( numSavePoints: number, inputs: Float64Array, outputs: Float64Array, - outputIndices: Int32Array | undefined + outputIndices: Int32Array | undefined, + lookups: LookupDef[] | undefined, + stopAfterTime: number | undefined ): void { // Initialize time with the required `INITIAL TIME` control variable let time = initialTime @@ -110,6 +124,13 @@ function runJsModel( // Initialize constants to their default values model.initConstants() + // Apply lookup overrides, if provided + if (lookups !== undefined) { + for (const lookupDef of lookups) { + model.setLookup(lookupDef.varSpec, lookupDef.points) + } + } + if (inputs.length > 0) { // Set the user-defined input values. This needs to happen after `initConstants` // since the input values will override the default constant values. @@ -120,7 +141,12 @@ function runJsModel( model.initLevels() // Set up a run loop using a fixed number of time steps + // TODO: For now we run up to and including `finalTime` (even when `stopAfterTime` + // is defined), storing undefined for values after passing the `stopAfterTime`. + // We should change this to instead stop running the model after passing the + // `stopAfterTime` and have a simpler loop that stores undefined values. const lastStep = Math.round((finalTime - initialTime) / timeStep) + const stopTime = stopAfterTime !== undefined ? stopAfterTime : finalTime let step = 0 let savePointIndex = 0 let outputVarIndex = 0 @@ -134,7 +160,7 @@ function runJsModel( // Write each value into the preallocated buffer; each variable has a "row" that // contains `numSavePoints` values, one value for each save point const outputBufferIndex = outputVarIndex * numSavePoints + savePointIndex - outputs[outputBufferIndex] = value + outputs[outputBufferIndex] = time <= stopTime ? value : undefined outputVarIndex++ } if (outputIndices !== undefined) { diff --git a/packages/runtime/src/model-listing/model-listing.spec.ts b/packages/runtime/src/model-listing/model-listing.spec.ts index 4d96bf03..4ce5fc7c 100644 --- a/packages/runtime/src/model-listing/model-listing.spec.ts +++ b/packages/runtime/src/model-listing/model-listing.spec.ts @@ -1,8 +1,8 @@ // Copyright (c) 2023 Climate Interactive / New Venture Fund import { describe, expect, it } from 'vitest' -import { ModelListing } from './model-listing' import { Outputs } from '../_shared/outputs' +import { ModelListing } from './model-listing' const json = ` { diff --git a/packages/runtime/src/model-runner/model-runner.ts b/packages/runtime/src/model-runner/model-runner.ts index 8ebb2980..ddb93201 100644 --- a/packages/runtime/src/model-runner/model-runner.ts +++ b/packages/runtime/src/model-runner/model-runner.ts @@ -1,6 +1,7 @@ // Copyright (c) 2020-2022 Climate Interactive / New Venture Fund import type { InputValue, Outputs } from '../_shared' +import type { RunModelOptions } from '../runnable-model' /** * Abstraction that allows for running the wasm model on the JS thread @@ -20,21 +21,23 @@ export interface ModelRunner { * * @param inputs The model input values (must be in the same order as in the spec file). * @param outputs The structure into which the model outputs will be stored. + * @param options Additional options that influence the model run. * @return A promise that resolves with the outputs when the model run is complete. */ - runModel(inputs: (number | InputValue)[], outputs: Outputs): Promise + runModel(inputs: (number | InputValue)[], outputs: Outputs, options?: RunModelOptions): Promise /** * Run the model synchronously. * * @param inputs The model input values (must be in the same order as in the spec file). * @param outputs The structure into which the model outputs will be stored. + * @param options Additional options that influence the model run. * @return The outputs of the run. * * @hidden This is only intended for internal use; some implementations may not support * running the model synchronously, in which case this will be undefined. */ - runModelSync?(inputs: (number | InputValue)[], outputs: Outputs): Outputs + runModelSync?(inputs: (number | InputValue)[], outputs: Outputs, options?: RunModelOptions): Outputs /** * Terminate the runner by releasing underlying resources (e.g., the worker thread or diff --git a/packages/runtime/src/model-runner/synchronous-model-runner.spec.ts b/packages/runtime/src/model-runner/synchronous-model-runner.spec.ts index 26282028..56128efe 100644 --- a/packages/runtime/src/model-runner/synchronous-model-runner.spec.ts +++ b/packages/runtime/src/model-runner/synchronous-model-runner.spec.ts @@ -2,7 +2,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest' -import { createInputValue /*, createLookupDef*/ } from '../_shared' +import { createInputValue, createLookupDef } from '../_shared' import { ModelListing } from '../model-listing' @@ -20,20 +20,20 @@ function createMockJsModel(): MockJsModel { initialTime: startTime, finalTime: endTime, outputVarIds: ['_output_1', '_output_2'], - onEvalAux: (vars /*, lookups*/) => { + onEvalAux: (vars, lookups) => { const time = vars.get('_time') - // if (lookups.size > 0) { - // const lookup1 = lookups.get('_output_1_data') - // const lookup2 = lookups.get('_output_2_data') - // expect(lookup1).toBeDefined() - // expect(lookup2).toBeDefined() - // vars.set('_output_1', lookup1.getValueForX(time, 'interpolate')) - // vars.set('_output_2', lookup2.getValueForX(time, 'interpolate')) - // } else { - vars.set('_output_1', time - startTime + 1) - vars.set('_output_2', time - startTime + 4) - vars.set('_x', time - startTime + 7) - // } + if (lookups.size > 0) { + const lookup1 = lookups.get('_output_1_data') + const lookup2 = lookups.get('_output_2_data') + expect(lookup1).toBeDefined() + expect(lookup2).toBeDefined() + vars.set('_output_1', lookup1.getValueForX(time, 'interpolate')) + vars.set('_output_2', lookup2.getValueForX(time, 'interpolate')) + } else { + vars.set('_output_1', time - startTime + 1) + vars.set('_output_2', time - startTime + 4) + vars.set('_x', time - startTime + 7) + } } }) } @@ -43,42 +43,42 @@ function createMockWasmModule(): MockWasmModule { initialTime: startTime, finalTime: endTime, outputVarIds: ['_output_1', '_output_2'], - onRunModel: (inputs, outputs, /*lookups,*/ outputIndices) => { + onRunModel: (inputs, outputs, lookups, outputIndices) => { // Verify inputs expect(inputs).toEqual(new Float64Array([7, 8, 9])) - // if (lookups.size > 0) { - // // Pretend that outputs are derived from lookup data - // const lookup1 = lookups.get('_output_1_data') - // const lookup2 = lookups.get('_output_2_data') - // expect(lookup1).toBeDefined() - // expect(lookup2).toBeDefined() - // for (let i = 0; i < 3; i++) { - // outputs[i] = lookup1.getValueForX(2000 + i, 'interpolate') - // outputs[i + 3] = lookup2.getValueForX(2000 + i, 'interpolate') - // } - // } else { - if (outputIndices === undefined) { - // Store 3 values for the _output_1, and 3 for _output_2 - outputs.set([1, 2, 3, 4, 5, 6]) + if (lookups.size > 0) { + // Pretend that outputs are derived from lookup data + const lookup1 = lookups.get('_output_1_data') + const lookup2 = lookups.get('_output_2_data') + expect(lookup1).toBeDefined() + expect(lookup2).toBeDefined() + for (let i = 0; i < 3; i++) { + outputs[i] = lookup1.getValueForX(2000 + i, 'interpolate') + outputs[i + 3] = lookup2.getValueForX(2000 + i, 'interpolate') + } } else { - // Verify output indices - expect(outputIndices).toEqual( - new Int32Array([ - // _x - 3, 0, 0, 0, - // _output_2 - 2, 0, 0, 0, - // _output_1 - 1, 0, 0, 0, - // (zero terminator) - 0, 0, 0, 0 - ]) - ) - // Store 3 values for each of the three variables - outputs.set([7, 8, 9, 4, 5, 6, 1, 2, 3]) + if (outputIndices === undefined) { + // Store 3 values for the _output_1, and 3 for _output_2 + outputs.set([1, 2, 3, 4, 5, 6]) + } else { + // Verify output indices + expect(outputIndices).toEqual( + new Int32Array([ + // _x + 3, 0, 0, 0, + // _output_2 + 2, 0, 0, 0, + // _output_1 + 1, 0, 0, 0, + // (zero terminator) + 0, 0, 0, 0 + ]) + ) + // Store 3 values for each of the three variables + outputs.set([7, 8, 9, 4, 5, 6, 1, 2, 3]) + } } - // } } }) } @@ -130,6 +130,71 @@ describe.each([ expect(outOutputs.getSeriesForVar('_output_2').points).toEqual([p(2000, 4), p(2001, 5), p(2002, 6)]) }) + it('should run the model (with lookup overrides)', async () => { + const json = ` +{ + "dimensions": [ + ], + "variables": [ + { + "refId": "_output_1", + "varName": "_output_1", + "varIndex": 1 + }, + { + "refId": "_output_1_data", + "varName": "_output_1_data", + "varIndex": 2 + }, + { + "refId": "_output_2", + "varName": "_output_2", + "varIndex": 3 + }, + { + "refId": "_output_2_data", + "varName": "_output_2_data", + "varIndex": 4 + } + ] +} +` + + const listing = new ModelListing(json) + mock.setListing(listing) + + const inputs = [createInputValue('_input_1', 7), createInputValue('_input_2', 8), createInputValue('_input_3', 9)] + let outputs = runner.createOutputs() + + // Run once without lookup overrides + outputs = await runner.runModel(inputs, outputs) + + // Verify that outputs contain the original values + expect(outputs.getSeriesForVar('_output_1').points).toEqual([p(2000, 1), p(2001, 2), p(2002, 3)]) + expect(outputs.getSeriesForVar('_output_2').points).toEqual([p(2000, 4), p(2001, 5), p(2002, 6)]) + + // Run again, this time with lookup overrides + const lookup1Points = [p(2000, 101), p(2001, 102), p(2002, 103)] + const lookup2Points = [p(2000, 104), p(2001, 105), p(2002, 106)] + outputs = await runner.runModel(inputs, outputs, { + lookups: [ + createLookupDef(listing.varSpecs.get('_output_1_data'), lookup1Points), + createLookupDef(listing.varSpecs.get('_output_2_data'), lookup2Points) + ] + }) + + // Verify that outputs contain the values from the overridden lookups + expect(outputs.getSeriesForVar('_output_1').points).toEqual(lookup1Points) + expect(outputs.getSeriesForVar('_output_2').points).toEqual(lookup2Points) + + // Run again without lookup overrides + outputs = await runner.runModel(inputs, outputs) + + // Verify that the lookup overrides are still in effect from the previous run + expect(outputs.getSeriesForVar('_output_1').points).toEqual(lookup1Points) + expect(outputs.getSeriesForVar('_output_2').points).toEqual(lookup2Points) + }) + it('should run the model (when output var specs are included)', async () => { const json = ` { diff --git a/packages/runtime/src/model-runner/synchronous-model-runner.ts b/packages/runtime/src/model-runner/synchronous-model-runner.ts index c92c4356..f063354e 100644 --- a/packages/runtime/src/model-runner/synchronous-model-runner.ts +++ b/packages/runtime/src/model-runner/synchronous-model-runner.ts @@ -4,7 +4,8 @@ import { type InputValue, Outputs } from '../_shared' import { type JsModel, initJsModel } from '../js-model' import { type WasmModule, initWasmModel } from '../wasm-model' -import { type RunnableModel, ReferencedRunModelParams } from '../runnable-model' +import type { RunnableModel, RunModelOptions } from '../runnable-model' +import { ReferencedRunModelParams } from '../runnable-model' import type { ModelRunner } from './model-runner' @@ -53,9 +54,9 @@ function createRunnerFromRunnableModel(model: RunnableModel): ModelRunner { // Disallow `runModel` after the runner has been terminated let terminated = false - const runModelSync = (inputs: (InputValue | number)[], outputs: Outputs) => { + const runModelSync = (inputs: (InputValue | number)[], outputs: Outputs, options: RunModelOptions | undefined) => { // Update the I/O parameters - params.updateFromParams(inputs, outputs) + params.updateFromParams(inputs, outputs, options) // Run the model synchronously using those parameters model.runModel(params) @@ -68,18 +69,18 @@ function createRunnerFromRunnableModel(model: RunnableModel): ModelRunner { return new Outputs(model.outputVarIds, model.startTime, model.endTime, model.saveFreq) }, - runModel: (inputs, outputs) => { + runModel: (inputs, outputs, options) => { if (terminated) { return Promise.reject(new Error('Model runner has already been terminated')) } - return Promise.resolve(runModelSync(inputs, outputs)) + return Promise.resolve(runModelSync(inputs, outputs, options)) }, - runModelSync: (inputs, outputs) => { + runModelSync: (inputs, outputs, options) => { if (terminated) { throw new Error('Model runner has already been terminated') } - return runModelSync(inputs, outputs) + return runModelSync(inputs, outputs, options) }, terminate: async () => { diff --git a/packages/runtime/src/runnable-model/base-runnable-model.ts b/packages/runtime/src/runnable-model/base-runnable-model.ts index 332989f9..b40baa32 100644 --- a/packages/runtime/src/runnable-model/base-runnable-model.ts +++ b/packages/runtime/src/runnable-model/base-runnable-model.ts @@ -1,6 +1,6 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund -import type { OutputVarId } from '../_shared' +import type { LookupDef, OutputVarId } from '../_shared' import { perfElapsed, perfNow } from '../perf' import type { RunModelParams } from './run-model-params' import type { RunnableModel } from './runnable-model' @@ -11,7 +11,10 @@ import type { RunnableModel } from './runnable-model' export type OnRunModelFunc = ( inputs: Float64Array, outputs: Float64Array, - outputIndices: Int32Array | undefined + options?: { + outputIndices?: Int32Array + lookups?: LookupDef[] + } ) => void /** @@ -87,7 +90,10 @@ export class BaseRunnableModel implements RunnableModel { // Run the model const t0 = perfNow() - this.onRunModel?.(inputsArray, outputsArray, outputIndicesArray) + this.onRunModel?.(inputsArray, outputsArray, { + outputIndices: outputIndicesArray, + lookups: params.getLookups() + }) const elapsed = perfElapsed(t0) // Copy the outputs that were stored into our array back to the `RunModelParams` diff --git a/packages/runtime/src/runnable-model/buffered-run-model-params.spec.ts b/packages/runtime/src/runnable-model/buffered-run-model-params.spec.ts index 9671a76c..19322f2d 100644 --- a/packages/runtime/src/runnable-model/buffered-run-model-params.spec.ts +++ b/packages/runtime/src/runnable-model/buffered-run-model-params.spec.ts @@ -2,10 +2,10 @@ import { describe, expect, it } from 'vitest' -import { Outputs } from '../_shared' -import { ModelListing } from '../model-listing' +import { Outputs, createLookupDef, type LookupDef } from '../_shared' import { BufferedRunModelParams } from './buffered-run-model-params' +import { ModelListing } from '../model-listing' const json = ` { @@ -248,4 +248,50 @@ describe('BufferedRunModelParams', () => { expect(outputs.getSeriesForVar('_y').points).toEqual([p(2000, 4), p(2001, 5), p(2002, 6)]) expect(outputs.runTimeInMillis).toBe(42) }) + + it('should copy lookups', () => { + const listing = new ModelListing(json) + + const inputs = [1, 2, 3] + const outputs = new Outputs(['_x', '_y'], 2000, 2002, 1) + + const lookups: LookupDef[] = [ + createLookupDef(listing.varSpecs.get('_a'), [p(2000, 0), p(2001, 1), p(2002, 2)]), + createLookupDef(listing.varSpecs.get('_b'), [p(2000, 5), p(2001, 6), p(2002, 7)]) + ] + + const runnerParams = new BufferedRunModelParams() + const workerParams = new BufferedRunModelParams() + + // Run once without providing lookups + runnerParams.updateFromParams(inputs, outputs) + workerParams.updateFromEncodedBuffer(runnerParams.getEncodedBuffer()) + + // Verify that lookups array is undefined + expect(workerParams.getLookups()).toBeUndefined() + + // Run again with lookups + runnerParams.updateFromParams(inputs, outputs, { lookups }) + workerParams.updateFromEncodedBuffer(runnerParams.getEncodedBuffer()) + + // Verify that lookups array on the worker side contains the expected values + expect(workerParams.getLookups()).toEqual(lookups) + + // Run again without lookups + runnerParams.updateFromParams(inputs, outputs) + workerParams.updateFromEncodedBuffer(runnerParams.getEncodedBuffer()) + + // Verify that lookups array is undefined + expect(workerParams.getLookups()).toBeUndefined() + + // Run again with an empty lookup + const emptyLookup = createLookupDef(listing.varSpecs.get('_a'), []) + runnerParams.updateFromParams(inputs, outputs, { + lookups: [emptyLookup] + }) + workerParams.updateFromEncodedBuffer(runnerParams.getEncodedBuffer()) + + // Verify that lookups array on the worker side contains the expected values + expect(workerParams.getLookups()).toEqual([emptyLookup]) + }) }) diff --git a/packages/runtime/src/runnable-model/buffered-run-model-params.ts b/packages/runtime/src/runnable-model/buffered-run-model-params.ts index 1493ba9b..bfb266ea 100644 --- a/packages/runtime/src/runnable-model/buffered-run-model-params.ts +++ b/packages/runtime/src/runnable-model/buffered-run-model-params.ts @@ -1,6 +1,8 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund -import { indicesPerVariable, updateVarIndices, type InputValue, type Outputs } from '../_shared' +import { indicesPerVariable, updateVarIndices } from '../_shared' +import type { InputValue, LookupDef, Outputs, VarSpec } from '../_shared' +import type { RunModelOptions } from './run-model-options' import type { RunModelParams } from './run-model-params' const headerLengthInElements = 16 @@ -62,6 +64,8 @@ export class BufferedRunModelParams implements RunModelParams { * inputs * outputs * outputIndices + * lookups (data) + * lookupIndices */ private encoded: ArrayBuffer @@ -83,6 +87,12 @@ export class BufferedRunModelParams implements RunModelParams { /** The output indices section of the `encoded` buffer. */ private readonly outputIndices = new Int32Section() + /** The lookup data section of the `encoded` buffer. */ + private readonly lookups = new Float64Section() + + /** The lookup indices section of the `encoded` buffer. */ + private readonly lookupIndices = new Int32Section() + /** * Return the encoded buffer from this instance, which can be passed to `updateFromEncodedBuffer`. */ @@ -154,6 +164,17 @@ export class BufferedRunModelParams implements RunModelParams { this.outputs.view?.set(array) } + // from RunModelParams interface + getLookups(): LookupDef[] | undefined { + if (this.lookupIndices.lengthInElements === 0) { + return undefined + } + + // Reconstruct the `LookupDef` instances using the data from the lookup data and + // indices buffers + return decodeLookups(this.lookups.view, this.lookupIndices.view) + } + // from RunModelParams interface getElapsedTime(): number { return this.extras.view[0] @@ -187,11 +208,14 @@ export class BufferedRunModelParams implements RunModelParams { * * @param inputs The model input values (must be in the same order as in the spec file). * @param outputs The structure into which the model outputs will be stored. + * @param options Additional options that influence the model run. */ - updateFromParams(inputs: (number | InputValue)[], outputs: Outputs): void { - // Determine the number of elements in each section + updateFromParams(inputs: (number | InputValue)[], outputs: Outputs, options?: RunModelOptions): void { + // Determine the number of elements in the input and output sections const inputsLengthInElements = inputs.length const outputsLengthInElements = outputs.varIds.length * outputs.seriesLength + + // Determine the number of elements in the output indices section let outputIndicesLengthInElements: number const outputVarSpecs = outputs.varSpecs if (outputVarSpecs !== undefined && outputVarSpecs.length > 0) { @@ -203,6 +227,20 @@ export class BufferedRunModelParams implements RunModelParams { outputIndicesLengthInElements = 0 } + // Determine the number of elements in the lookup data and indices sections + let lookupsLengthInElements: number + let lookupIndicesLengthInElements: number + if (options?.lookups !== undefined && options.lookups.length > 0) { + // Compute the required lengths + const encodedLengths = getEncodedLookupBufferLengths(options.lookups) + lookupsLengthInElements = encodedLengths.lookupsLength + lookupIndicesLengthInElements = encodedLengths.lookupIndicesLength + } else { + // Don't use the lookup data and indices buffers when lookup overrides are not provided + lookupsLengthInElements = 0 + lookupIndicesLengthInElements = 0 + } + // Compute the byte offset and byte length of each section let byteOffset = 0 function section(kind: 'float64' | 'int32', lengthInElements: number): number { @@ -222,6 +260,8 @@ export class BufferedRunModelParams implements RunModelParams { const inputsOffsetInBytes = section('float64', inputsLengthInElements) const outputsOffsetInBytes = section('float64', outputsLengthInElements) const outputIndicesOffsetInBytes = section('int32', outputIndicesLengthInElements) + const lookupsOffsetInBytes = section('float64', lookupsLengthInElements) + const lookupIndicesOffsetInBytes = section('int32', lookupIndicesLengthInElements) // Get the total byte length const requiredLengthInBytes = byteOffset @@ -248,6 +288,10 @@ export class BufferedRunModelParams implements RunModelParams { headerView[headerIndex++] = outputsLengthInElements headerView[headerIndex++] = outputIndicesOffsetInBytes headerView[headerIndex++] = outputIndicesLengthInElements + headerView[headerIndex++] = lookupsOffsetInBytes + headerView[headerIndex++] = lookupsLengthInElements + headerView[headerIndex++] = lookupIndicesOffsetInBytes + headerView[headerIndex++] = lookupIndicesLengthInElements // Update the views // TODO: We can avoid recreating the views every time if buffer and section offset/length @@ -256,6 +300,8 @@ export class BufferedRunModelParams implements RunModelParams { this.extras.update(this.encoded, extrasOffsetInBytes, extrasLengthInElements) this.outputs.update(this.encoded, outputsOffsetInBytes, outputsLengthInElements) this.outputIndices.update(this.encoded, outputIndicesOffsetInBytes, outputIndicesLengthInElements) + this.lookups.update(this.encoded, lookupsOffsetInBytes, lookupsLengthInElements) + this.lookupIndices.update(this.encoded, lookupIndicesOffsetInBytes, lookupIndicesLengthInElements) // Copy the input values into the internal buffer // TODO: Throw an error if inputs.length is less than number of inputs declared @@ -278,6 +324,11 @@ export class BufferedRunModelParams implements RunModelParams { if (this.outputIndices.view) { updateVarIndices(this.outputIndices.view, outputVarSpecs) } + + // Copy the lookup data and indices into the internal buffers, if needed + if (lookupIndicesLengthInElements > 0) { + encodeLookups(options.lookups, this.lookups.view, this.lookupIndices.view) + } } /** @@ -311,18 +362,26 @@ export class BufferedRunModelParams implements RunModelParams { const outputsLengthInElements = headerView[headerIndex++] const outputIndicesOffsetInBytes = headerView[headerIndex++] const outputIndicesLengthInElements = headerView[headerIndex++] + const lookupsOffsetInBytes = headerView[headerIndex++] + const lookupsLengthInElements = headerView[headerIndex++] + const lookupIndicesOffsetInBytes = headerView[headerIndex++] + const lookupIndicesLengthInElements = headerView[headerIndex++] // Verify that the buffer is long enough to contain all sections const extrasLengthInBytes = extrasLengthInElements * Float64Array.BYTES_PER_ELEMENT const inputsLengthInBytes = inputsLengthInElements * Float64Array.BYTES_PER_ELEMENT const outputsLengthInBytes = outputsLengthInElements * Float64Array.BYTES_PER_ELEMENT const outputIndicesLengthInBytes = outputIndicesLengthInElements * Int32Array.BYTES_PER_ELEMENT + const lookupsLengthInBytes = lookupsLengthInElements * Float64Array.BYTES_PER_ELEMENT + const lookupIndicesLengthInBytes = lookupIndicesLengthInElements * Int32Array.BYTES_PER_ELEMENT const requiredLengthInBytes = headerLengthInBytes + extrasLengthInBytes + inputsLengthInBytes + outputsLengthInBytes + - outputIndicesLengthInBytes + outputIndicesLengthInBytes + + lookupsLengthInBytes + + lookupIndicesLengthInBytes if (buffer.byteLength < requiredLengthInBytes) { throw new Error('Buffer must be long enough to contain sections declared in header') } @@ -332,5 +391,128 @@ export class BufferedRunModelParams implements RunModelParams { this.inputs.update(this.encoded, inputsOffsetInBytes, inputsLengthInElements) this.outputs.update(this.encoded, outputsOffsetInBytes, outputsLengthInElements) this.outputIndices.update(this.encoded, outputIndicesOffsetInBytes, outputIndicesLengthInElements) + this.lookups.update(this.encoded, lookupsOffsetInBytes, lookupsLengthInElements) + this.lookupIndices.update(this.encoded, lookupIndicesOffsetInBytes, lookupIndicesLengthInElements) + } +} + +/** + * Return the lengths of the arrays that are required to store the lookup data and indices for + * the given `LookupDef` instances. + * + * @param lookupDefs The `LookupDef` instances to encode. + */ +function getEncodedLookupBufferLengths(lookupDefs: LookupDef[]): { + lookupsLength: number + lookupIndicesLength: number +} { + // The lookups buffer includes all data points for the provided lookup overrides + // (added sequentially, with no padding between datasets). The lookup indices + // buffer has the following format: + // lookup count + // lookupN var index + // lookupN sub0 index + // lookupN sub1 index + // lookupN sub2 index + // lookupN data offset (relative to the start of the lookups buffer, in float64 elements) + // lookupN data length (in float64 elements) + // ... (repeat for each lookup) + const numIndexElementsForTotalCount = 1 + const numIndexElementsPerLookup = 6 + let lookupsLength = 0 + let lookupIndicesLength = numIndexElementsForTotalCount + for (const lookupDef of lookupDefs) { + lookupsLength += lookupDef.points.length + lookupIndicesLength += numIndexElementsPerLookup } + return { + lookupsLength, + lookupIndicesLength + } +} + +/** + * Encode lookup data and indices to the given buffer views. + * + * @param lookupDefs The `LookupDef` instances to encode. + * @param lookupsView The view on the lookup data buffer. This can be undefined in + * the case where the data for the lookup(s) is empty. + * @param lookupIndicesView The view on the lookup indices buffer. + */ +function encodeLookups( + lookupDefs: LookupDef[], + lookupsView: Float64Array | undefined, + lookupIndicesView: Int32Array +): void { + // Store the total lookup count + let li = 0 + lookupIndicesView[li++] = lookupDefs.length + + // Store the data and indices for each lookup + let lookupDataOffset = 0 + for (const lookupDef of lookupDefs) { + // Store lookup indices + const subs = lookupDef.varSpec.subscriptIndices + const subCount = lookupDef.varSpec.subscriptIndices?.length || 0 + lookupIndicesView[li++] = lookupDef.varSpec.varIndex + lookupIndicesView[li++] = subCount > 0 ? subs[0] : -1 + lookupIndicesView[li++] = subCount > 1 ? subs[1] : -1 + lookupIndicesView[li++] = subCount > 2 ? subs[2] : -1 + lookupIndicesView[li++] = lookupDataOffset + lookupIndicesView[li++] = lookupDef.points.length + + // Store lookup data. Note that `lookupsView` can be undefined in the case + // where the lookup data is empty. + lookupsView?.set(lookupDef.points, lookupDataOffset) + lookupDataOffset += lookupDef.points.length + } +} + +/** + * Decode lookup data and indices from the given buffer views and return the + * reconstruct `LookupDef` instances. + * + * @param lookupsView The view on the lookup data buffer. This can be undefined in + * the case where the data for the lookup(s) is empty. + * @param lookupIndicesView The view on the lookup indices buffer. + */ +function decodeLookups(lookupsView: Float64Array | undefined, lookupIndicesView: Int32Array): LookupDef[] { + const lookupDefs: LookupDef[] = [] + + let li = 0 + const lookupCount = lookupIndicesView[li++] + for (let i = 0; i < lookupCount; i++) { + // Read the metadata from the lookup indices buffer + const varIndex = lookupIndicesView[li++] + const subIndex0 = lookupIndicesView[li++] + const subIndex1 = lookupIndicesView[li++] + const subIndex2 = lookupIndicesView[li++] + const lookupDataOffset = lookupIndicesView[li++] + const lookupDataLength = lookupIndicesView[li++] + const subscriptIndices: number[] = subIndex0 >= 0 ? [] : undefined + if (subIndex0 >= 0) subscriptIndices.push(subIndex0) + if (subIndex1 >= 0) subscriptIndices.push(subIndex1) + if (subIndex2 >= 0) subscriptIndices.push(subIndex2) + const varSpec: VarSpec = { + varIndex, + subscriptIndices + } + + // Copy the data from the lookup data buffer. Note that `lookupsView` can be undefined + // in the case where the lookup data is empty. + // TODO: We can use `subarray` here instead of `slice` and let the model implementations + // copy the data if needed on their side + let points: Float64Array + if (lookupsView) { + points = lookupsView.slice(lookupDataOffset, lookupDataOffset + lookupDataLength) + } else { + points = new Float64Array(0) + } + lookupDefs.push({ + varSpec, + points + }) + } + + return lookupDefs } diff --git a/packages/runtime/src/runnable-model/index.ts b/packages/runtime/src/runnable-model/index.ts index 78789894..c7f0f646 100644 --- a/packages/runtime/src/runnable-model/index.ts +++ b/packages/runtime/src/runnable-model/index.ts @@ -1,5 +1,7 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund +export * from './run-model-options' + export * from './run-model-params' export * from './buffered-run-model-params' export * from './referenced-run-model-params' diff --git a/packages/runtime/src/runnable-model/referenced-run-model-params.spec.ts b/packages/runtime/src/runnable-model/referenced-run-model-params.spec.ts index b65735db..87cf4a93 100644 --- a/packages/runtime/src/runnable-model/referenced-run-model-params.spec.ts +++ b/packages/runtime/src/runnable-model/referenced-run-model-params.spec.ts @@ -2,10 +2,10 @@ import { describe, expect, it } from 'vitest' -import { Outputs } from '../_shared' -import { ModelListing } from '../model-listing' +import { Outputs, createLookupDef, type LookupDef } from '../_shared' import { ReferencedRunModelParams } from './referenced-run-model-params' +import { ModelListing } from '../model-listing' const json = ` { @@ -162,4 +162,45 @@ describe('ReferencedRunModelParams', () => { expect(outputs.getSeriesForVar('_x').points).toEqual([p(2000, 1), p(2001, 2), p(2002, 3)]) expect(outputs.getSeriesForVar('_y').points).toEqual([p(2000, 4), p(2001, 5), p(2002, 6)]) }) + + it('should copy lookups', () => { + const listing = new ModelListing(json) + + const inputs = [1, 2, 3] + const outputs = new Outputs(['_x', '_y'], 2000, 2002, 1) + + const lookups: LookupDef[] = [ + createLookupDef(listing.varSpecs.get('_a'), [p(2000, 0), p(2001, 1), p(2002, 2)]), + createLookupDef(listing.varSpecs.get('_b'), [p(2000, 5), p(2001, 6), p(2002, 7)]) + ] + + const params = new ReferencedRunModelParams() + + // Run once without providing lookups + params.updateFromParams(inputs, outputs) + + // Verify that lookups array is undefined + expect(params.getLookups()).toBeUndefined() + + // Run again with lookups + params.updateFromParams(inputs, outputs, { lookups }) + + // Verify that lookups array contains the expected values + expect(params.getLookups()).toEqual(lookups) + + // Run again without lookups + params.updateFromParams(inputs, outputs) + + // Verify that lookups array is undefined + expect(params.getLookups()).toBeUndefined() + + // Run again with an empty lookup + const emptyLookup = createLookupDef(listing.varSpecs.get('_a'), []) + params.updateFromParams(inputs, outputs, { + lookups: [emptyLookup] + }) + + // Verify that lookups array contains the expected values + expect(params.getLookups()).toEqual([emptyLookup]) + }) }) diff --git a/packages/runtime/src/runnable-model/referenced-run-model-params.ts b/packages/runtime/src/runnable-model/referenced-run-model-params.ts index 1c277725..b9f031a7 100644 --- a/packages/runtime/src/runnable-model/referenced-run-model-params.ts +++ b/packages/runtime/src/runnable-model/referenced-run-model-params.ts @@ -1,6 +1,8 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund -import { indicesPerVariable, updateVarIndices, type InputValue, type Outputs } from '../_shared' +import type { InputValue, LookupDef, Outputs } from '../_shared' +import { indicesPerVariable, updateVarIndices } from '../_shared' +import type { RunModelOptions } from './run-model-options' import type { RunModelParams } from './run-model-params' /** @@ -16,6 +18,7 @@ export class ReferencedRunModelParams implements RunModelParams { private outputs: Outputs private outputsLengthInElements = 0 private outputIndicesLengthInElements = 0 + private lookups: LookupDef[] // from RunModelParams interface getInputs(): Float64Array | undefined { @@ -99,6 +102,15 @@ export class ReferencedRunModelParams implements RunModelParams { } } + // from RunModelParams interface + getLookups(): LookupDef[] | undefined { + if (this.lookups !== undefined && this.lookups.length > 0) { + return this.lookups + } else { + return undefined + } + } + // from RunModelParams interface getElapsedTime(): number { return this.outputs?.runTimeInMillis @@ -117,13 +129,15 @@ export class ReferencedRunModelParams implements RunModelParams { * * @param inputs The model input values (must be in the same order as in the spec file). * @param outputs The structure into which the model outputs will be stored. + * @param options Additional options that influence the model run. */ - updateFromParams(inputs: (number | InputValue)[], outputs: Outputs): void { + updateFromParams(inputs: (number | InputValue)[], outputs: Outputs, options?: RunModelOptions): void { // Save the latest parameters; these values will be accessed by the `RunnableModel` // on demand (e.g., in the `copyInputs` method) this.inputs = inputs this.outputs = outputs this.outputsLengthInElements = outputs.varIds.length * outputs.seriesLength + this.lookups = options?.lookups // See if the output indices are needed const outputVarSpecs = outputs.varSpecs diff --git a/packages/runtime/src/runnable-model/run-model-options.ts b/packages/runtime/src/runnable-model/run-model-options.ts new file mode 100644 index 00000000..505350ed --- /dev/null +++ b/packages/runtime/src/runnable-model/run-model-options.ts @@ -0,0 +1,22 @@ +// Copyright (c) 2024 Climate Interactive / New Venture Fund + +import type { LookupDef } from '../_shared' + +/** + * Additional options that can be passed to a `runModel` call to influence the model run. + */ +export interface RunModelOptions { + /** + * If defined, override the data for the specified lookups and/or data variables. + * + * If data was already defined in the generated model, the data provided in a + * `LookupDef` here will override the default data in the generated model. + * + * Note that unlike the `inputs` parameter for `runModel` (which must be provided + * with each call), the data overrides provided here persist after the `runModel` + * call. If you pass `lookups` in your Nth `runModel` call, that lookup data will + * still be in effect for the (N+1)th call. In other words, if your lookup data + * is not changing, you do not need to supply it with every `runModel` call. + */ + lookups?: LookupDef[] +} diff --git a/packages/runtime/src/runnable-model/run-model-params.ts b/packages/runtime/src/runnable-model/run-model-params.ts index f0abeb7b..d8c8e336 100644 --- a/packages/runtime/src/runnable-model/run-model-params.ts +++ b/packages/runtime/src/runnable-model/run-model-params.ts @@ -1,6 +1,6 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund -import type { Outputs } from '../_shared' +import type { LookupDef, Outputs } from '../_shared' /** * Encapsulates the parameters that are passed to a `runModel` call. @@ -74,6 +74,12 @@ export interface RunModelParams { */ storeOutputs(array: Float64Array): void + /** + * Return an array containing lookup overrides, or undefined if no lookups were passed to + * the latest `runModel` call. + */ + getLookups(): LookupDef[] | undefined + /** * Return the elapsed time (in milliseconds) of the model run. */ diff --git a/packages/runtime/src/wasm-model/_mocks/mock-wasm-module.ts b/packages/runtime/src/wasm-model/_mocks/mock-wasm-module.ts index 6ae856f2..d5028a1d 100644 --- a/packages/runtime/src/wasm-model/_mocks/mock-wasm-module.ts +++ b/packages/runtime/src/wasm-model/_mocks/mock-wasm-module.ts @@ -1,7 +1,7 @@ // Copyright (c) 2024 Climate Interactive / New Venture Fund -import type { OutputVarId /*, VarId*/ } from '../../_shared' -// import { JsModelLookup } from '../../js-model/js-model-lookup' +import type { OutputVarId, VarId, VarSpec } from '../../_shared' +import { JsModelLookup } from '../../js-model/js-model-lookup' import type { ModelListing } from '../../model-listing' import type { WasmModule } from '../wasm-module' @@ -12,7 +12,7 @@ import type { WasmModule } from '../wasm-module' export type OnRunModel = ( inputs: Float64Array, outputs: Float64Array, - // lookups: Map, + lookups: Map, outputIndices?: Int32Array ) => void @@ -42,15 +42,23 @@ export class MockWasmModule implements WasmModule { private mallocOffset = 8 private readonly allocs: Map = new Map() - // private readonly lookups: Map = new Map() - // private listing: ModelListing + private readonly lookups: Map = new Map() + + private listing: ModelListing public readonly onRunModel: OnRunModel - constructor(options: { initialTime: number; finalTime: number; outputVarIds: string[]; onRunModel: OnRunModel }) { + constructor(options: { + initialTime: number + finalTime: number + outputVarIds: string[] + listing?: ModelListing + onRunModel: OnRunModel + }) { this.initialTime = options.initialTime this.finalTime = options.finalTime this.outputVarIds = options.outputVarIds + this.listing = options.listing this.onRunModel = options.onRunModel this.heap = new ArrayBuffer(8192) @@ -58,9 +66,18 @@ export class MockWasmModule implements WasmModule { this.HEAPF64 = new Float64Array(this.heap) } - // eslint-disable-next-line @typescript-eslint/no-unused-vars - setListing(_listing: ModelListing) { - // this.listing = listing + setListing(listing: ModelListing) { + this.listing = listing + } + + varIdForSpec(varSpec: VarSpec): VarId { + for (const [listingVarId, listingSpec] of this.listing.varSpecs) { + // TODO: This doesn't compare subscripts yet + if (listingSpec.varIndex === varSpec.varIndex) { + return listingVarId + } + } + return undefined } // from WasmModule interface @@ -73,29 +90,23 @@ export class MockWasmModule implements WasmModule { return () => this.finalTime case 'getSaveper': return () => 1 - // case 'setLookup': - // return (varIndex: number, _subIndicesAddress: number, pointsAddress: number, numPoints: number) => { - // let varId: VarId - // for (const [listingVarId, listingSpec] of this.listing.varSpecs) { - // // TODO: This doesn't compare subscripts yet - // if (listingSpec.varIndex === varIndex) { - // varId = listingVarId - // break - // } - // } - // if (varId === undefined) { - // throw new Error(`No lookup variable found for index ${varIndex}`) - // } - // // Note that we create a copy of the points array, since it may be reused - // const points = new Float64Array(this.getHeapView('float64', pointsAddress) as Float64Array) - // this.lookups.set(varId, new JsModelLookup(numPoints, points)) - // } + case 'setLookup': + return (varIndex: number, _subIndicesAddress: number, pointsAddress: number, numPoints: number) => { + // TODO: This doesn't check subIndices yet + const varId = this.varIdForSpec({ varIndex }) + if (varId === undefined) { + throw new Error(`No lookup variable found for var index ${varIndex}`) + } + // Note that we create a copy of the points array, since it may be reused + const points = new Float64Array(this.getHeapView('float64', pointsAddress) as Float64Array) + this.lookups.set(varId, new JsModelLookup(numPoints, points)) + } case 'runModelWithBuffers': return (inputsAddress: number, outputsAddress: number, outputIndicesAddress: number) => { const inputs = this.getHeapView('float64', inputsAddress) as Float64Array const outputs = this.getHeapView('float64', outputsAddress) as Float64Array const outputIndices = this.getHeapView('int32', outputIndicesAddress) as Int32Array - this.onRunModel(inputs, outputs, /*this.lookups,*/ outputIndices) + this.onRunModel(inputs, outputs, this.lookups, outputIndices) } default: throw new Error(`Unhandled call to cwrap with function name '${fname}'`) diff --git a/packages/runtime/src/wasm-model/wasm-model.ts b/packages/runtime/src/wasm-model/wasm-model.ts index ca576043..891022a4 100644 --- a/packages/runtime/src/wasm-model/wasm-model.ts +++ b/packages/runtime/src/wasm-model/wasm-model.ts @@ -28,7 +28,15 @@ class WasmModel implements RunnableModel { private inputsBuffer: WasmBuffer private outputsBuffer: WasmBuffer private outputIndicesBuffer: WasmBuffer - + private lookupDataBuffer: WasmBuffer + private lookupSubIndicesBuffer: WasmBuffer + + private readonly wasmSetLookup: ( + varIndex: number, + subIndicesAddress: number, + pointsAddress: number, + numPoints: number + ) => void private readonly wasmRunModel: (inputsAddress: number, outputsAddress: number, outputIndicesAddress: number) => void /** @@ -49,7 +57,8 @@ class WasmModel implements RunnableModel { this.numSavePoints = Math.round((this.endTime - this.startTime) / this.saveFreq) + 1 this.outputVarIds = wasmModule.outputVarIds - // Make the native `runModelWithBuffers` function callable + // Make the native functions callable + this.wasmSetLookup = wasmModule.cwrap('setLookup', null, ['number', 'number', 'number', 'number']) this.wasmRunModel = wasmModule.cwrap('runModelWithBuffers', null, ['number', 'number', 'number']) } @@ -59,6 +68,45 @@ class WasmModel implements RunnableModel { // and copy data to/from them because only that kind of buffer can be passed to // the `wasmRunModel` function. + // Apply lookup overrides, if provided + const lookups = params.getLookups() + if (lookups !== undefined) { + for (const lookupDef of lookups) { + // Copy the subscript index values to the `WasmBuffer`. If we don't have an + // existing `WasmBuffer`, or the existing one is not big enough, allocate a new one. + const numSubElements = lookupDef.varSpec.subscriptIndices?.length || 0 + let subIndicesAddress: number + if (numSubElements > 0) { + if (this.lookupSubIndicesBuffer === undefined || this.lookupSubIndicesBuffer.numElements < numSubElements) { + this.lookupSubIndicesBuffer?.dispose() + this.lookupSubIndicesBuffer = createInt32WasmBuffer(this.wasmModule, numSubElements) + } + this.lookupSubIndicesBuffer.getArrayView().set(lookupDef.varSpec.subscriptIndices) + subIndicesAddress = this.lookupSubIndicesBuffer.getAddress() + } else { + subIndicesAddress = 0 + } + + // Copy the lookup data to the `WasmBuffer`. If we don't have an existing `WasmBuffer`, + // or the existing one is not big enough, allocate a new one. + const numLookupElements = lookupDef.points.length + if (this.lookupDataBuffer === undefined || this.lookupDataBuffer.numElements < numLookupElements) { + this.lookupDataBuffer?.dispose() + this.lookupDataBuffer = createFloat64WasmBuffer(this.wasmModule, numLookupElements) + } + this.lookupDataBuffer.getArrayView().set(lookupDef.points) + const pointsAddress = this.lookupDataBuffer.getAddress() + + // Note that the native `numPoints` argument is the number of (x,y) pairs, but so divide + // the length of the flat points array by two + const numPoints = numLookupElements / 2 + + // Call the native `setLookup` function + const varIndex = lookupDef.varSpec.varIndex + this.wasmSetLookup(varIndex, subIndicesAddress, pointsAddress, numPoints) + } + } + // Copy the inputs to the `WasmBuffer`. If we don't have an existing `WasmBuffer`, // or the existing one is not big enough, the callback will allocate a new one. params.copyInputs(this.inputsBuffer?.getArrayView(), numElements => { @@ -125,7 +173,7 @@ class WasmModel implements RunnableModel { * Initialize the wasm model. * * @hidden This is not part of the public API; only the top-level `createRunnableModel` - * function will be exposed in the public API. + * function is exposed in the public API. * * @param wasmModule The `WasmModule` that wraps the `wasm` binary. * @return The initialized `WasmModel` instance. diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 09aad364..7666ca92 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -572,6 +572,27 @@ importers: specifier: workspace:* version: link:../../../packages/runtime-async + tests/integration/override-lookups: + dependencies: + '@sdeverywhere/build': + specifier: workspace:* + version: link:../../../packages/build + '@sdeverywhere/cli': + specifier: workspace:* + version: link:../../../packages/cli + '@sdeverywhere/plugin-wasm': + specifier: workspace:* + version: link:../../../packages/plugin-wasm + '@sdeverywhere/plugin-worker': + specifier: workspace:* + version: link:../../../packages/plugin-worker + '@sdeverywhere/runtime': + specifier: workspace:* + version: link:../../../packages/runtime + '@sdeverywhere/runtime-async': + specifier: workspace:* + version: link:../../../packages/runtime-async + tests/integration/saveper: dependencies: '@sdeverywhere/build': diff --git a/tests/integration/override-lookups/override-lookups.dat b/tests/integration/override-lookups/override-lookups.dat new file mode 100644 index 00000000..90c2e313 --- /dev/null +++ b/tests/integration/override-lookups/override-lookups.dat @@ -0,0 +1,36 @@ +A data[A1] +2000 100 +2001 200 +2002 300 +A data[A2] +2000 101 +2001 201 +2002 301 +B data[A1,B1] +2000 102 +2001 202 +2002 302 +B data[A1,B2] +2000 103 +2001 203 +2002 303 +B data[A1,B3] +2000 104 +2001 204 +2002 304 +B data[A2,B1] +2000 105 +2001 205 +2002 305 +B data[A2,B2] +2000 106 +2001 206 +2002 306 +B data[A2,B3] +2000 107 +2001 207 +2002 307 +C data +2000 110 +2001 210 +2002 310 diff --git a/tests/integration/override-lookups/override-lookups.mdl b/tests/integration/override-lookups/override-lookups.mdl new file mode 100644 index 00000000..aeaee91f --- /dev/null +++ b/tests/integration/override-lookups/override-lookups.mdl @@ -0,0 +1,44 @@ +{UTF-8} + +DimA: A1, A2 ~~| +DimB: B1, B2, B3 ~~| + +X = 0 + ~ dmnl [-10,10,0.1] + ~ This is an input variable. + | + +A data[DimA] + ~ dmnl + ~ This is a 1D subscripted data variable. + | + +A[DimA] = X + A data[DimA] + ~ dmnl + ~ This is a 1D subscripted output variable. + | + +B data[DimA, DimB] + ~ dmnl + ~ This is a 2D subscripted data variable. + | + +B[DimA, DimB] = X + B data[DimA, DimB] + ~ dmnl + ~ This is a 2D subscripted output variable. + | + +C data + ~ dmnl + ~ This is a non-subscripted data variable. + | + +C = X + C data + ~ dmnl + ~ This is a non-subscripted output variable. + | + +INITIAL TIME = 2000 ~~| +FINAL TIME = 2002 ~~| +TIME STEP = 1 ~~| +SAVEPER = TIME STEP ~~| diff --git a/tests/integration/override-lookups/package.json b/tests/integration/override-lookups/package.json new file mode 100644 index 00000000..3e484eca --- /dev/null +++ b/tests/integration/override-lookups/package.json @@ -0,0 +1,23 @@ +{ + "name": "override-lookups", + "version": "1.0.0", + "private": true, + "type": "module", + "scripts": { + "clean": "rm -rf sde-prep", + "build-js": "GEN_FORMAT=js sde bundle", + "build-wasm": "GEN_FORMAT=c sde bundle", + "run-tests": "./run-tests.js", + "test-js": "run-s build-js run-tests", + "test-wasm": "run-s build-wasm run-tests", + "ci:int-test": "run-s clean test-js clean test-wasm" + }, + "dependencies": { + "@sdeverywhere/build": "workspace:*", + "@sdeverywhere/cli": "workspace:*", + "@sdeverywhere/plugin-wasm": "workspace:*", + "@sdeverywhere/plugin-worker": "workspace:*", + "@sdeverywhere/runtime": "workspace:*", + "@sdeverywhere/runtime-async": "workspace:*" + } +} diff --git a/tests/integration/override-lookups/run-tests.js b/tests/integration/override-lookups/run-tests.js new file mode 100755 index 00000000..7bd2717e --- /dev/null +++ b/tests/integration/override-lookups/run-tests.js @@ -0,0 +1,154 @@ +#!/usr/bin/env node + +import { readFile } from 'fs/promises' +import { join as joinPath } from 'path' + +import { createInputValue, createLookupDef, createSynchronousModelRunner, ModelListing } from '@sdeverywhere/runtime' +import { spawnAsyncModelRunner } from '@sdeverywhere/runtime-async' + +import loadGeneratedModel from './sde-prep/generated-model.js' + +/* + * This is a JS-level integration test that verifies that both the synchronous + * and asynchronous `ModelRunner` implementations work when overriding lookup + * data at runtime. + */ + +function verify(runnerKind, run, outputs, inputX, varId, checkValue) { + const varName = varId.replaceAll('_', '') + const series = outputs.getSeriesForVar(varId) + if (series === undefined) { + console.error(`Test failed for ${runnerKind} runner: no outputs found for ${varName}\n`) + process.exit(1) + } + for (let time = 2000; time <= 2002; time++) { + const actual = series.getValueAtTime(time) + const expected = checkValue(time, inputX) + if (actual !== expected) { + console.error( + `Test failed for ${runnerKind} runner for run=${run} at time=${time}: expected ${varName}=${expected}, got ${varName}=${actual}\n` + ) + process.exit(1) + } + } +} + +function verifyDeclaredOutputs(runnerKind, run, outputs, inputX, dataOffset) { + const expect = offset => (time, inputX) => inputX + (time - 2000 + 1) * 100 + offset + if (run === 4) { + verify(runnerKind, run, outputs, inputX, '_a[_a1]', () => undefined) + } else { + verify(runnerKind, run, outputs, inputX, '_a[_a1]', expect(0 + dataOffset)) + } + verify(runnerKind, run, outputs, inputX, '_a[_a2]', expect(1)) + verify(runnerKind, run, outputs, inputX, '_b[_a1,_b1]', expect(2)) + verify(runnerKind, run, outputs, inputX, '_b[_a1,_b2]', expect(3)) + verify(runnerKind, run, outputs, inputX, '_b[_a1,_b3]', expect(4)) + verify(runnerKind, run, outputs, inputX, '_b[_a2,_b1]', expect(5 + dataOffset)) + verify(runnerKind, run, outputs, inputX, '_b[_a2,_b2]', expect(6)) + verify(runnerKind, run, outputs, inputX, '_b[_a2,_b3]', expect(7)) + verify(runnerKind, run, outputs, inputX, '_c', expect(10)) +} + +async function runTests(runnerKind, modelRunner) { + // Read the JSON model listing + const listingJson = await readFile(joinPath('sde-prep', 'build', 'processed.json'), 'utf8') + const listing = new ModelListing(listingJson) + + // Create the set of inputs + const inputX = createInputValue('_x', 0) + const inputs = [inputX] + + // Create the buffer to hold the outputs + let outputs = modelRunner.createOutputs() + + // Run the model with input at default (0) + outputs = await modelRunner.runModel(inputs, outputs) + + // Verify declared output variables + verifyDeclaredOutputs(runnerKind, 1, outputs, 0, 0) + + // Run the model with data overrides for a couple variables + const p = (x, y) => ({ x, y }) + outputs = await modelRunner.runModel(inputs, outputs, { + lookups: [ + createLookupDef(listing.varSpecs.get('_a_data[_a1]'), [p(2000, 160), p(2001, 260), p(2002, 360)]), + createLookupDef(listing.varSpecs.get('_b_data[_a2,_b1]'), [p(2000, 165), p(2001, 265), p(2002, 365)]) + ] + }) + + // Verify that the data overrides are reflected in the outputs + verifyDeclaredOutputs(runnerKind, 2, outputs, 0, 60) + + // Run the model again, but without specifying data overrides + outputs = await modelRunner.runModel(inputs, outputs) + + // Verify that the data overrides are still in effect + verifyDeclaredOutputs(runnerKind, 3, outputs, 0, 60) + + // Run the model with empty data override for one variable + outputs = await modelRunner.runModel(inputs, outputs, { + lookups: [createLookupDef(listing.varSpecs.get('_a_data[_a1]'), [])] + }) + + // Verify that the empty data override is in effect + verifyDeclaredOutputs(runnerKind, 4, outputs, 0, 60) + + // Terminate the model runner + await modelRunner.terminate() +} + +async function createSynchronousRunner() { + // TODO: This test app is using ESM-style modules, and `__dirname` is not defined + // in an ESM context. The `generated-model.js` file (if it contains a Wasm model) + // may contain a reference to `__dirname`, so we need to define it here. We should + // fix the generated Wasm file so that it works for either ESM or CommonJS. + global.__dirname = '.' + + // Load the generated model and verify that it exposes `outputVarIds` + const generatedModel = await loadGeneratedModel() + const actualVarIds = generatedModel.outputVarIds || [] + const expectedVarIds = [ + '_a[_a1]', + '_a[_a2]', + '_b[_a1,_b1]', + '_b[_a1,_b2]', + '_b[_a1,_b3]', + '_b[_a2,_b1]', + '_b[_a2,_b2]', + '_b[_a2,_b3]', + '_c' + ] + if (actualVarIds.length !== expectedVarIds.length || !actualVarIds.every((v, i) => v === expectedVarIds[i])) { + const expected = JSON.stringify(expectedVarIds, null, 2) + const actual = JSON.stringify(actualVarIds, null, 2) + throw new Error( + `Test failed: outputVarIds in generated JS model don't match expected values\nexpected=${expected}\nactual=${actual}` + ) + } + + // Initialize the synchronous `ModelRunner` that drives the model + return createSynchronousModelRunner(generatedModel) +} + +async function createAsynchronousRunner() { + // Initialize the aynchronous `ModelRunner` that drives the generated model + const modelWorkerJs = await readFile(joinPath('sde-prep', 'worker.js'), 'utf8') + return await spawnAsyncModelRunner({ source: modelWorkerJs }) +} + +async function main() { + // TODO: Verify JSON + + // Verify with the synchronous model runner + const syncRunner = await createSynchronousRunner() + await runTests('synchronous', syncRunner) + + // Verify with the asynchronous model runner + const asyncRunner = await createAsynchronousRunner() + await runTests('asynchronous', asyncRunner) + + console.log('Tests passed!\n') +} + +main() diff --git a/tests/integration/override-lookups/sde.config.js b/tests/integration/override-lookups/sde.config.js new file mode 100644 index 00000000..f4a2243a --- /dev/null +++ b/tests/integration/override-lookups/sde.config.js @@ -0,0 +1,30 @@ +import { wasmPlugin } from '@sdeverywhere/plugin-wasm' +import { workerPlugin } from '@sdeverywhere/plugin-worker' + +const genFormat = process.env.GEN_FORMAT === 'c' ? 'c' : 'js' + +const outputVarNames = ['A[A1]', 'A[A2]', 'B[A1,B1]', 'B[A1,B2]', 'B[A1,B3]', 'B[A2,B1]', 'B[A2,B2]', 'B[A2,B3]', 'C'] + +export async function config() { + return { + genFormat, + modelFiles: ['override-lookups.mdl'], + modelInputPaths: ['*.dat'], + modelSpec: async () => { + return { + inputs: [{ varName: 'X', defaultValue: 0, minValue: -10, maxValue: 10 }], + outputs: outputVarNames.map(varName => ({ varName })), + datFiles: ['../override-lookups.dat'] + } + }, + + plugins: [ + // If targeting WebAssembly, generate a `generated-model.js` file + // containing the Wasm model + genFormat === 'c' && wasmPlugin(), + + // Generate a `worker.js` file that runs the generated model in a worker + workerPlugin() + ] + } +}