feat: allow for overriding data variables and lookups at runtime (#490)

Fixes #472

README.md

@@ -4,7 +4,7 @@
 SDEverywhere allows for unit testing, continuous integration, the ability to compare model runs, and a toolchain to transform a model into C, JavaScript, and WebAssembly code.
 
 Using SDEverywhere, you can deploy interactive System Dynamics models on mobile, desktop, and the web for policymakers and the public.
-Or you could perform model analysis using general-purpose languages, running the model as high-performance C code.
+Or you could perform model analysis using general-purpose languages, running the model as high-performance JavaScript or C code.
 
 By following the ["Quick Start"](#quick-start) instructions below, within minutes you can turn a Vensim model like this:
 
@@ -20,7 +20,7 @@ By following the ["Quick Start"](#quick-start) instructions below, within minute
 ## Supported Platforms
 
 The SDEverywhere tools can be used on on any computer running macOS, Windows, or Linux.
-The libraries and code generated by SDEverywhere can be run in web browsers on almost any device (including phones and tablets) as well as on the server side (using C/C++ or WebAssembly).
+The libraries and code generated by SDEverywhere can be run in web browsers on almost any device (including phones and tablets) as well as on the server side (using C/C++, JavaScript, and/or WebAssembly).
 
 ## Supported File Formats
 
@@ -46,7 +46,7 @@ https://github.com/climateinteractive/SDEverywhere/assets/438425/cd495776-5744-4
 
 ### Quality Checks and Comparisons
 
-SDEverywhere includes extensive QA (quality assurance) tools that you can run locally on your machine as you develop your model or in the cloud in a continuous integration environment.
+SDEverywhere includes extensive QA (quality assurance) tools that you can run as you develop your model, either locally on your machine or in the cloud in a continuous integration environment (or both).
 There is support for two different kinds of tests:
 
 - **_Checks:_** Check tests can be used to verify that your model behaves according to some guidelines (or conforms to your mental models). For example, you can have checks that verify that stocks are always positive, or that some output variable produces values that are close to historical reference data.
@@ -63,17 +63,17 @@ https://github.com/climateinteractive/SDEverywhere/assets/438425/b6f05b3f-f18a-4
 
 ## Core Functionality
 
-At its core, SDEverywhere includes a [transpiler](https://en.wikipedia.org/wiki/Source-to-source_compiler) that can read a [Vensim](https://www.vensim.com/documentation/index.html?ref_language.htm) model and generate a high-performance version of that model in the C programming language.
+At its core, SDEverywhere includes a [transpiler](https://en.wikipedia.org/wiki/Source-to-source_compiler) that can read a [Vensim](https://www.vensim.com/documentation/index.html?ref_language.htm) model and generate a high-performance version of that model as JavaScript or C code.
 
-The [`sde`](./packages/cli) command line tool — in addition to generating C code — provides a plugin-based build system for extended functionality:
+The [`sde`](./packages/cli) command line tool — in addition to generating JavaScript or C code — provides a plugin-based build system for extended functionality:
 
 - [plugin-config](./packages/plugin-config) allows you to configure your model/library/app using CSV files
-- [plugin-wasm](./packages/plugin-wasm) converts the generated C model into a fast WebAssembly module for use in a web browser or Node.js application
-- [plugin-worker](./packages/plugin-worker) generates a Web Worker that can be used to run the WebAssembly model in a separate thread for improved user experience
+- [plugin-wasm](./packages/plugin-wasm) converts a generated C model into a fast WebAssembly module for use in a web browser or Node.js application
+- [plugin-worker](./packages/plugin-worker) generates a worker module that can be used to run your model in a separate thread for improved user experience
 - [plugin-vite](./packages/plugin-vite) provides integration with [Vite](https://github.com/vitejs/vite) for developing a library or application around your generated model
 - [plugin-check](./packages/plugin-check) allows for running QA checks and comparison tests on your model every time you make a change
 
-Additionally, the [runtime](./packages/runtime) and [runtime-async](./packages/runtime-async) packages make it easy to interact with your generated WebAssembly model in a JavaScript- or TypeScript-based application.
+Additionally, the [runtime](./packages/runtime) and [runtime-async](./packages/runtime-async) packages make it easy to interact with your generated model in a JavaScript- or TypeScript-based application.
 
 For more details on all of these packages, refer to the ["Packages"](#packages) section below.
 
@@ -296,6 +296,15 @@ Most users won't need to interact with these implementation packages directly, b
       <a href="./packages/compile/CHANGELOG.md">Changelog</a>
     </td>
   </tr>
+  <tr>
+    <td><a href="./packages/parse">@sdeverywhere/parse</a> *</td>
+    <td><a href="https://www.npmjs.com/package/@sdeverywhere/parse"><img style="vertical-align:middle;" src="https://img.shields.io/npm/v/@sdeverywhere/parse.svg?label=%20"></a></td>
+    <td>
+      <a href="./packages/parse">Source</a>&nbsp;|&nbsp;
+      <a href="./packages/parse/README.md">Docs</a>&nbsp;|&nbsp;
+      <a href="./packages/parse/CHANGELOG.md">Changelog</a>
+    </td>
+  </tr>
   <tr>
     <td colspan="3"><em>`model-check` implementation</em></td>
   </tr>
@@ -328,7 +337,7 @@ There is still much to contribute, for example:
 - Enhance the C code generator to produce code for new language features now that you can parse them.
 - Implement more Vensim functions. This is the easiest way to help out.
 - Add support for the XMILE file format.
-- Target languages other than C, such as R or Rust. (If you want Python, check out the excellent [PySD](https://github.com/JamesPHoughton/pysd)).
+- Target languages other than JavaScript and C, such as R or Rust. (If you want Python, check out the excellent [PySD](https://github.com/JamesPHoughton/pysd)).
 
 For more guidance on contributing to SDEverywhere, please consult the [wiki](https://github.com/climateinteractive/SDEverywhere/wiki).
 

packages/cli/src/c/vensim.c

@@ -94,7 +94,7 @@ double __lookup(Lookup* lookup, double input, bool use_inverted_data, LookupMode
   // Interpolate the y value from an array of (x,y) pairs.
   // NOTE: The x values are assumed to be monotonically increasing.
 
-  if (lookup == NULL) {
+  if (lookup == NULL || lookup->n == 0) {
     return _NA_;
   }
 
@@ -162,9 +162,16 @@ double __lookup(Lookup* lookup, double input, bool use_inverted_data, LookupMode
 // This function is similar to `__lookup` in concept, but Vensim produces results for
 // the GET DATA BETWEEN TIMES function that differ in unexpected ways from normal lookup
 // behavior, so we implement it as a separate function here.
-double __get_data_between_times(double* data, size_t n, double input, LookupMode mode) {
+double __get_data_between_times(Lookup* lookup, double input, LookupMode mode) {
   // Interpolate the y value from an array of (x,y) pairs.
   // NOTE: The x values are assumed to be monotonically increasing.
+
+  if (lookup == NULL || lookup->n == 0) {
+    return _NA_;
+  }
+
+  const double* data = lookup->data;
+  const size_t n = lookup->n;
   const size_t max = n * 2;
 
   switch (mode) {

packages/cli/src/c/vensim.h

@@ -74,9 +74,9 @@ double __lookup(Lookup *lookup, double input, bool use_inverted_data, LookupMode
 #define _WITH_LOOKUP(x, lookup) __lookup(lookup, x, false, Interpolate)
 double _LOOKUP_INVERT(Lookup* lookup, double y);
 
-double __get_data_between_times(double *data, size_t n, double input, LookupMode mode);
+double __get_data_between_times(Lookup* lookup, double input, LookupMode mode);
 #define _GET_DATA_MODE_TO_LOOKUP_MODE(mode) ((mode) >= 1) ? Forward : (((mode) <= -1) ? Backward : Interpolate)
-#define _GET_DATA_BETWEEN_TIMES(lookup, x, mode) __get_data_between_times((lookup)->data, (lookup)->n, x, _GET_DATA_MODE_TO_LOOKUP_MODE(mode))
+#define _GET_DATA_BETWEEN_TIMES(lookup, x, mode) __get_data_between_times(lookup, x, _GET_DATA_MODE_TO_LOOKUP_MODE(mode))
 
 //
 // DELAY FIXED

packages/compile/src/generate/gen-code-c.js

@@ -81,51 +81,54 @@ ${section(Model.dataVars())}
   //
   function emitInitLookupsCode() {
     mode = 'init-lookups'
-    let code = `// Internal state
+    let code = `\
+// Internal state
 bool lookups_initialized = false;
 bool data_initialized = false;
+
 `
     code += chunkedFunctions(
       'initLookups',
       Model.lookupVars(),
-      `  // Initialize lookups.
-  if (!lookups_initialized) {
-`,
-      `      lookups_initialized = true;
-  }
-`
+      `\
+  // Initialize lookups.
+  if (lookups_initialized) {
+    return;
+  }`,
+      '  lookups_initialized = true;'
     )
     code += chunkedFunctions(
       'initData',
       Model.dataVars(),
-      `  // Initialize data.
-  if (!data_initialized) {
-`,
-      `      data_initialized = true;
-  }
-`
+      `\
+  // Initialize data.
+  if (data_initialized) {
+    return;
+  }`,
+      '  data_initialized = true;'
     )
     return code
   }
 
   function emitInitConstantsCode() {
     mode = 'init-constants'
-    return `
-${chunkedFunctions('initConstants', Model.constVars(), '  // Initialize constants.', '  initLookups();\n  initData();')}
-`
+    return chunkedFunctions(
+      'initConstants',
+      Model.constVars(),
+      '  // Initialize constants.',
+      '  initLookups();\n  initData();'
+    )
   }
 
   function emitInitLevelsCode() {
     mode = 'init-levels'
-    return `
-${chunkedFunctions(
-  'initLevels',
-  Model.initVars(),
-  `
+    return chunkedFunctions(
+      'initLevels',
+      Model.initVars(),
+      `\
   // Initialize variables with initialization values, such as levels, and the variables they depend on.
   _time = _initial_time;`
-)}
-`
+    )
   }
 
   //
@@ -134,11 +137,9 @@ ${chunkedFunctions(
   function emitEvalCode() {
     mode = 'eval'
 
-    return `
-${chunkedFunctions('evalAux', Model.auxVars(), '  // Evaluate auxiliaries in order from the bottom up.')}
-
-${chunkedFunctions('evalLevels', Model.levelVars(), '  // Evaluate levels.')}
-`
+    return `\
+${chunkedFunctions('evalAux', Model.auxVars(), '  // Evaluate auxiliaries in order from the bottom up.')}\
+${chunkedFunctions('evalLevels', Model.levelVars(), '  // Evaluate levels.')}`
   }
 
   //
@@ -148,9 +149,35 @@ ${chunkedFunctions('evalLevels', Model.levelVars(), '  // Evaluate levels.')}
     let headerVarNames = outputAllVars ? expandedVarNames(true) : spec.outputVarNames
     let outputVarIds = outputAllVars ? expandedVarNames() : spec.outputVars
     mode = 'io'
-    return `void setInputs(const char* inputData) {${inputsFromStringImpl()}}
+    return `\
+void setInputs(const char* inputData) {
+${inputsFromStringImpl()}
+}
 
-void setInputsFromBuffer(double* inputData) {${inputsFromBufferImpl()}}
+void setInputsFromBuffer(double* inputData) {
+${inputsFromBufferImpl()}
+}
+
+void replaceLookup(Lookup** lookup, double* points, size_t numPoints) {
+  if (lookup == NULL) {
+    return;
+  }
+  if (*lookup != NULL) {
+    __delete_lookup(*lookup);
+    *lookup = NULL;
+  }
+  if (points != NULL) {
+    *lookup = __new_lookup(numPoints, /*copy=*/true, points);
+  }
+}
+
+void setLookup(size_t varIndex, size_t* subIndices, double* points, size_t numPoints) {
+  switch (varIndex) {
+${setLookupImpl(Model.varIndexInfo())}
+    default:
+      break;
+  }
+}
 
 const char* getHeader() {
   return "${R.map(varName => varName.replace(/"/g, '\\"'), headerVarNames).join('\\t')}";
@@ -178,9 +205,9 @@ ${fullOutputSection(Model.varIndexInfo())}
   function chunkedFunctions(name, vars, preStep, postStep) {
     // Emit one function for each chunk
     let func = (chunk, idx) => {
-      return `
+      return `\
 void ${name}${idx}() {
-  ${section(chunk)}
+${section(chunk)}
 }
 `
     }
@@ -208,22 +235,25 @@ void ${name}${idx}() {
       chunks = [vars]
     }
 
-    if (!preStep) {
-      preStep = ''
+    let funcsPart = funcs(chunks)
+    let callsPart = funcCalls(chunks)
+
+    let out = ''
+    if (funcsPart.length > 0) {
+      out += funcsPart + '\n'
     }
-    if (!postStep) {
-      postStep = ''
+    out += `void ${name}() {\n`
+    if (preStep) {
+      out += preStep + '\n'
     }
-
-    return `
-${funcs(chunks)}
-
-void ${name}() {
-${preStep}
-${funcCalls(chunks)}
-${postStep}
-}
-    `
+    if (callsPart.length > 0) {
+      out += callsPart + '\n'
+    }
+    if (postStep) {
+      out += postStep + '\n'
+    }
+    out += '}\n\n'
+    return out
   }
 
   //
@@ -314,7 +344,10 @@ ${postStep}
     return section(varNames)
   }
   function fullOutputSection(varIndexInfo) {
-    // Emit output calls for all variables.
+    // Emit `storeValue` calls for all variables that can be accessed as an output.
+    // This excludes data and lookup variables; at this time, the data for these
+    // cannot be output like for other types of variables.
+    const outputVars = R.filter(info => info.varType !== 'lookup' && info.varType !== 'data')
     const code = R.map(info => {
       let varAccess = info.varName
       if (info.subscriptCount > 0) {
@@ -326,13 +359,12 @@ ${postStep}
       if (info.subscriptCount > 2) {
         varAccess += '[subIndex2]'
       }
-      let c = ''
-      c += `    case ${info.varIndex}:\n`
-      c += `      outputVar(${varAccess});\n`
-      c += `      break;`
-      return c
+      return `\
+    case ${info.varIndex}:
+      outputVar(${varAccess});
+      break;`
     })
-    const section = R.pipe(code, lines)
+    const section = R.pipe(outputVars, code, lines)
     return section(varIndexInfo)
   }
   function inputsFromStringImpl() {
@@ -341,7 +373,7 @@ ${postStep}
     let inputVars = ''
     if (spec.inputVars && spec.inputVars.length > 0) {
       let inputVarPtrs = R.reduce((a, inputVar) => R.concat(a, `    &${inputVar},\n`), '', spec.inputVars)
-      inputVars = `
+      inputVars = `\
   static double* inputVarPtrs[] = {\n${inputVarPtrs}  };
   char* inputs = (char*)inputData;
   char* token = strtok(inputs, " ");
@@ -354,21 +386,37 @@ ${postStep}
       *inputVarPtrs[modelVarIndex] = value;
     }
     token = strtok(NULL, " ");
-  }
-`
+  }`
     }
     return inputVars
   }
   function inputsFromBufferImpl() {
-    let inputVars = ''
+    let inputVars = []
     if (spec.inputVars && spec.inputVars.length > 0) {
-      inputVars += '\n'
       for (let i = 0; i < spec.inputVars.length; i++) {
         const inputVar = spec.inputVars[i]
-        inputVars += `  ${inputVar} = inputData[${i}];\n`
+        inputVars.push(`  ${inputVar} = inputData[${i}];`)
       }
     }
-    return inputVars
+    return inputVars.join('\n')
+  }
+  function setLookupImpl(varIndexInfo) {
+    // Emit `createLookup` calls for all lookups and data variables that can be overridden
+    // at runtime
+    const lookupAndDataVars = R.filter(info => info.varType === 'lookup' || info.varType === 'data')
+    const code = R.map(info => {
+      let lookupVar = info.varName
+      for (let i = 0; i < info.subscriptCount; i++) {
+        lookupVar += `[subIndices[${i}]]`
+      }
+      let c = ''
+      c += `    case ${info.varIndex}:\n`
+      c += `      replaceLookup(&${lookupVar}, points, numPoints);\n`
+      c += `      break;`
+      return c
+    })
+    const section = R.pipe(lookupAndDataVars, code, lines)
+    return section(varIndexInfo)
   }
 
   return {