Calculation Engine

Conceptual Overview

The application is built around a calculation engine - an algorithm that applies a hierarchical set of formulas to a set of input values to calculate a result. Each component of the calculation is stored in the database as a rule. Each calculation rule's value is calculated by a javascript function that depends on the value of its dependent (child) rules, so the entire graph is essentially a directed graph (tree) of calculation rules. When we get to the leaf nodes of this tree, they are simply values that have no children, and represent values that the user provides as input to the calculation. Somewhat arbitrarily, these leaf nodes are categorized as either input rules or measure (aka strategy) rules. This is specific to TDM and distinguish between project specification inputs (like Project Name, Number of square feet of retail space, etc.) and TDM Strategies that earn TDM points.

There is currently only one set of rules that comprise the official TDM calculation, though we expect in the future to need to keep different versions of the overall calculation as the rules change over time once the application is in production. This will require keeping multiple sets of rules and keeping track of which rules were in effect for which projects.

Calculation Implementation

Calculations and their rules are stored in the database. The database design allows for potentially keeping multiple different calculations, each with its own set of rules. The different calculations could be different versions for the same application, or potentially for entirely different purposes (such as calculating carbon credits or anything else that could use this type of engine). However, at this time there is only a single Calculation, represented by a single record in the Calculation database table (with calculationId = 1).

A Calculation is comprised of a set of Rules. There is one record in the Calculation table for each calculation. Each rule in a calculation is stored as a record in the CalculationRule table in the database with a foreign key (calculationId) referring to the parent calculation. The database structure allows for the possibility of having different calculations, each with its own set of rules, but there is currently only one calculation. When a new calculation is started in the client application, the rules for calculation 1 are retrieved from the database and used to initialize a corresponding javascript array as a rules data structure for the calculation engine (implemented in javascript code on the client) to manipulate.

There are many columns in the CalculationRule table (and corresponding properties in the rule objects of the rules data structure in javascript), but to understand how the calculation works, we'll start with just a few of them:

• Code (alphanumeric) - identifies a rule within a calculation. These were assigned when rules were created and should not be changed, as they allow the client application UI to display specific rules where desired, and also allow rules to be linked to other rules.
• Name (alphanumeric) - This is the name of the rule that will be shown to the user in the UI (if the rule is visible to the user at all).
• Value (alphanumeric) - In the database, this value will always be null, but when the calculation engine runs on the client, this represents either an input provided by the user (for rules with a category of input or measure rules) or the result of the calculation for rules with a category of calculation.
• Category (one of input, measure or calculation) - The rule's category. Input and Measure rules are values entered or selected by the user through the UI. Calculation rules perform a calculation one or more values from other rules as inputs. Then the rule is evaluated, the result is stored in the Value property of the rule object.
• DataType (one of number, string, boolean, or choice) - Indicates the data type of the value.
• FunctionBody (alphanumeric) - For calculation rules, this field contains the body of a javascript function to run to perform the calculation. The calculation body will include rule codes in double angle brackets that get replaced with the value of the corresponding rule to create a javascript function that is then run to perform the calculation for the rule.

You can view the rules in the application by navigating to the semi-hidden (i.e., there is no menu item to get to the page, but it is accessible to anyone) admin page at https://tdm.ladot.lacity.org/admin. The drop-down menu allows you to see the FunctionBody of any rule - a good one to start with is PTS_DIFFERENCE, which has the function body return <<PTS_EARNED>> - <<TARGET_POINTS_PARK>>; PTS_DIFFERENCE is the end result of the entire calculation - the difference between the Earned Points and Target Points (the page UI displays the placeholders as a badge, though in the database placeholders are in double angle brackets). When the calculation starts, it starts to evaluate the rule, and finds that it needs the values from two other rules, PTS_EARNED and TARGET_POINTS_PARK. So it determines that it first needs to evaluate these two rules before PTS_DIFFERENCE can be calculated, and then goes to each of these rules, in turn. The placeholders for calculation rules are shown in green on the admin page, and in yellow for input or measure/strategy rules. The calculation recursively follows dependency rules until it encounter rules that do not depend upon other calculation rules. Placeholders that are input or measure rules are values entered by users and can be replaced by the user-provided value, allowing the calculation rule to be evaluated immediately, and pass its resulting value to any parent calculations that need its value.

Calculating Applicability of a Rule

In addition to the calculation itself, there are a number of constraints about which rules are applicable under various circumstances. This is represented by the DisplayFunctionBody column in the database, which include a javascript snippet that is based on the values of other rules. This function is written to return true if a particular rule is valid based on other rule values. For example, the CAR_SHARE_PARKING input rule has a DisplayFunctionBody of return (!!<<LAND_USE_RESIDENTIAL>> || !!<<LAND_USE_HOTEL>> || !!<<LAND_USE_RETAIL>> || !!<<LAND_USE_COMMERCIAL>> || !!<<LAND_USE_WAREHOUSE>> || !!<<LAND_USE_MEDICAL>> || !!<<LAND_USE_OTHER>>) && (<<PARK_SPACES>> > 0); This is evaluated by the calculation engine after the entire calculation is performed, and returns true if any of the LAND_USE_* rules are true and the PARK_SPACES rule value is > 0. This result tells the UI whether the input or measure rule should be enabled in the UI, preventing the user from applying rules that are not allowed under specific circumstance. The most common situations for this calculation are situations where two or more rules are mutually exclusive, or where a rule is only applicable to specific land uses.

Data-Driven User Interface

Most of the other fields in each rule are related to how the rule is rendered in the User Interface. For example:

• The description column provides the HTML-formatted contents for a tooltip that applies to a rule.
• The link column provided a url if there is a related hyperlink providing more information about a rule.
• The units column is the text that should be shown as the units for a value (e.g. pts, parking spaces, sq ft).
• The calculationPanelId indicates which UI panel the rule should be rendered on,
• The displayOrder provides a column to control what order different rules are rendered within a UI panel.
• The minValue, maxValue and required fields indicate validation rules on inputs or measures that the UI should enforce.
• For rules with a dataType of "choice", the choices column is a JSON representation of an array of possible choices.

Performing Calculations and Saving as Projects

When the user opens a new project, the rules from the database are used to create an array of rules for the calculation engine to operate on. The "empty" rules structure is used to construct the Wizard UI as described in the previous section. Initially there are no values in any of the input or measure fields, but then any time the user modifies any of the inputs, the entire calculation is run, and the resulting modified rules array will contain values for the rules used in the calculation. When a project is saved, selected elements of the calculated rules array are saved in the Project table in the database, including a formInputs field that contains a JSON representation of all the input and measure rule values entered by the user.

When a user later re-opens a project, the empty rules array is again fetched from the database, and the formInputs field from the project table is used to populate the resulting javascript rules array values for those rules, the calculation is run and the resulting rules array is again displayed in the Calculation Wizard. The Projects table contains one record for each saved project, and is used to populate the My Projects Page. This table also contains meta-data about a project, such as its visiblity, status, dateCreated, dateLastModified, dateSnapshotted, dateSubmitted, author, etc.

The Calculation Wizard

The user interface for working with Projects is the Calculation Wizard. Though people commonly refer to this as the Create Project Wizard, because that is the label on the main menu item to create a new project, it is also used to view or edit existing projects and perform many other operations that depend upon the Calculation Engine.

The Wizard currently consists of as many as five steps (or pages):

• Step 1: Project Information
• Step 2: Project Specifications
• Step 3: Target Points
• Step 4: Strategies
• Step 5: Project Summary

When the Wizard is opened, it first gets the calculation rules from the database as described above, which populates the Rules array at the client. If the wizard is being opened for an existing project, the project information is also loaded from the database, and the project's previously entered input values are used to populate the corresponding rule values in the Rules array, and the calculation engine is run to re-create the entire calculation in the in-memory Rules array. Each page displays appropriate UI elements for viewing or editing specific rules.

When you save an existing project, the project record in the database is updated with the modified values. If you save a new project, a new record is created in the project table. The CRUD operations on the Project table also set the dateCreated and loginId fields when the record is first created and update the dateModified when any changes are made to the calculation. The formInputs field is updated with any modified user inputs to the calculation, and some of these rule values (i.e., name, address, description, targetPoints, earnedPoints and projectLevel) are redundantly stored as separate columns in the Project table to make them easier to query.

The Wiki pages describing each step briefly describe which rules are associated with each page.

The Project table also contains some other meta-data about the project, that are driven by executing various operations on the project:

• dateHidden is filled in with the date on which a user hides a project and re-set to null if the project is unhidden
• dateTrashed is filled in with the date on which a user "deletes" a project and re-set to null if the project is removed from the trash before it is permanantly deleted,
• dateSnapshotted is filled in with the date on which a user hides a project and re-set to null if the project is unhidden
• dateSubmitted is filled in with the date on which a user submits a project
• archivedAt is filled in with the date on which a security admin archives a project, and re-set to null if the project is un-archived
• droId contains the DRO id that an admin can set from the My Projects Page
• adminNotes contains any admin notes that an admin can editfrom the My Projects Page
• dateModifiedAdmin is the date on which the adminNotes were last modified These operations are mostly performed from the My Projects Page, though many will probably be added to the Project Summary Page of the Wizard at some point.