Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrating formula node deserialization to code block nodes #14196

Merged
merged 17 commits into from
Aug 7, 2023

Conversation

aparajit-pratap
Copy link
Contributor

@aparajit-pratap aparajit-pratap commented Jul 25, 2023

Purpose

Attempt at migrating formula node deserialization to code block nodes.

The regex expressions used so far do not support arbitrarily nested expressions just yet, for example, expressions like
if(sin(90+10) > log10(100), 45, tan(90)), are not translated properly into DS for CBNs.

Auto-migration of Formula node to Codeblock node during workspace deserialization. In cases where the auto-conversion, fails, the unchanged formula code is copied to a CBN with an appropriate warning. Even in cases where there is a successful conversion to a codeblock node, a warning is issued on the converted node indicating that it has been created by migrating to a Formula node.

This PR removes the dependency on ncalc and deprecates both Formula nodemodel node as well as EvaluateFormula ZT node.

TODO:

  • Unit tests to test migration
  • Cleanup (Formula.cs file needs to be deleted, not just removed)
  • One test has been commented out, it needs to be updated
  • Fix any test failures in build job

Failed migration case:
image

Declarations

Check these if you believe they are true

  • The codebase is in a better state after this PR
  • Is documented according to the standards
  • The level of testing this PR includes is appropriate
  • User facing strings, if any, are extracted into *.resx files
  • All tests pass using the self-service CI.
  • Snapshot of UI changes, if any.
  • Changes to the API follow Semantic Versioning and are documented in the API Changes document.
  • This PR modifies some build requirements and the readme is updated
  • This PR contains no files larger than 50 MB

Release Notes

(FILL ME IN) Brief description of the fix / enhancement. Mandatory section

Reviewers

(FILL ME IN) Reviewer 1 (If possible, assign the Reviewer for the PR)

(FILL ME IN, optional) Any additional notes to reviewers or testers.

FYIs

(FILL ME IN, Optional) Names of anyone else you wish to be notified of

@aparajit-pratap aparajit-pratap marked this pull request as draft July 25, 2023 15:49
@aparajit-pratap aparajit-pratap marked this pull request as ready for review July 28, 2023 13:57
@aparajit-pratap aparajit-pratap changed the title [WIP] Attempt at migrating formula node deserialization to code block nodes Migrating formula node deserialization to code block nodes Jul 28, 2023
@aparajit-pratap aparajit-pratap changed the title Migrating formula node deserialization to code block nodes [WIP] Migrating formula node deserialization to code block nodes Jul 28, 2023
Copy link
Member

@mjkkirschner mjkkirschner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking pretty good, a few questions and an interesting failure in the build - I guess NCalc brought in a dependency we didn't know we had...

Net6 has transitive dependencies active by default. Thats what I'm assuming is going on.

src/DynamoCore/Engine/MigrateFormulaToDS.cs Show resolved Hide resolved
src/DynamoCore/Graph/Workspaces/SerializationConverters.cs Outdated Show resolved Hide resolved
src/DynamoCore/Graph/Workspaces/SerializationConverters.cs Outdated Show resolved Hide resolved
src/DynamoCore/Graph/Nodes/CodeBlockNode.cs Outdated Show resolved Hide resolved
src/DynamoCore/Graph/Nodes/CodeBlockNode.cs Outdated Show resolved Hide resolved
internal string ConvertFormulaToDS(string formula)
{
CodeBlockNode cbn = null;
var match = ifRgx.Match(formula);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it simpler to use the DS parser to check for conditional statements - or theres a high chance of failure trying to parse the entire formula string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This syntax for the if statement will fail to parse using the DS parser, which is why I'm splitting it up using regex into parts that can be sent individually to the DS parser.

Copy link
Contributor Author

@aparajit-pratap aparajit-pratap Jul 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the parser fails, it will throw an exception and the entire deserialization of the formula node will be skipped, which is undesirable. We want to convert it to DS as much as possible, if not keep the node and issue an appropriate warning to the user saying they need to manually convert it. In this case, I'm just copying the original string in the formula node and pasting it in a code block node with a warning to that effect.

@github-actions
Copy link

github-actions bot commented Jul 31, 2023

":white_check_mark: Bin-Diff Issue Resolved."
(Updated: 2023-07-31-18:28:01)

@aparajit-pratap aparajit-pratap changed the title [WIP] Migrating formula node deserialization to code block nodes Migrating formula node deserialization to code block nodes Jul 31, 2023
inputAstNodes))
};
newNode.Attributes["CodeText"].Value = convertedCode;
//(node as CodeBlockNodeModel).FormulaMigrationWarning(Resources.FormulaMigrated);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this not accessible from here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, over here I don't have a handle to the Nodemodel, just the xml node. Trying to figure this one out as the last thing to do here.

Copy link
Member

@mjkkirschner mjkkirschner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with one question - and consider holding off until after 2.19 cut.

@aparajit-pratap
Copy link
Contributor Author

4 failures in DynamoAnalyticsTests and 1 with PreferenceSettingsTests that are unrelated to this PR, they also occur in master currently. Merging.

@aparajit-pratap aparajit-pratap merged commit 130d3c3 into DynamoDS:master Aug 7, 2023
@aparajit-pratap aparajit-pratap deleted the dyn-6018 branch August 7, 2023 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants