Python template support 2.0

[radumg]

Note : this PR replaces the previous PR #8034 as that was based on 1.3.1 branch and not on master. Re-basing was messy, so cherry-picked to this new PR instead.

Purpose

This PR fixes #7604 , wish for adding a Python template that can be used to populate any Python Script nodes when adding to workspace.

Current behaviour

• user adds Python Script node to canvas
• the node is populated with a few lines of code, but these are hard-coded in the Dynamo source code, so users are unable to change them.

New behaviour

• user adds Python Script node to canvas
• Dynamo checks if the PythonTemplate.py file exists at the user location root (%appdata%/Dynamo/Core/{version}/) and if it does, it reads the file and populates the Python script node with its contents
• if the file doesn't exits or it's empty, Dynamo falls back on the hard-coded behaviour

Behaviour when template file is missing or empty :

Behaviour when template file is found :

This allows users and organisations to rollout the template as part of users profiles or as logon scripts.

Implementation

• refactored PreferenceSettings class to include code regions, moving properties into their logical group, also fixing some comments/documentation at same time, making it clearer.
• implemented a PythonTemplateFilePath property in the settings class so it serialises and deserialises with user settings
• implemented required properties in the interfaces IPreferences, IPathResolver and PathManager class
• implemented a static backing store called pythonTemplateFilePath. This is what the property above (part of interface) gets and sets. This was necessary as static props cannot be part of the interface.
• exposed the private static property with a public static method called GetPythonTemplateFilePath, so it's accessible by nodes and anything else running in Dynamo context, such as ZT nodes.
• added a few additional lines to the hardcoded python template, providing pointers to users where to start writing code and explaining the first few lines (clr imports, etc). This is following my observations in-person and on forums that most new users struggle with those 2 things : where to put code and what those bits mean.
• replaced \n in hardcoded python code template with .NET standard Environment.NewLine line breaks, should behave more predictably when compiled on different platforms
• replaced hardcoded comments in python code with texts extracted into .resx variables, to add support for localisation of code comments too.
• got rid of un-necessary initialisation of new PreferenceSettings or PathManager classes as per @mjkkirschner 's review. What happens now is that everytime a new DynamoModel instance is created, the backing static property is overwritten.

Limitations

• the name and location of template file is hardcoded. however, this is consistent/same as DynamoSettings.XML file itself. Since the interfaces implement the property, it can be exposed via UI in later PR (working on this already but I'm not great with WPF, could use a helping hand)
  
• I couldn't figure out how to access the current DynamoModel instance from the PythonNodeModel class to retrieve current settings, hence the static property, as discussed in previous PR Python template support #8034
• the template file is read every time a Python Script node is added to canvas, there's no caching - can add it if required ?
• As a bonus though, the above allows for quick iterations of the template in external editor :)
• the public static method technically duplicates the private static property, but I felt this was cleaner than exposing 2 properties with almost same name with just different capitalisation and one being static

Declarations

• The code base is in a better state after this PR
• Is documented according to the standards
• The level of testing this PR includes is appropriate
• User facing strings, if any, are extracted into *.resx files
• All tests pass using the self-service CI.
• Snapshot of UI changes, if any.
• Changes to the API follow Semantic Versioning, and are documented in the API Changes document.
• Based on master branch with 2.0 reported version

Reviewers

@kronz
@mjkkirschner
@smangarole
@QilongTang

FYIs

@ksobon , @ThomasMahon, @wynged , @eibre , @andydandy74

[radumg]

FYI @mjkkirschner , checked for dependency on changed DynamoCore interfaces in the DynamoRevit repo and they should be fine.
The file you referenced depends on IPathResolver which was not changed, only IPathManager was updated.

[mjkkirschner]

I'm not sure, but I believe there is a reason we don't have this as part of the gitignore, you can simply not commit changes to this file until we dig deeper.

[radumg]

Thanks @mjkkirschner , looked around and it was already part of the .gitignore, see a few lines above - my GitHub desktop app added it again when i ignored it. I untracked the AssemblySharedInfo.cs file in my local repo now, just need to do the same for this .gitignore file too.

[radumg]

fixed it now 👍

[mjkkirschner]

@radumg thanks for looking into how to avoid the creation of a new pathManager, but this smells funny to me, and it is a bit inconsistent with other preferences which are similar, like the customPackageFolders paths.

I'm thinking we could follow a similar pattern like here:

Dynamo/src/DynamoCore/Models/DynamoModel.cs

Line 583 in 7304ba1

if (PreferenceSettings.CustomPackageFolders.Count == 0)

so we would assign the default path to "" in the preferences constructor and instead set it to path specified in the pathManager during DynamoModel creation time.

Let me know what you think of this - both seem okay on their own, but this seems a more consistent solution and we can avoid the static method on pathManager

[radumg]

Thanks @mjkkirschner - think your suggestion is good, it's indeed inconsistent with rest of props. I'll go ahead and implement the above, was just trying to avoid modifying the DynamoModel class/ctor as per our previous conversations.

[radumg]

Ok, I've implemented this in c4bff15
I added some logic so it follows this flow :

• tries to use template file specified in DynamoSettings.xml first
• if that fails, it looks for the default PythonTemplate.py in the user folder. If this is found, the setting will be updated to point to it.
• if both those fail, it will fall back on the hardcoded string.

Also added logging so if users are having a hard time getting their templates to be picked up (most likely due to invalid paths), they would have some indication of what's going on.

Let me know if this is what you had in mind and if there's any changes you'd like.

[mjkkirschner]

since this is the public entry point for most use cases lets add some description of what this file at this path is used for.

[radumg]

Good point, wasn't being descriptive enough 👍 . Added now, have a look please 👁

[mjkkirschner]

is this reference still needed for something?

[radumg]

nope, forgot to remove it - I had a MessageBox.Show() during testing. Will keep an eye out for these going forward.
It's now removed.

[mjkkirschner]

@radumg thanks for the work here. I notice there are no new tests. Would it be possible to add a test for this preference, like:

https://github.com/DynamoDS/Dynamo/blob/dec6240ded0c4369617775336b9af60c2aba4103/test/DynamoCoreTests/Settings.cs

which just asserts the deserialization is working correctly

and then something which modifies the template then asserts the python script changes? A potential test location is here:

https://github.com/DynamoDS/Dynamo/blob/master/test/Libraries/DynamoPythonTests/PythonEditTests.cs

[radumg]

Thanks @mjkkirschner for the review. I thought about potential tests for a second but couldn't imagine any useful ones - obvs my imagination failed me 😁 - I can certainly try & add the tests you suggest above.

I'm fairly new to TDD so I really appreciate the guidance!

[radumg]

@mjkkirschner - if you happen to see this before i start working on it again tonight, i'm having a hard time adding any Dynamo node to the Workspace for this last test. Tried using the same base class and Workspace.AddAndRegisterNode method as here but that method is not accessible from the IronPythonTests assembly. Can't find a replacement mechanism to add a node to current workspace from this context.

Just thought I'd check before I delve deeper, pretty sure I'm missing an obvious trick here 🤓

[mjkkirschner]

@radumg what you can do is use a createNode command and execute it on the dynamoModel

Dynamo/test/DynamoCoreTests/NodeExecutionTest.cs

Line 58 in dec6240

model.ExecuteCommand(new DynamoModel.CreateNodeCommand(numberNode1, 0, 0, true, false));

Another possibility is to make the python test assembly able to access internal members of other assemblies with
https://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.internalsvisibletoattribute(v=vs.110).aspx - we use this for tests though it's not ideal.

[radumg]

Ok, I think I've figured out these tests 🎉 & 🤞

I've added 3 tests in total @mjkkirschner (screenshot only shows 2) :

Test 1 : LoadInvalidPythonTemplateFromSetting in test/DynamoCoreTests/Settings.cs#39

• loads the settings from XML
• checks the settings were deserialised correctly
• checks the file with invalid path doesn't exist

Test 2 : CanReadPythonTemplateSetting in test/DynamoCoreTests/CoreTests.cs#55

• same as test 1 but in this different execution context

Test 3 : CanUpdatePythonTemplateSettings in test/DynamoCoreTests/CoreTests.cs#77

• loads the settings from XML
• checks the settings were deserialised correctly
• updates the DynamoModel with settings
• creates & adds a PythonNode to workspace
• checks added node has been initialised with the custom initial Python template
• changes the Python template
• reloads the settings from the updated XML & checks the settings were deserialised correctly again
• updates the DynamoModel with settings again
• creates & adds another PythonNode to workspace
• checks added node has been initialised with the custom changed Python template

It's ready for review again @mjkkirschner , thanks so much for the pointers !
Let me know if there's any more changes needed.

[mjkkirschner]

@radumg this looks solid to me, will take another look and bring it in soon.

[mjkkirschner]

@QilongTang tests pass- going to merge this in, lets watch the revit build.

[radumg]

Awesome, just wanted to say thanks again @mjkkirschner for the guidance along the way!
On to DYN template support now... 🤓

[eibre]

Thank you so much for this @radumg , it's working great.

[MarkAckerley]

Thanks @radumg it's working on a network path :)

Just an FYI, when I follow the instruction in the primer, if I reopen the DynamoSettings.xml it has changed:

<PythonTemplateFilePath>
<string>Z:\DYNAMO\PythonTemplate.py</string>
</PythonTemplateFilePath>

to:

  <CustomPackageFolders>
    <string>Z:\DYNAMO\PythonTemplate.py</string>
    <string>C:\Users\mka\AppData\Roaming\Dynamo\Dynamo Revit\2.0</string>
  </CustomPackageFolders>
  <PackageDirectoriesToUninstall />
  <PythonTemplateFilePath>Z:\DYNAMO\PythonTemplate.py</PythonTemplateFilePath>
  <BackupInterval>60000</BackupInterval>

Thanks again,

Mark