Creating Your First Custom Tool

This is a walkthrough of creating a custom tool for Alteryx using the Omnibus framework. The tool we are going to create is an XML Input tool along the lines of the JSON Input tool.

It based on version 0.6 of the Omnibus.Framework Nuget packages.

I generally have XML documentation comments within the code samples but this is optional, and in no way used by Alteryx.

TMD: sections

• These: sections will contain a few extra details, but feel free to skip!

Prerequisites

You will need to have a version of Visual Studio installed. Visual Studio Community edition should be perfectly fine, but screenshots in this post are from Visual Studio 2017 Professional. I have only tested with 2017, but I think it should work with 2015 as well.

You need to have Alteryx Designer installed on the same machine. The framework needs access to this to get the required DLLs. It will work with either an Admin or User install but will choose the Admin version over the User version if both are present. I do not believe you need to have a license installed for developing and building the tool in this tutorial but you won't be able to debug it.

The OmniBus Framework is built on top of .Net 4.6. As I believe this is a requirement for Alteryx (the GUI is a .Net application) if you have Alteryx installed it should be present. This is the same .Net version as 2018.3 uses.

The process was developed on Windows 10 and has dependencies on PowerShell version 3.0 or above. This was installed inside Windows 8 and can be downloaded for Windows 7 if needed. Again, I have done only very limited testing outside of Windows 10.

As the Omnibus AddIns also use this library, you need to ensure you either remove them or are running the same version of them as the Nuget packages. If not then you may find some issues with compatibility.

Getting Set Up

• Open Visual Studio

• Create a New C# Class Library Project (File => New Project... or Ctrl-Shift-N)

  • This must be a Windows Classic Desktop, not .Net Core or .Net Standard
  • You need to target at least .Net 4.6 (as this is the version used by the framework)

• Add the following NuGet Packages:

  • Omnibus.Framework.GUI
  • Omnibus.Framework
  • Omnibus.Framework.Shared This will be added when you add either of the first two

  You can either do this using the Package Manager Console or via Manage Nuget Packages:

  • Right click on the Project and go to Manage Nuget Packages
  • Search for Omnibus
  • Click the Install button next to GUI and Framework to install

• After this your project should look like this:

TMD: Package Actions

• Installation of the Omnibus.Framework.Shared will do a couple of things to your project
  • It sets the target platform to be 64-bit (x64)
  • It will locate the Alteryx install directory and add a reference to AlteryxRecordInfo.Net.dll
  • It also sets Copy Local for this DLL to be false (Alteryx fails to load the tools if the DLL is copied into the output folder)
• Installation of the Omnibus.Framework.GUI will do the following
  • It will set the debug action to start Alteryx Designer within Debug mode
  • It will add System.Drawing and System.Windows.Forms
  • It will locate the Alteryx install directory and add a reference to AlteryxGuiToolkit.dll
  • It also sets Copy Local for this DLL to be false
  • Finally, it will set up and configure Install.bat and Uninstall.bat scripts
    • These will create a tool group with the project name
    • They are set to copy to the output directory on a build
    • When you upgrade or uninstall the packages, there will be a warning message that the scripts have been modified or already exist. Unless you have also modified them, then it is fine to overwrite.

Creating The Required Classes

A custom tool within OmniBus consists of three parts:

• Configuration object
• Engine
• Plug-In (The name of this class will be used by Alteryx in the toolbar) with an associated image (optional)

As a convention, I like calling the engine class <PlugIn>Engine and the configuration class <PlugIn>Config.

Create the Configuration Class

• Remove the default Class1.cs
• Add a new class (Shift-Alt-C) with code like:

namespace OmniBus.XmlTools
{
    /// <summary>Configuration Class for <see cref="XmlInputEngine"/></summary>
    public class XmlInputConfig
    {
        /// <summary>Returns a string that represents the current object.</summary>
        /// <returns>A string that represents the current object, used as the Default Annotation for Alteryx.</returns>
        public override string ToString() => string.Empty;
    }
}

TMD: Configuration Class

This class will hold the configuration of the object. It is also responsible for creating the Default Annotation for the tool, using the ToString method of the class. To get started, we will just create a simple class returning an empty string:

Note: The default serialiser in the framework is based off the XmlSerilaiser in .Net. There is also a pretty basic prototype of one which more completely matches Alteryx formats. This will be improved in future versions of the framework, with the intention of making it easier to create HTML UI for custom tools.

Create the Engine Class

• Add another new class for the engine (the computation part)
• This needs to be derived from BaseEngine with a type argument of the Config type you created before.
• The interface Alteryx needs to use this class (AlteryxRecordInfo.Net.INetPlugIn) is implemented by the base class.

using OmniBus.Framework;

namespace OmniBus.XmlTools
{
    /// <summary>Read An Xml File Into Alteryx.</summary>
    public class XmlInputEngine : BaseEngine<XmlInputConfig>
    {
    }
}

TMD: - Engine Class

The engine class is the guts of the tool. The Omnibus.Framework package has a BaseEngine<TConfig> which has a basic implementation of the interface needed by Alteryx AlteryxRecordInfo.Net.INetPlugIn.

Create the Plug-In Class

• Add another class for the designer part of the tool
• This needs to be derived from BasePlugIn with type arguments of the Engine class and Config class you created.
• It must also implement the AlteryxGuiToolkit.Plugins.IPlugin interface (Alteryx does not read this from parent classes).

using AlteryxGuiToolkit.Plugins;

using OmniBus.Framework;

namespace OmniBus.XmlTools
{
    /// <summary>Tell Alteryx About The Config And Engine.</summary>
    public class XmlInput : BaseTool<XmlInputConfig, XmlInputEngine>, IPlugin
    {
    }
}

TMD: Plug-In Class

This class tells Alteryx Designer about the tool. The Omnibus.Framework.GUI package has a BaseTool<TConfig,TEngine> which will provide all the bootstrapping needed for Alteryx. It also will provide a PropertyGrid as a configuration panel for Alteryx to use to set up this tool.

If you move to an HTML GUI then this class isn't needed. However, for getting started and debugging this class and it's built in GUI is an easy way forward. You can also provide your own Windows.Forms based UI if you prefer.

Install Into Alteryx And Check All Is Working

• Build In Debug Mode
• Go to the bin/Debug folder
• Run Install.bat
• Allow the script to make changes within User Account Control
• Start Debugging within Visual Studio
• Alteryx Designer should start and a new entry in the ribbon should be there
• I find it helpful to pin this ToolGroup so you can easily get to it
• Exit Alteryx

TMD: Install.bat

• This batch file uses PowerShell to Request UAC permission and then calls Scripts/Installer.ps1.
• The batch file specifies the name of the Ini file which will be written to Alteryx's Settings\AdditionalPlugins\ folder (the 2nd parameter in the Argument list)
• It also specifies the ToolGroup (the 3rd parameter) for Alteryx to add the tool.
• The ini file will tell Alteryx where to load the DLL from and where to place it in the ribbon:

[Settings]
x64Path=C:\OneDrive\MyDocs\Visual Studio 2017\Projects\OmniBus.XmlTools\bin\Debug
x86Path=C:\OneDrive\MyDocs\Visual Studio 2017\Projects\OmniBus.XmlTools\bin\Debug
ToolGroup=OmniBus.XmlTools

Add an Output Connection

• Go back to Engine class
• Add using statements for OmniBus.Framework.Attributes and OmniBus.Framework.Interfaces:

using OmniBus.Framework.Attributes;
using OmniBus.Framework.Interfaces;

• Add a new property of type IOutputHelper with get and set methods called Output
• Add a CharLabel attribute with 'O' as a parameter to the class. It should look like:

        /// <summary>Gets or sets the output stream.</summary>
        [CharLabel('O')]
        public IOutputHelper Output { get; set; }

• Rerun the debugger
• Drag the new tool to workflow
• It should have an O output connection and the Configuration panel should show an empty Property grid.
• If run the workflow all should be happy but no data will be pushed to the output.

TMD: Connections In Omnibus Framework

• The framework looks for all IOutputHelper properties as output connections
• The CharLabel attribute is optional and adds a letter label if found
• If you have more than one, then they are sorted by default by Property name
• You can use an Ordering attribute to override the name ordering
• While this walkthrough is not going to deal with Input connections, they work in a similar way
  • The framework searches for IInputProperty
  • They should be readonly auto properties (i.e. no set needed)
  • The base constructor will set them up with instance classes
  • There are events allowing you to hook into the lifecycle from Alteryx
  • This will be covered in another walkthrough

Return Metadata and a Mock row to Alteryx

• By default, the Omnibus Framework does not return debug messages to the UI. This can be useful and is worthwhile at an early. Add the following block to the engine class:

#if DEBUG
        /// <summary>Tells Alteryx whether to show debug error messages or not.</summary>
        /// <returns>A value indicating whether to show debug error messages or not.</returns>
        public override bool ShowDebugMessages() => true;
#endif

• As we have no Input connections we need to override PI_PushAllRecords in the Engine class

• Our first task is to define the metadata. This tool wilL output 3 columns in its first version:

  • XPath
  • InnerText value
  • InnerXml value

• The code below will Init the output connection with columns and then Close it to tell Alteryx it is complete.

        /// <summary>
        ///     The PI_PushAllRecords function pointed to by this property will be called by the Alteryx Engine when the plugin
        ///     should provide all of it's data to the downstream tools.
        ///     This is only pertinent to tools which have no upstream (input) connections (such as the Input tool).
        /// </summary>
        /// <param name="nRecordLimit">
        ///     The nRecordLimit parameter will be &lt; 0 to indicate that there is no limit, 0 to indicate
        ///     that the tool is being configured and no records should be sent, or &gt; 0 to indicate that only the requested
        ///     number of records should be sent.
        /// </param>
        /// <returns>Return true to indicate you successfully handled the request.</returns>
        public override bool PI_PushAllRecords(long nRecordLimit)
        {
            this.Output.Init(
                new RecordInfoBuilder()
                .AddFields(
                    new FieldDescription("XPath", FieldType.E_FT_V_WString),
                    new FieldDescription("InnerText", FieldType.E_FT_V_WString),
                    new FieldDescription("InnerXml", FieldType.E_FT_V_WString))
                .Build());
  
            this.Output.Close(true);
            return true;
        }

• Rerun the debugger

• Drag the new tool onto the workflow, run it and then we can look at the Browse Anywhere output.

• Now let's give Alteryx some data, first add a line before the this.Output.Init( line which will print out debug messages of the record limit asked for:

  • If Alteryx asks for 0 records then it is wanting the metadata only and we should handle that case
  • If a negative number is passed then all record should be sent
  • Otherwise, you should limit the records to that amount.

            this.DebugMessage($"{nameof(nRecordLimit)} Called with {nameof(nRecordLimit)} = {nRecordLimit}");

• Re-running the debugger, we can see the new message in the Messages pane:
• Add a place holder function returning an initial value:

        /// <summary>
        /// Read All The Xml From A Document
        /// </summary>
        /// <returns>List of nodes</returns>
        public IEnumerable<object[]> ReadNodes()
        {
            yield return new object[] { "/doc", "Text & Help", "Test &amp; Help" };
        }

• Now we need to handle this within PI_PushAllRecords. If we add the following just before the this.Output.Close(true); then it will iterate over all the results from ReadNodes and push them to Alteryx. In addition, it will also send out row count and data updates every 100 records (and at the end).

            if (nRecordLimit != 0)
            {
                var nodes = this.ReadNodes();
                if (nodes == null)
                {
                    return false;
                }

                long recordCount = 0;
                foreach (var data in nodes)
                {
                    this.Output.PushData(data);
                    recordCount++;

                    if (recordCount % 100 == 0)
                    {
                        this.Output.PushCountAndSize();
                    }

                    if (nRecordLimit == recordCount)
                    {
                        break;
                    }
                }
            }

            this.Output.PushCountAndSize();

• At this point, we have a functional if slightly useless tool. Rerunning the debugger and running the tool in a workflow yields the following:

TMD: Engine Overrides

• You can override various functions of the Engine
• This permits you to get to the raw Alteryx SDK if you need
• ShowDebugMessages allows you turn on error trapping
  • Override with a lambda expression returning true and extra details will be shown in the Designer
• If you override the PI_Add connections methods, you will need to call through to the base method for the framework to work correctly.
• There isn't access to the PI_Close or the PI_Init methods.
  • OnInitCalled allows you to do initialisation after the Framework has handled it.
  • OnCloseCalled will be added in a later version

TMD: Field Sizes

• The Omnibus framework will set the size parameter itself for various types
• For variable length strings it default to 1,073,741,824 as a maximum length (taken from Alteryx's formula tool default)
• For fixed length string fields you must set a size
• You must set both a size and scale for fixed decimals

TMD: Pushing Data and Progress Updates

• It is not a good idea to send row counts and size too often as this has a detrimental effect on performance. Once every 100 (or even more) rows is sufficient.
• As working with an IEnumerable, there is not a computed count. If you have a known size then you can use this.Output.UpdateProgress(percent) method to update progress as you read through.
• The PushData function is easy for programming against but you may want to use the Push function working with the Record object for higher performance (I haven't benchmarked the effect of the helper).

Add an Icon

• Alteryx expects a 171 x 171 pixel image
• The Omnibus.Framework will search for an embedded image in the DLL(s) containing the engine or the plug-in
• It must be named the same as the plug-in
• Create your tool's logo, add to the project, then set build action to be embedded resource
• Rerun the debugger and your tool should have its logo

Adding Configuration Properties

• We need to add the ability to specify the filename of the XML

• Add a string property to the config class called FileName

• The UI is based on top of the System.Windows.Forms.PropertyGrid

• The System.ComponentModel.Description attribute gives some text displayed to the user at the bottom of the PropertyGrid

        /// <summary>
        /// Gets or sets the file name containing the Xml
        /// </summary>
        [Description("Specify the filename of the XML file")]
        public string FileName { get; set; }

• To add an OpenFileDialog we need to do the following:
  • Add a reference to System.Design
  • Add the following attribute to the property:

        [Editor(typeof(System.Windows.Forms.Design.FileNameEditor), typeof(System.Drawing.Design.UITypeEditor))]

It is beyond the scope of this tutorial to show you how to customise the OpenFileDialog. This is something I am thinking of for later versions of the framework

• We also want the configuration object to produce the default annotation. Following Alteryx's Input tool style, this should be the filename.
• Change the ToString method to the following:

        /// <summary>Returns a string that represents the current object.</summary>
        /// <returns>A string that represents the current object, used as the Default Annotation for Alteryx.</returns>
        public override string ToString()

        {
            try
            {
                return string.IsNullOrWhiteSpace(this.FileName) ? string.Empty : System.IO.Path.GetFileName(this.FileName);
            }
            catch
            {
                return String.Empty;
            }
        }

• Rerun Alteryx and now when you access the tool you will have the option to select a filename which will then be displayed as a default annotation: assets/newTool/withFIleNameConfig.jpg

TMD: Property Grid

• The Omnibus framework has a few extra attributes for setting up the property grid
• FieldListAttribute specifies a set of valid values (useful for fixed list of types)
• InputPropertyAttribute specifies an input property (useful for picking a specific incoming field)
• In additional there are some TypeConverter classes for dealing with field types.
• As the framework matures and development on the HTML side progresses, these attributes will increase and add more functionality.

Reading Real Files

• The configuration will be automatically deserialised into a ConfigObject property of the engine.
• Now we want to adjust the existing ReadNodes function to read the configuration
  • If the filename is set and the file exists then it should try and load it
  • If the filename doesn't exist then a nice error message should be sent
  • If the XML fails to parse then a nice error message should be sent
  • Like all other Alteryx inputs, relative filenames should work as well
• Replace the exisitng ReadNodes function with:

        /// <summary>
        /// Read All The Xml Nodes From A Document
        /// </summary>
        /// <returns>List of nodes</returns>
        public IEnumerable<object[]> ReadNodes()
        {
            if (string.IsNullOrWhiteSpace(this.ConfigObject.FileName))
            {
                this.Message("You need to specify a filename.", MessageStatus.STATUS_Error);
                return null;
            }

            var fileName = System.IO.Path.IsPathRooted(this.ConfigObject.FileName)
                               ? this.ConfigObject.FileName
                               : System.IO.Path.Combine(
                                   this.Engine.GetInitVar(nameof(InitVar.DefaultDir)),
                                   this.ConfigObject.FileName);

            if (!System.IO.File.Exists(fileName))
            {
                this.Message($"File {this.ConfigObject.FileName} could not be read.", MessageStatus.STATUS_Error);
                return null;
            }

            var document = new XmlDocument();

            try
            {
                document.Load(fileName);
            }
            catch (XmlException ex)
            {
                this.Message($"Error reading XML: {ex.Message}");
                return null;
            }

            return this.ReadNodes(document.DocumentElement);

            return this.ReadNodes(document.DocumentElement);
        }

• Next, add the method below to recursively scan an XmlNode's child nodes and produce a IEnumerable set of results (the detail of this method is beyond scope of this tutorial):

        /// <summary>
        /// Read All The Xml Nodes From A Node
        /// Recursive scan from input node
        /// </summary>
        /// <returns>List of nodes</returns>
        public IEnumerable<object[]> ReadNodes(XmlNode node, string path = "")
        {
            if (node == null)
            {
                return Enumerable.Empty<object[]>();
            }

            path = path + "/" + (node is XmlAttribute ? "@" : "") + node.Name;

            var nodes = node.ChildNodes.Cast<XmlNode>();
            var txtNodes = nodes.Where(x => x.Name == "#text").ToArray();

            var txt = txtNodes.Length == 0 ? node.InnerText : null;
            if (txtNodes.Length == 1)
            {
                txt = txtNodes[0].InnerText;
                nodes = nodes.Where(n => n.Name != "#text");
            }

            IEnumerable<object[]> result = new[] { new object[] { path, txt, node.InnerXml } };

            if (node.Attributes != null)
            {
                result = result.Concat(node.Attributes.Cast<XmlNode>().SelectMany(n => this.ReadNodes(n, path)));
            }

            return result.Concat(nodes.SelectMany(n => this.ReadNodes(n, path)));
        }

• Rerun the debugger, Drag the tool, pick an XML file (or an Alteryx Workflow), run the flow, and congratulations you have a working Custom Tool!

• If you want to install it on another machine, all you need to do is copy the bin/Debug or bin/Release folder and then run Install.bat.

Complete Example

• This tool, based on the code above, has been included in the 0.5.3 version of OmniBus tools.
• The complete source code is available here.

Additional Next Steps

These are not included in this tutorial but will eventually be part of the XML Input tool within the OmniBus add ins

• Allow for wildcards in the filename and searching subfolders (much like Alteryx's Input tool)
• Allow user to decide to include the filename in output
• Allow user to include/exclude the InnerXml
• Move to an XmlReader to avoid loading entire XML file into memory