DTT2 - Observability module - Develop Workflow Dashboard

[fcaffieri]

Epic: #4837

---

Description

The objective of this issue is to develop all the necessary components for the Workflow to report the real status to the observability module.

Tasks

• Develop Workflow Dasboard
• Develop API
• Made modifications to Workflow
• Configure integration
• Test the integration and Dashboard

Working branch: enhancement/5075-dtt2-iteration-3-develop-workflow-dashboard

[fcaffieri]

Dashboard

The following dashboard was generated

This graph represent this execution:

[2024-03-05 17:44:40] [INFO] [68316] [MainThread] [workflow_engine]: Execution plan: {
  "test-agent-linux-redhat-8-amd64": {
    "provision-agent-linux-redhat-8-amd64": {
      "allocate-agent-linux-redhat-8-amd64": {},
      "provision-manager-linux-amazon-2023-amd64": {
        "allocate-manager-linux-amazon-2023-amd64": {}
      }
    }
  },
  "test-agent-linux-redhat-7-amd64": {
    "provision-agent-linux-redhat-7-amd64": {
      "allocate-agent-linux-redhat-7-amd64": {},
     },
      "provision-manager-linux-amazon-2023-amd64": {
        "allocate-manager-linux-amazon-2023-amd64": {}
      }
  }
}

I will continue developing the API

[fcaffieri]

Update

The first approach for the API (this will connect with the Grafana Node graph API plugin):

Endpoints defined:

Define the fields:

@app.route('/api/graph/fields')
def fetch_graph_fields():

Defined the data: Fields data + connection of the nodes, the "Edges" data.

@app.route('/api/graph/data')
def fetch_graph_data():

API status:

@app.route('/api/health')
def check_health():

For now, this run with Python Flask in the same VM where Grafana is running

Decorators in workflow:

def send_to_api(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        # Call the API with data
        return result
    return wrapper

Execution example:

@send_to_api
def run(self) -> None:
        """Main entry point."""
        try:
            if not self.dry_run:
                logger.info("Executing DAG tasks.")
                dag = DAG(self.task_collection)
                self.execute_tasks_parallel(dag)

                logger.info("Executing Reverse DAG tasks.")
                reversed_dag = DAG(self.task_collection, reverse=True)
                self.execute_tasks_parallel(reversed_dag, reverse=True)
            else:
                dag = DAG(self.task_collection)
                logger.info("Execution plan: %s", json.dumps(dag.get_execution_plan(), indent=2))

        except Exception as e:
            logger.error("Error in Workflow: %s", e)

I am currently working on the problem that the decorator is going to add methods before or after the function that the decorator contains. The problem is how to execute the call in the middle of said method. Otherwise the decorator will move to the methods that provide me with information.

[fcaffieri]

Update report

Testing API and Dashboard with three parallel Workflows executions:

API is Running:

Populate manually the Node Graph plugin with the API (simulating that the workflow with the decorators invokes the API):

$ curl http://172.16.1.58:5000/api/health
API is working well!$ 
$ 
$ 
$ 
$ curl -X PUT -H "Content-Type: application/json" -d '{"execution_id": "61057","nodes": [   {       "id": "1",       "title": "Test",       "subtitle": "run-agent-tests-linux-ubuntu-22.04-amd64: install,register,stop",       "mainstat": "Waiting",       "secondarystat": 4,       "color": "grey",       "icon": "exclamation-circle"   },   {       "id": "2",       "title": "Provision",       "subtitle": "provision-install-linux-ubuntu-22.04-amd64",       "mainstat": "Canceled",       "secondarystat": 1,       "color": "orange",       "icon": "times"   },   {       "id": "3",       "title": "Provision",       "subtitle": "provision-manager",       "mainstat": "Running",       "secondarystat": 1,       "color": "blue",       "icon": "fa fa-spinner"   },   {       "id": "4",       "title": "Allocation",       "subtitle": "allocate-manager",       "mainstat": "Finished",       "secondarystat": 0,       "color": "green",       "icon": "check"   }],"edges": [   {       "id": "1",       "source": "1",       "target": "2"   },   {       "id": "2",       "source": "1",       "target": "3"   },    {       "id": "3",       "source": "3",       "target": "4"   }]}' http://172.16.1.58:5000/api/graph/data
{"message":"Data updated successfully"}
$ 
$ curl -X PUT -H "Content-Type: application/json" -d '{"execution_id": "69825","nodes": [{"id": "11", "title": "New Node 1", "subTitle": "Subtitle 1", "mainStat": "Running", "secondaryStat": 3, "color": "blue", "icon": "fa fa-cogs"},{"id": "12", "title": "asdfasdfgasfdgadf", "subTitle": "Subtitle 2", "mainStat": "Finished", "secondaryStat": 1, "color": "green", "icon": "check"}],"edges": [{"id": "9", "source": "11", "target": "12"}]}' http://172.16.1.58:5000/api/graph/data
{"message":"Data updated successfully"}
$ 
$ curl -X PUT -H "Content-Type: application/json" -d '{"execution_id": "69535","nodes": [{"id": "1","title": "Test","subtitle": "run-agent-tests-linux-ubuntu-22.04-amd64: install,register,stop","mainstat": "Waiting","secondarystat": 4,"color": "grey","icon": "exclamation-circle"},{"id": "2","title": "Provision","subtitle": "provision-install-linux-ubuntu-22.04-amd64","mainstat": "Canceled","secondarystat": 1,"color": "orange","icon": "times"},{"id": "3","title": "Provision","subtitle": "provision-manager","mainstat": "Running","secondarystat": 1,"color": "blue","icon": "fa fa-spinner"},{"id": "4","title": "Allocation","subtitle": "allocate-manager","mainstat": "Finished","secondarystat": 0,"color": "green","icon": "check"},{"id": "5","title": "Allocation","subtitle": "allocate-linux-ubuntu-22.04-amd64","mainstat": "Failed","secondarystat": 0,"color": "red","icon": "exclamation-triangle"},{"id": "6","title": "Test","subtitle": "test-agent-linux-redhat-8-amd64: install,register,stop","mainstat": "Waiting","secondarystat": 4,"color": "grey","icon": "exclamation-circle"},{"id": "7","title": "Provision","subtitle": "provision-agent-linux-redhat-8-amd64","mainstat": "Canceled","secondarystat": 1,"color": "orange","icon": "times"},{"id": "8","title": "Provision","subtitle": "provision-manager","mainstat": "Running","secondarystat": 1,"color": "blue","icon": "fa fa-spinner"},{"id": "9","title": "Allocation","subtitle": "allocate-manager","mainstat": "Finished","secondarystat": 0,"color": "green","icon": "check"},{"id": "10","title": "Allocation","subtitle": "allocate-agent-linux-redhat-8-amd64","mainstat": "Running","secondarystat": 0,"color": "blue","icon": "fa fa-spinner"}],"edges": [{"id": "1","source": "1","target": "2"},{"id": "2","source": "1","target": "3"},{"id": "3","source": "3","target": "4"},{"id": "4","source": "2","target": "5"},{ "id": "5", "source": "6", "target": "7"},{ "id": "6", "source": "7", "target": "8"},{ "id": "7", "source": "7", "target": "10"},{ "id": "8", "source": "8", "target": "9"}]}'  http://172.16.1.58:5000/api/graph/data
{"message":"Data updated successfully"}
$ 

Consulting the API to search for information:

$ curl http://172.16.1.58:5000/api/graph/execution_ids
["61057","69825","69535"]
$ 
$ 
$ curl -X GET http://172.16.1.58:5000/api/graph/data?execution_id=69535
{"edges":[{"id":"1","source":"1","target":"2"},{"id":"2","source":"1","target":"3"},{"id":"3","source":"3","target":"4"},{"id":"4","source":"2","target":"5"},{"id":"5","source":"6","target":"7"},{"id":"6","source":"7","target":"8"},{"id":"7","source":"7","target":"10"},{"id":"8","source":"8","target":"9"}],"nodes":[{"color":"grey","icon":"exclamation-circle","id":"1","mainstat":"Waiting","secondarystat":4,"subtitle":"run-agent-tests-linux-ubuntu-22.04-amd64: install,register,stop","title":"Test"},{"color":"orange","icon":"times","id":"2","mainstat":"Canceled","secondarystat":1,"subtitle":"provision-install-linux-ubuntu-22.04-amd64","title":"Provision"},{"color":"blue","icon":"fa fa-spinner","id":"3","mainstat":"Running","secondarystat":1,"subtitle":"provision-manager","title":"Provision"},{"color":"green","icon":"check","id":"4","mainstat":"Finished","secondarystat":0,"subtitle":"allocate-manager","title":"Allocation"},{"color":"red","icon":"exclamation-triangle","id":"5","mainstat":"Failed","secondarystat":0,"subtitle":"allocate-linux-ubuntu-22.04-amd64","title":"Allocation"},{"color":"grey","icon":"exclamation-circle","id":"6","mainstat":"Waiting","secondarystat":4,"subtitle":"test-agent-linux-redhat-8-amd64: install,register,stop","title":"Test"},{"color":"orange","icon":"times","id":"7","mainstat":"Canceled","secondarystat":1,"subtitle":"provision-agent-linux-redhat-8-amd64","title":"Provision"},{"color":"blue","icon":"fa fa-spinner","id":"8","mainstat":"Running","secondarystat":1,"subtitle":"provision-manager","title":"Provision"},{"color":"green","icon":"check","id":"9","mainstat":"Finished","secondarystat":0,"subtitle":"allocate-manager","title":"Allocation"},{"color":"blue","icon":"fa fa-spinner","id":"10","mainstat":"Running","secondarystat":0,"subtitle":"allocate-agent-linux-redhat-8-amd64","title":"Allocation"}]}
$ 
$ curl -X GET http://172.16.1.58:5000/api/graph/data?execution_id=69825
{"edges":[{"id":"9","source":"11","target":"12"}],"nodes":[{"color":"blue","icon":"fa fa-cogs","id":"11","mainStat":"Running","secondaryStat":3,"subTitle":"Subtitle 1","title":"New Node 1"},{"color":"green","icon":"check","id":"12","mainStat":"Finished","secondaryStat":1,"subTitle":"Subtitle 2","title":"asdfasdfgasfdgadf"}]}
$ 
$ curl -X GET http://172.16.1.58:5000/api/graph/data?execution_id=61057
{"edges":[{"id":"1","source":"1","target":"2"},{"id":"2","source":"1","target":"3"},{"id":"3","source":"3","target":"4"}],"nodes":[{"color":"grey","icon":"exclamation-circle","id":"1","mainstat":"Waiting","secondarystat":4,"subtitle":"run-agent-tests-linux-ubuntu-22.04-amd64: install,register,stop","title":"Test"},{"color":"orange","icon":"times","id":"2","mainstat":"Canceled","secondarystat":1,"subtitle":"provision-install-linux-ubuntu-22.04-amd64","title":"Provision"},{"color":"blue","icon":"fa fa-spinner","id":"3","mainstat":"Running","secondarystat":1,"subtitle":"provision-manager","title":"Provision"},{"color":"green","icon":"check","id":"4","mainstat":"Finished","secondarystat":0,"subtitle":"allocate-manager","title":"Allocation"}]}
$

Visualizing data in Grafana selecting the execution dynamically:

Currently I have generated the decorator and I am finalizing details to move on to the testing stage in the local environment that I have prepared with Jenkins, grafana and the services that it needs.

[fcaffieri]

Update

I have encountered a problem with the generated Graphs, where as mentioned in the documentation you can generate links to other parts of the graph by clicking on a node, but after many tests I have not been able to use that functionality. After investigating, he came across a couple of unanswered issues written recently, where several users ask about this same thing. Apparently it is only possible with a specific data source, which is used only for development.
That said, I am analyzing other possibilities, since in order to successfully exploit the Workflow graph, we understand that it is necessary to be able to click on each node and that this action shows information about the execution of said node, for example all the information of the test.
I have found this alternative, which seems much more powerful but requires much greater study and learning due to its complexity.
https://volkovlabs.io/plugins/volkovlabs-echarts-panel/tutorials/graph/

At the moment I have managed to generate Node Graph with the same information but with java script code and hardcoding the data of the nodes and edges.

Hardcode code:

return {
  tooltip: {},
  animationDurationUpdate: 1500,
  animationEasingUpdate: 'quinticInOut',
  series: [
    {
      type: 'graph',
      layout: 'none',
      symbolSize: 50,
      roam: true,
      label: {
        show: true
      },
      edgeSymbol: ['circle', 'arrow'],
      edgeSymbolSize: [4, 10],
      edgeLabel: {
        fontSize: 20
      },
      data: [
        {
          id: '1',
          name: 'run-agent-tests-linux-ubuntu-22.04-amd64',
          x: 300,
          y: 300
        },
        {
          id: '2',
          name: 'provision-install-linux-ubuntu-22.04-amd64',
          x: 600,
          y: 500
        },
        {
          id: '3',
          name: 'provision-manager',
          x: 50,
          y: 500
        },
        {
          id: '4',
          name: 'allocate-manager',
          x: 50,
          y: 800
        },
        {
          id: '5',
          name: 'allocate-linux-ubuntu-22.04-amd64',
          x: 600,
          y: 800
        }
      ],
      // links: [],
      links: [
        {
          source: '1',
          target: '2',
        },
        {
            source: '1',
            target: '3',
        },
        {
          source: '3',
          target: '4'
        },
        {
          source: '2',
          target: '5'
        },
        {
          source: 'Node 1',
          target: 'Node 4'
        }
      ],
      lineStyle: {
        opacity: 0.9,
        width: 2,
        curveness: 0
      }
    }
  ]
};

Version with real datasource:

let sources = [];
let targets = [];
let titles = [];
let mainstats = [];
let icons = [];
let nodes, links;


data.series.map((s) => {
  if (s.name === "nodes") {
    titles = s.fields.find((f) => f.name === "title").values.buffer ||
             s.fields.find((f) => f.name === "title").values;
    mainstats = s.fields.find((f) => f.name === "mainstat").values.buffer ||
                s.fields.find((f) => f.name === "mainstat").values;
    icons = s.fields.find((f) => f.name === "icon").values.buffer ||
            s.fields.find((f) => f.name === "icon").values;

    nodes = titles.map((title, i) => ({
      name: title,
      symbolSize: icons[i],
      value: mainstats[i],
      category: 0,
      label: {
        show: true,
      },
    }));
  }

  if (s.name === "edges") {
    sources = s.fields.find((f) => f.name === "source").values.buffer ||
              s.fields.find((f) => f.name === "source").values;
    targets = s.fields.find((f) => f.name === "target").values.buffer ||
              s.fields.find((f) => f.name === "target").values;

    links = sources.map((source, i) => ({
      source: source,
      target: targets[i],
    }));
  }
});

return {
  tooltip: {},
  animationDurationUpdate: 1500,
  animationEasingUpdate: "quinticInOut",
  series: [
    {
      name: "test2",
      type: "graph",
      layout: "circular",
      circular: {
        rotateLabel: true,
      },
      data: nodes,
      links: links,
      roam: true,
      label: {
        position: "right",
        formatter: "{b}",
      },
      lineStyle: {
        color: "source",
        curveness: 0.3,
      },
    },
  ],
};

[fcaffieri]

Update report

Finally, I was able to generate a graph using the js code that obtains the data from the previously generated API.
Then when clicking on each node or edge I have managed to capture the click event and then redirect to another part of the graph to display the specific information of that node.
It remains to generate the correct URL, this will be dynamic and depend on the graphs that are generated for each module since it will set a series of specific global variables so that each module graph can filter the execution of that selected node.

Pending and ongoing tasks:

• Generating the graphics of each module
• Then it is necessary to generate the configuration to obtain the information of each module with the Loki agent
• Solving a problem with the visualization of the nodes in the Workflow dashboard, since a DAG is required to be generated, and the "graph" type of graph requires setting a layout to generate the nodes dynamically. If this is not set, it requires configuring the x and y axis on each node. This is not viable. It continued analyzing the types of layouts to generate a dynamically visible graph.

[fcaffieri]

Issue Bloqued due to DTT 1 priority to release Tier 1