Epic - Build Trust In Impact Framework

[jmcook1186]

Overview

Although we've been using the QA name as shorthand for this epic, it's really about "Building trust in the IF". This is a much broader scope than QA alone, although QA is an important part of it.

We want to put processes and tools in place so that people feel confident using IF and participating in the IF community as builders, contributors, users, advocates, evangelists or honest critics!

To achieve this we will focus on seven main themes:

• Communication
• Tooling
• Github processes
• QA/Testing
• Errors and logs
• Bug handling
• Tracking our own performance

By completing specific tasks that contribute to one or more of these topics we can achieve the following:

• it is easier to identify and report bugs and vulnerabilities
• the core team can respond to issues more efficiently
• we set clear expectations for the IF community
• the product becomes more stable and reliable
• we build trust and reputation
• the IF ecosystem is easier to maintain

These outcomes lead to a safer product and more vibrant community, and that will translate into adoption. We want this to stimulate a flywheel effect where more high integrity interactions leads to more adoption, which leads to more high integrity interactions, with trust being a positive externality.

Epic goals

The high levels goals we want to achieve are:

• Improve our communication with the community
• Add tooling to empower contributors to get the msot out of IF
• Refine our Github processes to give a consistent experience for core developers and contributors across IF repositories
• Add QA/testing processes to ensure the integrity of our repositories and packages
• Track and report our own performance with a monthly cadence

User stories

As a core developer I want a clear process for SDLC so that I we don’t block each other, do throw away work and maintain developer velocity.

• Define process for categorizing bugs into H, M, L severity #588
• Add bug triage call to core team weekly calendar #587
• Define process for categorizing bugs into H, M, L severity #588
• Refine bug reporting guidance documentation #590
• Document our bug reporting requirements and create an issue template which meets them. #586
• Revise issue refinement/readiness process #613
  • It should contain at least one runnable complete manifest file for use in development and for testing/UAT.
• Define process for ensuring docs and code stay in sync #611

As a QA engineer I want a clear test process so we can ensure the quality of our product doesn’t dip below what the community will accept.

• Define and implement automated testing protocol #620
• Define process for integration/regression testing #605
• Define process for documenting test cases #610
• Document the QA requirements for merging into main/release branches #632
• 👍🏼 Update IF to dump logs and errors to yaml file #592
  • audit
    • log
      • Critical logs (exceptions)
      • We can use this for negative testing, if we expect a failure we expect a manifest file with a log entry for the failure we expect
• 👍🏼 Update IF to report runtime information to outputs #591
  • audit
    • command
      • The exact command used to run
    • environment
      • Information about the runtime environment, machine, os, node version, date-time of the run etc...
  • plugins
    • In the output manifest file we should record the versions of the plugins that were used so we can be sure of exactly the code that was used to run this manifest
      • plugins
        • name
          • version
            • "^1.0.3",
      • When using our if-env tool it will generate a package.json with these exact version numbers
• 👍🏽 Create a folder of manifest files that broadly tests IF across a range of test cases positive and negative. #615
• Add negative flow manifests to test scenarios #619
• 👍🏼 Set up CI/CD to trigger automated tests for PRs #621

As a core developer I want to encourage community contribution so that we grow and nurture the community and ecosystem for IF.

• Revise issue/pr acceptance process and update documentation #612
• Document how we label our issues and expectations for each label. #630
  • core-team-only: We’re happy for community feedback, but the development will be driven via core team members.
  • good-first-issue: Well refined low priority issues, with clear test cases, that we are open to community contributions over.
• Create repository for plugin registry webpage #614
• Define acceptance process for listing plugins on if registry #617
• Epic - Community plugin registry #633
• Add documentation covering the plugin registry and its governance process #634
• Create public google group for if contributors #628
• Clarify contribution expectations in contributing.md #635
• Add a simple js-only plugin template #603
• Add documentation for error handling best practices #602
• Write new docs for debugging IF #601

As a developer I want better tooling so that I can run, test and report issues with manifest files and plugins

• Improve error messages emitted from IF and plugins #593
• Identify and update error messages to conform with new standard #606
• Design if-env command line tool #624
• Build if-env #636
  • Given a manifest file, sets up a local environment with everything requires to run that manifest file installed.
    • exports a package.json
• Design if-diff command line tool #623
• Build if-diff core feature #637
  • Given two manifest files compares them to see if they match and if not what is different.
    • different levels of matching, order of nodes is unimportant
      • exact
        • every field must match exactly
      • tree
        • the entire tree should match
      • ??
• Design if-verify command line tool #625
• Build if-check #638
  • Runs all the manifest files in a folder and either compares them to itself OR to another compared manifest file
    It’s sharing the same platform with integration tests?
  • Effectively runs if-env and if-diff on a manifest file in one go
  • Can be run as a one-off on one file OR on a folder of files
    • Outputs a json file which records the pass/fail log status.
    • Used by any plugin repo that wants to use it
    • Use by if and if-plugins
    • Used by our plugin registry repo
• Create a ci/cd pipeline that runs if-verify on a folder of manifests
  • a github action or somethig which runs if-verify and outputs the results somehow
  • This can be a published GH action we can recommend other plugin developers to use.
• Design if-blitz command line tool #627
• Build if-blitz #639
  • Create a stackblitz for a manifest file e.g. https://stackblitz.com/edit/nestjs-typescript-starter-9g3p4g?file=README.md&terminal=start
  • Using the Stackblitz POST API (https://developer.stackblitz.com/platform/api/post-api) and if-env tool you can create an online environment to run a manifest file.
  • Users can then share a link to their manifest file to others and easily share to us for bug reporting.

As a plugin developer I want clarity regarding the IF interface so that I can take advantage of IF features and help manifest developers diagnose issues faster.

• Document the list of exceptions we accept from plugins and the expected behaviour when raising that exception. #337
• Improve IF logging protocols #600

As a core developer I want to cleanup and consolidate our collection of plugins so we can reduce the resources required to QA/maintain plugins

• Audit existing plugins to determine what can be replaced with generics #640
• Migrate plugins from if-plugins to if as builtins #616,
• Delete, hand-off or migrate plugins out of if-unofficial-plugins #618.
• Reach out to orgs to take responsibility for plugins #546.

As a user of IF I want security so that I can trust that using it won’t create security holes

• add npm audit to install script to surface supply chain issues #626
• Explore running plugin code in a sandbox environment
  • E.g. https://stackoverflow.com/questions/7446729/how-to-run-user-submitted-scripts-securely-in-a-node-js-sandbox
  • Consider adding a flat —run-untrusted to allow the execution of code in a trusted environment and make it more explicit to the user that they are taking responsibility for security.

Success metrics

• Agree a set of QA KPIs #594
• Set up KPI tracking sheet #598
• Agree process for KPI actions #599

Timeline

Ticket refinement and work will begin on Monday 15th April 2024

[zanete]

@jmcook1186 I was trying to reconcile this original epic with all the tasks against the structured by feature approach we have going in the #693 , and have a few of questions:

• "Create a ci/cd pipeline that runs if-verify on a folder of manifests" - I feel this is covered by a set of issues, correct?
• "Explore running plugin code in a sandbox environment" - same comment here, but I already asked it on Epic - Security #658
• what is to be done with the KPI related ones, If i remember correctly you said that there are KPIs somewhere defined already, can you share more detail on that?
• is there anything on here that I we've missed and I just can't see?
  🙏

[jmcook1186]

@zanete

• "Create a ci/cd pipeline that runs if-verify on a folder of manifests" : if-verify is now called if-check. These tickets cover this task:

#638
#621

• explore running in sandbox: personally think this should get backlogged, but wait for Asim to respond on Epic - Security #658
• KPI related ones can be closed out - Asim wanted to handle this separately. We do have some OKRs for N plugins (>50) and N contributors (>20) by the end of the year.

I think we can pretty much call this epic done when we have if-check operating over positive and negative test manifests as part of our CI/CD.

[zanete]

@jmcook1186 awesome, I did a quick review on what's remaining and you can see all the tasks here: https://github.com/orgs/Green-Software-Foundation/projects/26/views/13