Custom Test Metadata/Tags for use in Reporters

[azohra]

Are you requesting a feature or reporting a bug?

Feature Request

What is the current behavior?

TestCafe does not support pipelining custom metadata/tags to a reporter. However, I do see in the previous incarnation there was a form of this: https://testcafe.devexpress.com/Documentation/API_Reference/Test_Fixture_API/Common_Concepts/#Test_Run_Metadata

What is the expected behavior?

Ideally we would like to be able to define metadata/taga in our tests and have it available to the reporters.

Use Cases:

• Severity
• Test Management Tool ID / URL

Example:

test('My first test', {severity: critical, testID: TC-0013}, async t => {
    // Test code
});

or

test('My first test', async t => {
    t.meta('severity', 'critical')
    t.meta('testID', 'TC-0013')
});

such that the testRunInfo object in reportTestDone would have access

reportTestDone (name, testRunInfo) {
    if (testRunInfo.severity)
        title += ' (testRunInfo.severity)';

    if (testRunInfo.testID)
        title += ` (testRunInfo.testID)`;
}

If you are familiar with RSpec, you will know this concept well. Here is what it looks like: https://relishapp.com/rspec/rspec-core/docs/metadata/user-defined-metadata

However, in RSpec it is much more powerful as tags/metadata can be used to drive whether tests should be run/skipped - and also drive logic within the tests themselves.

[azohra]

@AlexanderMoskovkin - can you confirm my suspicion that TestCafe does not currently support this?
What is the underlying test framework, is it mocha? Wondering if I can put together a PR to add support.

[AlexanderMoskovkin]

Yep, TestCafe doesn't support it right now. TestCafe uses its own compiler under the hood.

TestCafe doesn't provide an ability to add some custom info from the test to its report. But we have test context and fixture context objects. We can implement sharing info between tests and reports by using them. It may look in the following way:

// test.js 
test('Test1', async t => {
     t.fixtureCtx.severity = 'critical';
     t.ctx.testID = 'TC-0013';
});

// custom-report.js
reportTestDone (name, testRunInfo) {
    const severity = testRunInfo.fixtureCtx.severity;
    const testID   = testRunInfo.ctx.testID;

    ...
}

How do you like this way? /CC @tk8817 @DevExpress/testcafe-docs @VasilyStrelyaev

Wondering if I can put together a PR to add support

When we will up with the final API the PR is welcome. We will provide any assistance.

[azohra]

@AlexanderMoskovkin - thanks for pointing out the ctx functionality, I think they are solid entry-points that your users would already be familiar with. Forwarding those objects to the reporters would solve our use case perfectly.

[hdorgeval]

Hi @AlexanderMoskovkin, I am currently looking on how to inject test/fixture context in the reporter. Would you accept a PR for this kind of job or are you already working/planning on it?

[AlexanderMoskovkin]

Hi @hdorgeval, Your PR is welcome. Let me know if you need any assistance.

[azohra]

@AlexanderMoskovkin - we missed @hdorgeval's post and took it on ourselves. I have had one of my team members raise PR #2258 which we hope is to your satisfaction.

[inikulin]

It doesn't feel right to use ctx for test metadata. It's purpose is just to share variables between tests/fixtures in runtime. We have test/fixture tags proposal somewhere here as far as I remember, seems like it's a good fit for this kind of things.

[azohra]

@inikulin, I’m not sure I understand. What is the purpose for ctx outside of this usecase?

The ‘context’ of a fixture or test is exactly what we are lookinng to expose to the reporter. Interestingly, if you look at our PR - it’s a 2 line code change, as you guys were already capturing this and passing it forward, but simply leaving it out of the final object the reporter receives.

[inikulin]

@tk8817 ctx has nothing to do with reporting. The only purpose of this property is to share variables between tests and test hooks: https://devexpress.github.io/testcafe/documentation/test-api/test-code-structure.html#sharing-variables-between-test-hooks-and-test-code. I believe there should be dedicated property for such reporting purposes, otherwise it will be quite confusing.

[azohra]

@inikulin - if you look at the code, this object is where the reporter gets all of its data from, it was literally a one line update to further expose the same object...

[inikulin]

if you look at the code, this object is where the reporter gets all of its data from

It's not.

it was literally a one line update to further expose the same object...

It's a doubtful excuse for improper API design choices.

[AlexanderMoskovkin]

Here is one more use case: https://testcafe-discuss.devexpress.com/t/is-it-possible-to-pass-custom-values-to-custom-report-functions/706?u=amoskovkin It doesn't look like just a tag. Some kind of additional/meta info.
Your proposals?

[MatthewNielsen27]

I don't understand why data from the tests and fixtures are cherry picked and left out from the final testRunInfo. The data is literally dropped at the last step. What even is the rational for leaving out the test and fixture from the reporter? It only limits the capabilities of what a truly custom reporter can do. As for the ctx, that is exactly what we are trying to report on. For massive projects the context of a test run is as important as the outcome in terms of a reporting standpoint.

[inikulin]

@AlexanderMoskovkin

t.meta.foo = 'bar';

[inikulin]

@MatthewNielsen27 testRunInfo contains ctx property because it contains whole test and lots of other stuff that matters only for testcafe runtime. Please, re-read carefully the link that I've provided above about ctx and it's purpose. It's just wrong from semantical perspective to use this property to report meta information. We need a dedicated property for that.

[MatthewNielsen27]

@inikulin I understand that a separate meta field would be the best implementation for that use. But nevertheless I see value in the reporter having access to the shared context data between tests. Besides metadata, this context data would also be valuable to report on. So this PR still has merit as a useful addition from my perspective.

[hdorgeval]

@inikulin , @AlexanderMoskovkin,
I looked at the PR #2258, and I would have done the same job on my side: it is a quick solution but I admit this is a questionable solution.

Questionable because:

• pushing the contexts into the reporter may enable the reporter to modify these contexts and therefore this could create side effects. Only immutable or POJO objects should be pushed to the reporter;
• on parallel execution, I absolutely do not know how contexts are managed;
• there is no guaranty on the ctx life cycle: the ctx object may not exist any more when calling the reporter.

PR #2258 would have been a time saver and would give a true value. I would have use it in order to be able to create a cucumber-like json reporter and then be able to create nice html reports with cucumber-html-reporter (see my repo cucumber-ts-starter).

What is actually missing in TestCafe is the fact that there are only two levels for describing tests: the fixture level and the test level. In a way having only two levels prevent deep and unnecessary nesting.

What is lacking is the possibility to tag a test and the possibility to divide a test into steps.

Why?

• Different tests, in different files may cover the same business acceptance scenario.
• Not having steps inside a test may lead to have many code duplication and tests with many lines of code.

Using a Page Model may solve this problem on a technical point of view but not on a business point of view.

When the TestCafe code base start to grow, more time is needed to write new test and to maintain existing ones. Some dashboard has to be produced for the business in order to show what is going on and how TestCafe is a great product.

But dashboard is a sort of reporter and current reporter API cannot produce high level report targeted to the business.

I would love to be able to write the following code:

fixture("My Fixture")
    .page("http://myurl")
    .before(async (ctx) => {
        // code omitted for brevity
    })
    .beforeEach(async (t) => {
        await t.report({keyword: "BeforeEach"},/*  pass a POJO object of type IMyReportInfo */);
        // code omitted for brevity
    })


test("My Test", async (t) => {
    await t.report({keyword: "StepStart"},/*  pass a POJO object of type IMyReportInfo */);
    // some test code
    await t.report({keyword: "StepEnd"},/*  pass another POJO object of type IMyReportInfo */)
    // other code
});

//internally the test function should be wrapped by:
// t.report({keyword: "TestStart", name:"My Test"}, /* POJO object like testRunInfo */), 
// t.report({keyword: "TestEnd", name:"My Test"})

//internally the fixture should be wrapped by:
// t.report({keyword: "FixtureStart", name:"My Fixture"}, /* POJO object like testRunInfo */), 
// t.report({keyword: "FixtureEnd", name:"My Fixture"})

// custom POJO object that will be pushed to the reporter
export interface IMyReportInfo {
    tags: string[];
    keyword: string;
    description: string;
    // any custom information needed by the reporter
}

[MatthewNielsen27]

@inikulin @AlexanderMoskovkin We need a feature like this quickly to support our business and are willing to code it. PR #2258 would cover all of our needs immediately, but we would go your route with the additional meta property if you are willing to support it. We just need guidance on the best way to implement the feature if we go that route.

[AlexanderMoskovkin]

Hi guys,

When I made my proposal, I missed one thing. If we run a test in two browsers, we have two testRuns and it means we have two different test contexts. But the reportTestDone callback is called only once when the test is finished in both browsers. This is another reason to avoid this approach.

@MatthewNielsen27, I guess we can go with the meta approach. The API may look in the following way:

// test.js 
fixture `my fixture`
    .meta('field1', 'value1')
    .meta({ field2: 'value2', field3: 'value3' });

test
    .meta('field4', 'value4')
    ('Test1', async t => {
    // ... 
});

// custom-report.js
reportTestDone (name, testRunInfo) {
    const metaValue1 = testRunInfo.meta.field1;
    const metaValue5 = testRunInfo.meta.field5;
    ...
}

What do you think? @MatthewNielsen27 , @hdorgeval , @tk8817 , @inikulin , @DevExpress/testcafe-docs

[AlexanderMoskovkin]

What is lacking is the possibility to tag a test and the possibility to divide a test into steps.

@hdorgeval, It seems that you are talking about the Implement test sections #1381 feature?

[MatthewNielsen27]

@AlexanderMoskovkin Sounds good, would the meta for tests and fixtures be separate properties of the testRunInfo?

[AlexanderMoskovkin]

During the discussion, we came up with the following thoughts:

1. It's not good to mix fixture and test meta info. For example, it's not clear what we should do if we'd like to add the tag meta for both:

fixture.meta('tag', 'regression');

test.meta('tag', 'severe')(...)

2. Meta is not testRun info (since testRunInfo tells about the results of test run) but test info.

So we think the following solution will be better:

// test.js 
fixture `my fixture`
    .meta('field1', 'value1')
    .meta({ field2: 'value2', field3: 'value3' });

test
    .meta('field4', 'value4')
    ('Test1', async t => {
    // ... 
});

// custom-report.js
reportFixtureStart(name, path, meta) {
    this.currentFixtureName = name;
    this.currentFixtureMeta = meta;

    // ...
}
reportTestDone (name, testRunInfo, meta) {
    const testMeta = meta; // { field4: 'value4' }
    const fixtureMeta = this.currentFixtureMeta; // { field1: 'value1', field2: 'value2', field3: 'value3' }

    // ...
}

It will be consistent with the current reporting API.

[MatthewNielsen27]

@AlexanderMoskovkin I submitted a PR to support this issue #2298

[miherlosev]

Done in d4be3b1

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs or feature requests. For TestCafe API, usage and configuration inquiries, we recommend asking them on StackOverflow.