🐛 Bug: Tap reporter doesn't adhere to spec

[dszakallas]

It currently generates output like this:

1..1
ok 1 describe it
# tests 1
# pass 1
# fail 0

for

describe('describe', () => {
  it('it', () => {
    expect(true).to.eql(true)
  })
})

It just flattens out describes and prepends them the to test name.
This makes the tap-spec reporter unable to recognize test-cases.

echo '1..1\nok 1 describe it\n# tests 1\n# pass 1\n# fail 0' >> test
cat test | node_modules/.bin/tap-spec
    ✔ describe it


  ✖ No tests found

I think a viable alternative would be to print diagnostic messages for describe blocks.

1..1
# describe
ok 1 it
# tests 1
# pass 1
# fail 0

prints

>cat test | node_modules/.bin/tap-spec

  describe

    ✔ it


  total:     1
  passing:   1
  duration:  8ms

Nested describes should be concatened though I suppose.

[dszakallas]

see scottcorgan/tap-spec#47

[boneskull]

seems reasonable, thanks for reporting

[stale]

I am a bot that watches issues for inactivity.
This issue hasn't had any recent activity, and I'm labeling it stale. In 14 days, if there are no further comments or activity, I will close this issue.
Thanks for contributing to Mocha!

[boneskull]

not stale

[mollstam]

I'm looking at bringing the TAP reporter up to v13 spec (see #3452) and trying to conceive a Good Solution for this issue.

Currently this fixture

'use strict';

describe('Animals', function() {

  it('should consume organic material', function(done) { done(); });
  it('should breathe oxygen', function(done) { done(); });
  it('should be able to move', function(done) { done(); });
  it('should reproduce sexually', function(done) { done(); });
  it('should grow from a hollow sphere of cells', function(done) { done(); });

  describe('Vertebrates', function() {
    describe('Mammals', function() {
      it('should give birth to live young', function(done) { done(); });
      describe('Blue Whale', function() {
        it('should be the largest of all mammals', function(done) { done(); });
        it('should have a body in some shade of blue', function(done) { done(); });
      });
    });
    describe('Birds', function() {
      it('should have feathers', function(done) { done(); });
      it('should lay hard-shelled eggs', function(done) { done(); });
    });
  });

  describe('Tardigrades', function() {
    it('should answer to "water bear"', function(done) { done(); });
    it('should be able to survive global mass extinction events', function(done) { done(); });
  });
});

results in the following TAP report output

1..12
ok 1 Animals should consume organic material
ok 2 Animals should breathe oxygen
ok 3 Animals should be able to move
ok 4 Animals should reproduce sexually
ok 5 Animals should grow from a hollow sphere of cells
ok 6 Animals Vertebrates Mammals should give birth to live young
ok 7 Animals Vertebrates Mammals Blue Whale should be the largest of all mammals
ok 8 Animals Vertebrates Mammals Blue Whale should have a body in some shade of blue
ok 9 Animals Vertebrates Birds should have feathers
ok 10 Animals Vertebrates Birds should lay hard-shelled eggs
ok 11 Animals Tardigrades should answer to "water bear"
ok 12 Animals Tardigrades should be able to survive global mass extinction events
# tests 12
# pass 12
# fail 0

which piped into the latest version of tap-spec looks pretty flat

    ✔ Animals should consume organic material
    ✔ Animals should breathe oxygen
    ✔ Animals should be able to move
    ✔ Animals should reproduce sexually
    ✔ Animals should grow from a hollow sphere of cells
    ✔ Animals Vertebrates Mammals should give birth to live young
    ✔ Animals Vertebrates Mammals Blue Whale should be the largest of all mammals
    ✔ Animals Vertebrates Mammals Blue Whale should have a body in some shade of blue
    ✔ Animals Vertebrates Birds should have feathers
    ✔ Animals Vertebrates Birds should lay hard-shelled eggs
    ✔ Animals Tardigrades should answer to "water bear"
    ✔ Animals Tardigrades should be able to survive global mass extinction events


  total:     12
  passing:   12
  duration:  246ms

while the same fixture using the spec reporter directly has a nice hierarchy

  Animals
    ✓ should consume organic material
    ✓ should breathe oxygen
    ✓ should be able to move
    ✓ should reproduce sexually
    ✓ should grow from a hollow sphere of cells
    Vertebrates
      Mammals
        ✓ should give birth to live young
        Blue Whale
          ✓ should be the largest of all mammals
          ✓ should have a body in some shade of blue
      Birds
        ✓ should have feathers
        ✓ should lay hard-shelled eggs
    Tardigrades
      ✓ should answer to "water bear"
      ✓ should be able to survive global mass extinction events


  12 passing (11ms)

---

Possible solutions

So this is where we are. Now a first approach would be to output describes as # comments:

TAP version 13
1..12
# Animals
ok 1 should consume organic material
ok 2 should breathe oxygen
ok 3 should be able to move
ok 4 should reproduce sexually
ok 5 should grow from a hollow sphere of cells
#   Vertebrates
#   Mammals
ok 6 should give birth to live young
#       Blue Whale
ok 7 should be the largest of all mammals
ok 8 should have a body in some shade of blue
#     Birds
ok 9 should have feathers
ok 10 should lay hard-shelled eggs
#   Tardigrades
ok 11 should answer to "water bear"
ok 12 should be able to survive global mass extinction events
# tests 12
# pass 12
# fail 0

  Animals
    ✔  should consume organic material
    ✔  should breathe oxygen
    ✔  should be able to move
    ✔  should reproduce sexually
    ✔  should grow from a hollow sphere of cells

  Vertebrates

  Mammals
    ✔  should give birth to live young

  Blue Whale
    ✔  should be the largest of all mammals
    ✔  should have a body in some shade of blue

  Birds
    ✔  should have feathers
    ✔  should lay hard-shelled eggs

  Tardigrades
    ✔  should answer to "water bear"
    ✔  should be able to survive global mass extinction events

passed: 12  failed: 0  of 12 tests  (10ms)

All of 12 tests passed!

I kind of like it over the original flat output, but the trimming of prefixed whitespace is a bit sad (limitation in tap-spec I guess?). But hey, if we insert a prefix in the comment it won't trim following whitespace:

TAP version 13
1..12
# Animals
ok 1 should consume organic material
ok 2 should breathe oxygen
ok 3 should be able to move
ok 4 should reproduce sexually
ok 5 should grow from a hollow sphere of cells
# -  Vertebrates
# -    Mammals
ok 6 should give birth to live young
# -      Blue Whale
ok 7 should be the largest of all mammals
ok 8 should have a body in some shade of blue
# -    Birds
ok 9 should have feathers
ok 10 should lay hard-shelled eggs
# -  Tardigrades
ok 11 should answer to "water bear"
ok 12 should be able to survive global mass extinction events
# tests 12
# pass 12
# fail 0

  Animals
    ✔  should consume organic material
    ✔  should breathe oxygen
    ✔  should be able to move
    ✔  should reproduce sexually
    ✔  should grow from a hollow sphere of cells

  -  Vertebrates

  -    Mammals
    ✔  should give birth to live young

  -      Blue Whale
    ✔  should be the largest of all mammals
    ✔  should have a body in some shade of blue

  -    Birds
    ✔  should have feathers
    ✔  should lay hard-shelled eggs

  -  Tardigrades
    ✔  should answer to "water bear"
    ✔  should be able to survive global mass extinction events

passed: 12  failed: 0  of 12 tests  (10ms)

All of 12 tests passed!

Cool… but the actual results will never be paddded, and now Blue Whale is deeper in that its results, no good.

---

In the end it feels like we're trying to work around limitations in tap-spec, and the TAP specification v13 has no concept of suites and subtests as far a I can tell. Honestly, it would be more reasonable in my opinion to make a TAP consumer (an alternative to tap-spec in this case) that could take as input the original flat structure and deduce a hierarchy from the common prefixes and render it with indentation or whatever.

Any thoughts?

[mollstam]

Looking at the spec I think the best solution is saving "depth" or "parent" in the YAML block segment following the test. I've only seen it used in the spec and in implementations for extra information regarding failed tests (message, source line, stack) but the spec allows it after any test line.

But that requires TAP consumers to implement support for it, but I mean it wouldn't be that crazy.

---

For reference I recreated the fixture in tape, which does have the concept of subtests, but the report is flattened…

var test = require('tape');

test('Animals', function(a) {
  a.pass('should consume organic material');
  a.pass('should breathe oxygen');
  a.pass('should be able to move');
  a.pass('should reproduce sexually');
  a.pass('should grow from a hollow sphere of cells');

  a.test('Vertebrates', function(b) {
    b.test('Mammals', function(c) {
      c.pass('should give birth to live young');
      c.test('Blue Whale', function(d) {
        d.pass('should be the largest of all mammals');
        d.pass('should have a body in some shade of blue');
        d.end();
      });
      c.end();
    });
    b.test('Birds', function(e) {
      e.pass('should have feathers');
      e.pass('should lay hard-shelled eggs');
      e.end();
    });
  });

  a.test('Tardigrades', function(f) {
    f.pass('should answer to "water bear"');
    f.pass('should be able to survive global mass extinction events');
    f.end();
  });

  a.end();
});

TAP version 13
# Animals
ok 1 should consume organic material
ok 2 should breathe oxygen
ok 3 should be able to move
ok 4 should reproduce sexually
ok 5 should grow from a hollow sphere of cells
# Vertebrates
# Mammals
ok 6 should give birth to live young
# Blue Whale
ok 7 should be the largest of all mammals
ok 8 should have a body in some shade of blue
# Birds
ok 9 should have feathers
ok 10 should lay hard-shelled eggs
# Tardigrades
ok 11 should answer to "water bear"
ok 12 should be able to survive global mass extinction events

1..12
# tests 12
# pass  12

# ok

---

node-tap has the concept of subtests, but I'm not sure what's going on here. This isn't spec compliant if I'm not mistaken: https://www.node-tap.org/subtests/

🦊