Unrelated tests are bleeding into each other

[smashedtoatoms]

Checklist

• I have read Caveats documentation and didn't find a solution for this problem there.

Bug description

Unrelated tests are bleeding data into each other. Both tests pass when run separately. When they are run together, "returns a generic error when dynamodb fails to delete" fails consistently with a "fake ddb scene error" that only exists in the other unrelated test. My current workaround is to put the second test above the first, which resolves it. This is likely related to the caveats in some way, though it is peculiar behavior because it seems the mock resets are ignored or don't matter. When I don't use module-level mock, and put an individual in each tests, the same problem occurs. There is an implicit global state that is hard to reason about.

Environment

• Node version: 14.18.1
• Typescript version: 4.4.4
• AWS SDK v3 Client mock version: 0.5.6
• AWS JS SDK libs and versions: @aws-sdk/* 3.41.0

Here is an example test file, in case I am doing something obviously dumb:

import {
  CloudControlClient,
  DeleteResourceCommand,
  GetResourceRequestStatusCommand,
} from '@aws-sdk/client-cloudcontrol';
import { AttributeValue, DeleteItemCommand, DynamoDBClient, GetItemCommand } from '@aws-sdk/client-dynamodb';
import { mockClient } from 'aws-sdk-client-mock';
import faker from 'faker';
import { ulid } from 'ulid';

import { NodeNotFoundError, SceneNotFoundError } from '../../lib/errors';
import { handler } from '../nodeDelete';

const ddbMock = mockClient(DynamoDBClient);
const ccMock = mockClient(CloudControlClient);

function getDdbSceneOutput(accountId: string, sceneId: string): { [key: string]: AttributeValue } {
  return {
    accountId: { S: accountId },
    apiKey: { S: faker.internet.password() },
    createdAt: { S: faker.date.recent().toISOString() },
    dataPlaneAccountId: { S: faker.internet.password() },
    eventBus: { S: faker.internet.password() },
    id: { S: sceneId },
    managementRole: { S: faker.internet.password() },
    realtimeEventWsUrl: { S: faker.internet.password() },
    runtimeRole: { S: faker.internet.password() },
    status: { S: 'RUNNING' },
  };
}

function getDdbNodeOutput(sceneId: string, nodeId: string): { [key: string]: AttributeValue } {
  return {
    awsIdentifier: { S: faker.internet.password() },
    awsOperationStatus: { S: faker.internet.password() },
    awsState: { S: '{}' },
    awsTypeName: { S: 'AWS::Lambda::Function' },
    createdAt: { S: faker.date.recent().toISOString() },
    id: { S: nodeId },
    name: { S: faker.name.jobDescriptor() },
    sceneId: { S: sceneId },
    state: { S: 'RUNNING' },
  };
}

describe('tests nodeDelete', () => {
  beforeEach(() => {
    ddbMock.reset();
    ccMock.reset();
  });

  test('returns a node not found error when dynamo fails to return a node', async () => {
    const accountId = ulid();
    const sceneId = ulid();
    const nodeId = ulid();
    const tableName = 'fakeTableName';
    const event = { accountId, nodeId, sceneId, tableName };
    ddbMock
      .on(GetItemCommand, {
        Key: {
          PK: { S: `ACCOUNT#${accountId}` },
          SK: { S: `SCENE#${sceneId}` },
        },
        TableName: tableName,
      })
      .rejects(new Error('fake ddb scene get error'))
      .on(GetItemCommand, {
        Key: {
          PK: { S: `ACCOUNT#${accountId}|SCENE#${sceneId}` },
          SK: { S: `NODE#${nodeId}` },
        },
        TableName: tableName,
      })
      .resolves({});
    await expect(handler(event)).rejects.toThrow(NodeNotFoundError);
  });

  test('returns a generic error when dynamodb fails to delete', async () => {
    const accountId = ulid();
    const sceneId = ulid();
    const nodeId = ulid();
    const tableName = 'fakeTableName';
    const event = { accountId, nodeId, sceneId, tableName };
    ddbMock
      .on(GetItemCommand, {
        Key: {
          PK: { S: `ACCOUNT#${accountId}` },
          SK: { S: `SCENE#${sceneId}` },
        },
        TableName: tableName,
      })
      .resolves({ Item: getDdbSceneOutput(nodeId, sceneId) })
      .on(GetItemCommand, {
        Key: {
          PK: { S: `ACCOUNT#${accountId}|SCENE#${sceneId}` },
          SK: { S: `NODE#${nodeId}` },
        },
        TableName: tableName,
      })
      .resolves({ Item: getDdbNodeOutput(nodeId, sceneId) })
      .on(DeleteItemCommand, {
        Key: {
          PK: { S: `ACCOUNT#${accountId}|SCENE#${sceneId}` },
          SK: { S: `NODE#${nodeId}` },
        },
        TableName: tableName,
      })
      .rejects(new Error('fake ddb node delete error'));
    ccMock
      .on(DeleteResourceCommand)
      .resolves({ ProgressEvent: {} })
      .on(GetResourceRequestStatusCommand)
      .resolves({ ProgressEvent: {} });
    await expect(handler(event)).rejects.toThrow('fake ddb node delete error');
  });
});

[m-radzikowski]

Hey. What testing tool are you using? Mocha, Jest, etc? Please take a look at #64 if it solves your problem.

[smashedtoatoms]

I'm using Jest. Sorry about that. That's a pretty big detail to leave out. Let me try this and see what it does.

[smashedtoatoms]

Hi, I don't think this is related, but I could be doing something wrong. If you look at my example, you'll notice I am not using any instance mocks. It appears to be bleeding data across type mocks, which seems to be making the order of my tests matter. So far I have been able to work around it by changing the orders of my tests. Oddly enough, if I don't use module-scoped mocks, it still happens. What I mean is, if I create a mockClient in each test individually, I get the same behavior as if I declare them at the top and reuse them. It's the same way if I use something like lodash to clone them, so there appears to be some kind of module-level state persistence happening.

[m-radzikowski]

I'm unable to reproduce the issue. I've created a simple handler() function that does the operations you mocked, I run both tests together and it works well.

Please see https://github.com/m-radzikowski/aws-sdk-client-mock-issue65 - fork it or make PR with changes to reproduce the issue, if you can.

[m-radzikowski]

Please let me know if you reproduce the issue based on the repo I linked above, I'm unable to reproduce it.

[smashedtoatoms]

Thanks man! I'm really sorry I didn't get back to this in a reasonable amount of time. If it crops up again I will open a new ticket. Thanks so much and sorry to waste your time.