Add option to ignore columns in equality test

[brunocostalopes]

Describe the feature

I would like to be able to specify a list of columns to exclude on an equality test.

Describe alternatives you've considered

We can of course use compare_columns to list all columns, except the ones we want to exclude, but this leads to large yml files, difficult to maintain as our models evolve.

Additional context

Not database specific, it can be implemented across all different databases.

Who will this benefit?

This feature would be great for anyone who wants to compare data across environments where, for example, some of the fields are masked in one of the environments.
Another scenario where this can be useful is when updating the logic on a small subset of fields in a model and wanting to compare the development model with the production one, but obviously excluding the updated fields (which are expected to be different)

Are you interested in contributing this feature?

I have implemented this locally already. Happy to raise a PR if people think this feature is helpful.

[deanna-minnick]

@joellabes What do you think about moving forward with this? I'd be game to put some work into this in the next couple of weeks!

[joellabes]

@brunocostalopes thanks for this! I am very on board with this as an idea.

Some implementation notes:

• The new argument should be ignore_columns (as opposed to exclude; let me know if you have a different opinion)
• If both compare_columns and ignore_columns are provided, it should throw an error (see how union_relations does it)
• Please also add some integration tests which exercise the new behaviour. I went looking for the existing integration tests for this test and didn't find any, so instead here's an example from the recency test. Note in particular that you can check that the test catches failing records by setting error_if to <1.

Since it's your issue and you've got the code already to hand, you get first dibs on implementing it, but if you're not going to have time to do the other things then it looks like @deanna-minnick is also willing to lend a hand - just let us know on the issue thread.

Let me know if you run into any issues 🎉

[brunocostalopes]

@joellabes sounds good, thank you for the implementation notes. I'll take a first stab at it and if it seems like I'm stuck I'll ask @deanna-minnick for a bit of help.

[brunocostalopes]

@joellabes I've raised the PR.

Please feel free to comment and suggest any changes, particularly if you have any suggestions on how to compare the list of columns in the ignore list with the list returned by the adapter (feels a bit hacky with the lowercase but couldn't think of a better approach so maybe I'm just missing something).

EDIT: just realised that I was getting the quoted value from the column not the name 🤦 . Still, happy to hear what others think of what would be the best way of making that comparison. I guess we can have this discussion in the PR.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.