Investigate runtime grey box testing (db queries/file system/security etc.)

[stewartie4]

Now that we have good black box coverage, I think it would be useful to introduce some grey box tests.
These would be against the full running system, but have access to some internal state.
Some things I am thinking of but we can expand:

• Query DB state, is everything as expected
• Query M/S/B filesystem
• Network traffic between services
  etc.