Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quieter logs in unit tests (145,594 lines -> 2,134 lines) #6809

Merged
merged 11 commits into from
Nov 23, 2020

Conversation

mrmr1993
Copy link
Member

@mrmr1993 mrmr1993 commented Nov 21, 2020

This PR

  • creates the Inline_test_quiet_logs library, to silence logs for passing unit tests
    • this uses a hack with overwriting the unix file descriptor underlying the stdout out_channel
    • this can be opened at the top of any module containing unit tests to silence the output on stdout from any passing test
    • failing tests have their stdout dumped at the end of the test
  • removes the verification batch size log
    • this never gave useful output -- our batch size is always 1 (although maybe we should look into why we never batch?)
    • this single log line accounted for 130428/145594 = ~90% lines in the log file here (80% of the file by size)
  • silences logs from some unit test (percentages are of the remaining logs after all previous removals)
    • Staged_ledger -- accounted for ~38%
    • Transition_handler -- accounted for ~17%
    • Bootstrap_controller -- accounted for ~16%
    • Consensus -- accounted for ~25%
    • Ledger_catchup -- accounted for ~14%
    • Transition_frontier -- accounted for ~17%

The log file size is reduced from 145594 lines to 2134, most of which is build logs. Byte-wise, this takes us down from 45,201,099 (44MB) to 562,455 (552KB).

Of the remaining lines:

  • 771 are C++ compilation
  • 426 are go: finding library vX.Y.Z
  • 256 are remaining log messages from tests
  • 171 are rust downloading and compiling libraries
  • 144 are OCaml warnings
  • 124 are buildkite/docker messages

(Most of the rest is a failing test that I was struggling to find amidst 44MB of logs).

Checklist:

  • Document code purpose, how to use it
    • Mention expected invariants, implicit constraints
  • Tests were added for the new behavior
    • Document test purpose, significance of failures
    • Test names should reflect their purpose
  • All tests pass (CI will check this if you didn't)
  • Serialized types are in stable-versioned modules
  • Does this close issues? List them:

@mrmr1993 mrmr1993 added the ci-build-me Add this label to trigger a circle+buildkite build for this branch label Nov 21, 2020
@mrmr1993 mrmr1993 requested a review from a team as a code owner November 21, 2020 03:14
@mergify mergify bot merged commit e7025a2 into develop Nov 23, 2020
@mergify mergify bot deleted the feature/quieten-logs-in-unit-tests branch November 23, 2020 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-build-me Add this label to trigger a circle+buildkite build for this branch ready-to-merge-into-develop
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants