tests in amalgated source

[Thermi]

We need the tests in amalgated sources - or at least a version of the amalgated sources with the tests included (and being run against the amalgated code), because it is not known if libmdbx works correctly in different library constellations (e.g. different libc implementation, other Windows version ,...).
As of now, there is no reasonable way to get the tests in an "official" distributed ource code that is not compiled in "dev" mode so to say and also does not require manually specifying all the version parameters (both true for copies of the git archive that can be downloaded over Github, they don't contain the .git dir).

[erthink]

Okay, I'll try to explain again why, in fact, you don't need these tests in the amalgamated source code (i.e. amalgamated source code = the released code base for a users of library, but not a developers of libmdbx).

1. Some errors and inaccuracies may always remain. Therefore, regardless of libmdbx's tests, you should have your own tests (an application-level of which libmdbx is used), in order to test your use cases as fully as possible.
2. The libmdbx repository has a test framework and at the same time a test utility that allows me to run various test scenarios using command-line parameters. The source code of these tests is publicly available, but is intended for use by the library developers:
   • The tests are designed for long-term (from an hour to a week) stochastic execution with the iteration of various operating modes and parameters.
   • These tests are also used for fast (aka smoke) CI-checking of commits. For this reasons the make check target is available for convenience.
   • However, these tests are not intended (not oriented) for use by users, as requires the manual preparation of a test environment, a sufficiently long execution, manual investigation of failures under debugger, knowledge of many library internals, etc.
3. On supported/proven platforms, it is almost useless to check libmdbx with built-in tests:
   • libmdbx is checked by these tests quite regularly (after all significant changes), as well as before releases.
   • The tests work stochastically, but all their scenarios are repeated. So, on the one hand, mostly all code paths are well tested.
   • On the other hand, the probability of catching an undiscovered error is extremely small, since the tests in general are the same.
4. On new/unproven/unsupported platforms, it is almost useless to check libmdbx by yourself as a non-developer:
   • It is obvious that in such cases you should use the developer's version, i.e. the git repository but not an amalgamated source code.
   • Nonetheless, if target platform promises API compatibility with proven platform, then you may rely on your own application-level tests.
5. Through the above, I am trying to clarify some related points, but not to dissuade anyone from using internal libmdbx's tests.
   However, at the same time, only two methods of embedding/integrating libmdbx into other projects and/or libraries are still recommended:
   • As a copy of amalgamated source code without any internal tests.
   • As a git submodule with internal tests.
   • In addition, it is strongly recommended to use libmdbx as a shared library to ensure that only single version of the library exists/work in an address space of an any process.

[Thermi]

I'm using/packaging libmdbx for use in the Alpine Linux distribution which uses musl libc instead of GNU libc. There have been many issues found in software where it crashes (not the libc) because, for example, it exhaustes the stack (stack size on musl is just 128k, in GNU libc it's in the range of 2 to 10 MB.

I want to have the tests to catch regressions or new issues with new libc or libmdbx versions before a new fault is introduced with a new package version.
I assume you only test with GNU libc and on x86_64, so your tests do not cover that problem, or any problems that might occur on other CPU architectures that Alpine supports.

I'd like to have the standard tests from the unamalgated version so I can be sure that the tests cover the code that is built into the shared library, instead of some other code. Just having the tests be built/run from code that explicitely compiles from the amalgated sources is the best way to avoid human error (e.g. accidentally shipping a complete, unamalgated copy of the code with amalgated version and then running the tests against the unamalgated version).

I can not reasonably provide tests for all possible user applications and their explicite behaviour. That is why I'd only like to run the existing tests and see if they fail. I do not need any long term tests. The CI run time in the Alpine Linux CI is limited to an hour anyway. So that's the absolute maximum the test could run as.

We previously already found an issue with s390x, IIRC. So this is not a hypothetical problem.

[erthink]

Firstly, I want to repeat once again that two options are offered: either a git submodule with all tests, or amalgamated source code without internal tests. On the other hand, I do not intend to support any other options for a number of reasons.
Briefly:

• It is always easy to get an amalgamated version from git's repo just by running make dist on Linux, FreeBSD or OSX.
• libmdbx's own tests are focused on checking internal complex mechanisms (managing page lists, merging/splitting/rebalance pages, interacting via an lck-file, etc). However, it are not comprehensive, in the sense that ones are not intended for testing basic system libraries and/or services, and cannot replace your own tests in (your) specific use cases.
• The full and amalgamated versions of the source code have the same build methods (both via GNU make and CMake). So, with proper integration into your project, you can always easily switch between amalgamated and full versions, simply copying files when you need access to internal libmdbx's tests.

Thus, I am sure that the proposed two delivery options cover all rational cases, and there is no reason to complicate anything (i.e. do any additional work).

---

If you intend to use internal libmdbx tests to check the library's operation, then you should:

• pay attention to the test/long_stochastic.sh script, and to check, let it work for at least a couple of hours.
• think about supplementing the test scenarios with your own cases (for example, multithreaded ones) in C++, or (at least) adapt the noted script for your use cases.

Otherwise you will beat the air.

[erthink]

@Thermi, for now the devel branch have smoke and test targets in the GNUMakefile, and the make test perform test of libmdbx with a reasonable tradeoff of minimal/enough.

In other words, for minimal verification, you should use this or a more extensive testing scenario.