OrientDB next steps for the next release

[tglman]

Hi,

I wrote a blog post about the work happening and what is going to be in the next release here https://orientdb.dev/news/status-and-backlog/

Please check it and let me know any feedback, also happy to accept any contribution from anyone to make next release happen !

[andrii0lomakin]

Hi @tglman
Thank you for the clarification.
Could you clarify who are "we" in your blog post?
Could you introduce a team that will work on such a huge scope?

I am asking because I have seen only a single committer for years already.

You very often refer "we" in your posts, but I see only a single supporter, and all changes seen now are:

1. Minor bug fixes.
2. Dependency updates.

[tglman]

This project still have a few people with push rights on this repo, including you, also there are external contributors that do PR for fix specific problems that are more than happy to merge, anyone is welcome to contribute on the effort, this is all I can share on this.

[andrii0lomakin]

You wrote:

As today the integration of lucene in the engine is limited, it does not have durability guarantee, and does not have complete backup/restore operations support and as well is not integrated properly in the distributed environment, all of this will be worked on to make support of lucene index at the same level of native indexes

Could you share any design choices that will backup those plans ?

[tglman]

Hi, Yes,

Some refactor work has been already done on the develop branch to make the integration possible, the firs step is to provide a method in the OIndexEngine that receive all the changes and apply to the index, that do more or less the same logic of the OAbstractPaginatedStorage#applyTxChanges. Done that, the implementation of the Lucene IndexEngine will use lucene two phase transactions and than the sequential returned from the prepareCommit will be logged with a specific record in the WAL, this will allow to deterministically detect durning the recover/open process if the lucene indexes are at the same level of transaction of the main storage.

When detected if the lucene index is out of sync a index rebuild can be triggered.
It could also be possible to log the lucene key changes in the WAL for reapply them and that will avoid the rebuild, but more check need to be done on this solution to understand if is possible.

[andrii0lomakin]

Studio has been in low maintenance for a while

That is true. Could you introduce a contributor (and by this, I mean either person or company) who will work on this part to improve its maintenance?

[andrii0lomakin]

Fix major issues in query engine and query performance improvement

This paragraph does not provide any concrete plans.

1. Which optimization algorithms are going to be used?
2. What concrete problems will be fixed ?

From commits are seen there are quite minor improvements in this area.

[tglman]

Hi,

A lot of work is already been done on this, you can check the implementation differences between the query engine in 3.1.x, 3.2.x, and develop, a lot more need to be done, at first the work has been going for a while is migrate all the existing tests that used the old query engine to the new query engine, I started working on this more than 3 years ago with more than 5000 usages in the code, in the last develop we do have less than 300 uses of the old query engine in the test cases, while working on this I fixed also all the bug I found reproduced by those tests, and ported the relative fix to the last stable (3.2.x) so this I guess is the minor improvements you saw in the commits.

The main issues that need to solve are actually this: #9808 #10328

There are some implementations in the code that do the preliminary steps to fix 9808, the main problem is that for detect indexes is used an exponential algorithm (that allocate memory in an exponential way), an algorithm that is solve this problem and is not exponential is already wrote+tested+merged, but is not yet integrated, the integration will be done after the migration of the tests that use the legacy query engine will be complete.

There are additional improvement as well that can be done, especially on query profiling and error reports, but may be done after the release of the next major.

[andrii0lomakin]

Are there any companies that can support and ensure continuous development and the quality of the efforts that will be spent to implement such a big scope?

[tglman]

🤷

[andrii0lomakin]

Could you provide QA measures that you are going to use to ensure quality assurance that is needed to ensure quality characteristics of upcoming and current releases? I am pretty sure that running tests on GitHub actions is insufficient to ensure such quality.

[tglman]

Hi,

I do agree that the tests running on github actions are not complete, I do run before the release mvn clean install falisafe:integration-tests that ran for 5h+ and do additional tests to guarantee the quality, these tests are run periodically as well, mostly are triggered manually, also before the shipment of the release are run some minimal manual test to check the quality of the package.

I'm pretty sure you are pretty familiar with the process as we did it together many times, but is good also to make this information more public.

[andrii0lomakin]

Alas, that is not enough to ensure the user's data safety. :-( IMHO.

[tglman]

More than welcome to integrate and run additional tests, the integration tests are the base for everything, I do know the existence in the code of additional test that are run trough different profiles, like crash tests and secondary tests, feel free to share what would be what you would consider the right set of tests to run to guarantee data safety.