Performance when Generating Newspaper Processes

[thomaslow]

Related and possibly fixed issues:

• Performance problem: Creating newspaper processes - creation of processes is very slow #4760
• Cannot create many newspaper processes #4221

The problem is caused by a slow indexing performance of projects in ElasticSearch. The following snippet shows how a project with many processes is indexed at the moment:

{
	"_index": "kitodo_project",
	"_type": "_doc",
	"_id": "1",
	"_score": 1,
	"_source": {
		"active": true,
		"title": "Example Project",
		"processes": [
			{
				"id": 1,
				"title": "Process Title"
			},
			{
				"id": 2,
				"title": "Another Process Title"
			},
			# ...
			# repeating potentially over 10.000 times depending on the number of processes
			# ...
		],
		"startDate": "2016-01-01 00:00:00",
		"endDate": "2019-12-31 00:00:00",
		"numberOfVolumes": 0,
		"templates": [...],
		"users": [...],
		"folder": [...],
		"numberOfPages": 0,
		"metsRightsOwner": "Digital Library Kitodo",
		"client.name": "Client_ChangeMe",
		"client.id": 1
	}
}

For large projects with many >10.000 of processes, the search itself is still fast (ElasticSearch reports a query time of 2ms), howewer the json document becomes huge and will take a lot of time to parse. During the newspaper generating process, the corresponding project is saved repeatedly after generating a new newspaper process, so potentially hundereds of times, each time the project is indexed again with a single new entry added to the list of processes.

The proposed solution removes the list of processes from the index mapping, and replaces it with a property hasProcesses, which is required to allow to disable the "delete project icon" when showing a list of projects.

A disadvantage of this solution is that projects can no longer be searched for based on the name of their processes. As of my knowledge, the user interface currently does not support a keyword-based search for projects anyway. However, if it is desired to support such a search scenerio in the future, the next best solution would be to improve the newspaper generation task to not save a project multiple times while generating new processes (which has problems as well, e.g., an inconsistent database during the generation task).

A positive side effect of this solution is that both the indexing and query time of projects has improved, which improves the performance when re-indexing all projects, or even loading the dashboard user interface.

@solth
This might also be a problem in the Hibernate-Search branch, see line:

@IndexedEmbedded(includePaths = {"id", "title"})

effective-webwork@1a5de7d#diff-5f7246e37b075cd985e0d6b3bdeeb919b5368811377ab0a3517df08897f3c576R111