Utilize processing server proxy to mets servers

[MehmedGIT]

Allow the Processing Server to accept general mets server-related TCP requests, and translate them to UDS requests. This feature is useful for cases when the worker is located on a remote host and wants to communicate with a UDS Mets Server. One main benefit of this approach is to avoid allocating separate ports for different mets servers.

This PR is still a draft and the implementation is still on a conceptual level. Please feel free to suggest any ideas.

[bertsky]

I still don't fully understand: if the METS Server user (here: tcp_mets caller) is remote (relative to the workspace), then how is this useful? All the files that the METS updates will relate to will also be remote, and we concluded it's not a good design to squeeze file contents through the METS Server earlier.

[MehmedGIT]

I still don't fully understand: if the METS Server user (here: tcp_mets caller) is remote (relative to the workspace), then how is this useful? All the files that the METS updates will relate to will also be remote, and we #966 (review) earlier.

The idea is that the processing workers could send requests to any Mets server through the Processing Server without allocating separate ports per Mets server on the host where the Processing Server is running.

You are right that it will not work when the Processing Server (Mets Servers) and the Processing Workers are on different hosts with the current setup. You are also right that it has been decided to not transfer files over to the Mets Server as discussed in #966.

The forwarding through the Processing Server as a proxy is supposed to be used when:

• there is a Network File System that is accessible by both the Mets Servers and Processing Workers
• when running docker containers on the same host - to avoid extra allocation of ports for the Mets Servers

@joschrew, is there anything more to add that I have missed?

[joschrew]

I imagine a setup where workers (and processing server) are on different vm's. The workspace is shared through NFS so that every processor has access to the same files. Currently this would not work with the Mets-Server as the unix-domain-sockets cannot be shared through NFS. With this PR it should be possible for workers on different vms to make requests to the Mets-Server.

[bertsky]

Thanks @MehmedGIT @joschrew for the explanation. Absolutely makes sense now – fantastic idea!

[joschrew]

I did some commits. It is now working for me with processors in docker-containers. I still want to change little things and do additional tests. After that I would change the pr's status for reviews.

[MehmedGIT]

I did some commits. It is now working for me with processors in docker-containers. I still want to change little things and do additional tests. After that I would change the pr's status for reviews.

Thanks.

[MehmedGIT]

It is still unclear to me why one of the processors sent the request to /tcp_mets/workspace_path instead of /tcp_mets, leading to 404 errors. However, the error I was getting was related to an outdated ocrd_all local installation. All redirections work as expected with the latest version of ocrd_all and core.

[kba]

Thanks, very thorough, ready for release