Multithreading issue

[lumpidu]

When running the gRPC service and multiple clients connecting at the same time , there are exceptions thrown from time to time. I.e. it's currently not reliable for multi-threading.

tts-frontend-service    | ERROR:grpc._server:Exception calling application: Already borrowed
tts-frontend-service    | Traceback (most recent call last):
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/grpc/_server.py", line 443, in _call_behavior
tts-frontend-service    |     response_or_iterator = behavior(argument, context)
tts-frontend-service    |   File "src/tts_frontend_server.py", line 64, in NormalizeTokenwise
tts-frontend-service    |     normalized = self.normalizer.normalize_tokenwise(request.content, domain)
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/regina_normalizer/main.py", line 122, in normalize_tokenwise
tts-frontend-service    |     normalized_tuples = nf.handle_sentence_tokenwise(prenorm_sent, domain, self.tagger)
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/regina_normalizer/number_functions.py", line 182, in handle_sentence_tokenwise
tts-frontend-service    |     tagsent = tagger.tag_sent(sentsplit)
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/pos/api.py", line 84, in tag_sent
tts-frontend-service    |     return self.tag_bulk((tuple(sent),), batch_size=1)[0]
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/pos/api.py", line 100, in tag_bulk
tts-frontend-service    |     return self._infer(ds, batch_size=batch_size).get_field(field=Fields.Tags)
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/pos/api.py", line 50, in _infer
tts-frontend-service    |     chunked_ds = chunk_dataset(
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/pos/data/dataset.py", line 69, in chunk_dataset
tts-frontend-service    |     lengths = get_adjusted_lengths(tokens, tokenizer, max_sequence_length=max_sequence_length)
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/pos/data/dataset.py", line 41, in get_adjusted_lengths
tts-frontend-service    |     encodings = [tokenizer.encode_plus(" ".join(sentence), return_offsets_mapping=True) for sentence in sentences]
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/pos/data/dataset.py", line 41, in <listcomp>
tts-frontend-service    |     encodings = [tokenizer.encode_plus(" ".join(sentence), return_offsets_mapping=True) for sentence in sentences]
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2444, in encode_plus
tts-frontend-service    |     return self._encode_plus(
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 448, in _encode_plus
tts-frontend-service    |     batched_output = self._batch_encode_plus(
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 367, in _batch_encode_plus
tts-frontend-service    |     self.set_truncation_and_padding(
tts-frontend-service    |   File "/usr/local/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 325, in set_truncation_and_padding
tts-frontend-service    |     self._tokenizer.no_truncation()
tts-frontend-service    | RuntimeError: Already borrowed

One quick fix is to disable multithreading for gRPC globally, so only one request at a time could be served. One can achieve that by executing the following when executing the gRPC main loop:

grpc.server(futures.ThreadPoolExecutor(max_workers=1))

But of course, enabling multi-threaded work-loads would be best.