Kagome As a Server Side Tokenizer

Feeling Kagome Slow?

The goal of Kagome is to create an easy-to-use and fast tokenizer, which is why the dictionaries are integrated into the binary.

Kagome itself performs the tokenization quickly. However, the loading time of the application binary can be a bottleneck.

For example, the kagome command line tool is 124MB in size. This is because the binary includes two dictionaries, IPA and Uni.

This means that the kagome command is not suitable for use in shell scripts that are called many times in succession, as loading the binary files takes a considerable amount of time.

In that case, serving kagome as a server via kagome server command is a better option.

• See also the use of Kagome in WebAssembly and its responsiveness.

Benchmark

Check the "Requests per second" section of the Apache Bench results below.

• Machine Spec Tested:
  • macOS 12.7.4
  • MacBook Pro (Early 2015, Core i5 2.7 GHz, Mem 8 GB)

$ kagome version
2.9.5
   ipa v1.0.10
   uni v1.1.9

$ # Start the kagome server on port 6060 and detach it
$ kagome server -http=:6060 &

$ # How to request tokenization
$ curl -XPUT localhost:6060/tokenize -d'{"sentence":"すもももももももものうち", "mode":"normal"}' | jq .
**snip**

$ # Prepare data to PUT
$ echo '{"sentence":"すもももももももものうち", "mode":"normal"}' > put.json

$ # Run bench mark (expecting 100 users requesting 100 request at the same time)
$ ab -n10000 -c100 -uput.json http://localhost:6060/tokenize
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            6060

Document Path:          /tokenize
Document Length:        1705 bytes

Concurrency Level:      100
Time taken for tests:   2.054 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      18150000 bytes
Total body sent:        2050000
HTML transferred:       17050000 bytes
Requests per second:    4867.49 [#/sec] (mean)
Time per request:       20.544 [ms] (mean)
Time per request:       0.205 [ms] (mean, across all concurrent requests)
Transfer rate:          8627.43 [Kbytes/sec] received
                        974.45 kb/s sent
                        9601.88 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    7  34.4      5     641
Processing:     1   13  53.1      7     643
Waiting:        0   11  48.0      6     643
Total:          3   20  63.1     12     649

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     13
  75%     15
  80%     17
  90%     28
  95%     32
  98%     37
  99%     62
 100%    649 (longest request)