Commits on Jun 13, 2023

1. Improve TOKENIZER by 23% …

   From what I can see, this is done in linear time: 4*O(n)
   This tokenizer change converts that to something a little quicker: 3*O(n)
   
   Seems that not using a capture group and something other than split would be a big win.
   Other than that, the changes were meager.
   
   I used https://regex101.com/ (and pcre2) to evaluate the cost of the TOKENIZER.
   I verified with cruby 3.0.6
   
   ```
    /(%%\{[^\}]+\}|%\{[^\}]+\})/ =~ '%{{'*9999)+'}'
   
   /(%%\{[^\}]+\}|%\{[^\}]+\})/ ==> 129,990 steps
   /(%?%\{[^\}]+\})/            ==> 129,990 steps
   /(%%?\{[^\}]+\})/            ==>  99,992 steps (simple savings of 25%) <===
   /(%%?\{[^%}{]+\})/           ==>  89,993 steps (limiting variable contents has minimal gains)
   ```
   
   Also of note are the null/simple cases:
   
   ```
   /x/ =~ '%{{'*9999)+'}'
   /x/                          ==>  29,998 steps
   /(x)/                        ==>  59,996 steps
   /%{x/                        ==>  49,998 steps
   /(%%?{x)/                    ==>  89,993 steps
   ```
   
   And comparing against a the plain string of the same length.
   
   ```
   /x/ =~ 'abb'*9999+'c'
   
   /x/                          ==>  29,999
   /(%%?{x)/                    ==>  59,998
   /(%%?\{[^\}]+\})/            ==>  59,998
   /(%%\{[^\}]+\}|%\{[^\}]+\})/ ==>  89,997
   ```
   
   per ruby-i18n#667

   

   kbrock committed Jun 13, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for c5f9f49
   • Browse repository at this point

   Copy the full SHA

   c5f9f49 View commit details

   Browse the repository at this point in the history