Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

search query slowness due to athor replacement #1520

Closed
meskio opened this issue Jun 1, 2020 · 3 comments · Fixed by #1523
Closed

search query slowness due to athor replacement #1520

meskio opened this issue Jun 1, 2020 · 3 comments · Fixed by #1523

Comments

@meskio
Copy link
Contributor

meskio commented Jun 1, 2020

Describe the bug
If thread_authors_replace_me is set every search buffer reload will need to read every single email from disk. This makes the search very slow mostly in the case of a lot of encrypted emails.

Software Versions

  • Python version: 3.8.3
  • Notmuch version: 0.29.1
  • Alot version: git master (0.9.1 + 2348014)

To Reproduce

  1. Set thread_authors_replace_me to true
  2. do a search including a lot of encrypted email. It will take some seconds to load while gpg is working.

I'm personally happy to keep thread_authors_replace_me to false in my setup, if I'm the only one affected by it we can close the issue.

@paretje
Copy link
Contributor

paretje commented Jun 7, 2020

I have the same issue. It doesn't need to be encrypted though, signed messages trigger this as well. Quite a lot of time is spent loading the email's from disk, and in gpg (getting the key and verifying signatures). None of this seems necessary, and is a regression from previous versions. I might have a look later on.

These are the top calls:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    472/1    0.003    0.000   40.620   40.620 {built-in method builtins.exec}
        1    0.000    0.000   40.620   40.620 <string>:1(<module>)
        1    0.000    0.000   40.620   40.620 /usr/lib/python3.8/runpy.py:197(run_module)
        1    0.000    0.000   40.619   40.619 /usr/lib/python3.8/runpy.py:64(_run_code)
        1    0.000    0.000   40.619   40.619 /home/kevin/vcs/once/alot/alot/__main__.py:4(<module>)
        1    0.000    0.000   40.188   40.188 /home/kevin/vcs/once/alot/alot/__main__.py:86(main)
        1    0.000    0.000   37.591   37.591 /home/kevin/vcs/once/alot/alot/ui.py:47(__init__)
        1    0.000    0.000   37.590   37.590 /usr/lib/python3/dist-packages/urwid/main_loop.py:276(run)
        1    0.000    0.000   37.590   37.590 /usr/lib/python3/dist-packages/urwid/main_loop.py:374(_run)
        1    0.000    0.000   37.589   37.589 /usr/lib/python3/dist-packages/urwid/main_loop.py:1328(run)
        1    0.000    0.000   37.589   37.589 /usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py:265(run)
        1    0.000    0.000   37.588   37.588 /usr/lib/python3.8/asyncio/base_events.py:557(run_forever)
       35    0.001    0.000   37.588    1.074 /usr/lib/python3.8/asyncio/base_events.py:1784(_run_once)
       44    0.000    0.000   26.248    0.597 /usr/lib/python3.8/asyncio/events.py:79(_run)
       44    0.000    0.000   26.248    0.597 {method 'run' of 'Context' objects}
       15    0.000    0.000   26.141    1.743 /home/kevin/vcs/once/alot/alot/ui.py:244(apply_commandline)
    17/15    0.000    0.000   26.133    1.742 /home/kevin/vcs/once/alot/alot/ui.py:704(apply_command)
       16    0.000    0.000   25.903    1.619 /usr/lib/python3/dist-packages/urwid/main_loop.py:574(draw_screen)
        6    0.000    0.000   25.890    4.315 /home/kevin/vcs/once/alot/alot/ui.py:646(update)
  6458/16    0.021    0.000   25.813    1.613 /usr/lib/python3/dist-packages/urwid/widget.py:138(cached_render)
  2716/11    0.007    0.000   25.813    2.347 /usr/lib/python3/dist-packages/urwid/decoration.py:219(render)
       11    0.000    0.000   25.787    2.344 /usr/lib/python3/dist-packages/urwid/container.py:1062(render)
       11    0.000    0.000   25.782    2.344 /home/kevin/vcs/once/alot/alot/buffers/buffer.py:18(render)
       10    0.001    0.000   25.779    2.578 /usr/lib/python3/dist-packages/urwid/listbox.py:462(render)
      326    0.002    0.000   25.461    0.078 /home/kevin/vcs/once/alot/alot/walker.py:78(_get_next_item)
    23/19    0.006    0.000   25.330    1.333 /usr/lib/python3/dist-packages/urwid/listbox.py:327(calculate_visible)
      326    0.001    0.000   25.277    0.078 /home/kevin/vcs/once/alot/alot/widgets/search.py:20(__init__)
      326    0.008    0.000   25.273    0.078 /home/kevin/vcs/once/alot/alot/widgets/search.py:31(rebuild)
     1568    0.002    0.000   25.263    0.016 /home/kevin/vcs/once/alot/alot/walker.py:61(_get_at_pos)
     1439    0.001    0.000   25.253    0.018 /home/kevin/vcs/once/alot/alot/walker.py:45(get_next)
     1304    0.009    0.000   24.766    0.019 /home/kevin/vcs/once/alot/alot/widgets/search.py:116(build_text_part)
     1304    0.005    0.000   24.688    0.019 /home/kevin/vcs/once/alot/alot/widgets/search.py:193(prepare_string)
      326    0.001    0.000   24.665    0.076 /home/kevin/vcs/once/alot/alot/widgets/search.py:177(prepare_authors_string)
      326    0.002    0.000   24.664    0.076 /home/kevin/vcs/once/alot/alot/db/thread.py:177(get_authors_string)
      326    0.006    0.000   24.648    0.076 /home/kevin/vcs/once/alot/alot/db/thread.py:148(get_authors)
      326    0.004    0.000   24.555    0.075 /home/kevin/vcs/once/alot/alot/db/thread.py:224(get_messages)
  991/436    0.015    0.000   24.326    0.056 /home/kevin/vcs/once/alot/alot/db/thread.py:236(accumulate)
      991    0.027    0.000   24.268    0.024 /home/kevin/vcs/once/alot/alot/db/message.py:29(__init__)
      991    0.023    0.000   23.353    0.024 /home/kevin/vcs/once/alot/alot/db/message.py:99(get_email)
      991    0.007    0.000   23.277    0.023 /home/kevin/vcs/once/alot/alot/db/utils.py:302(decrypted_message_from_bytes)
       14    0.000    0.000   21.436    1.531 /home/kevin/vcs/once/alot/alot/ui.py:193(_apply_fire)
        4    0.000    0.000   21.381    5.345 /home/kevin/vcs/once/alot/alot/commands/globals.py:170(apply)
        5    0.000    0.000   20.719    4.144 /usr/lib/python3/dist-packages/urwid/listbox.py:710(_set_focus_complete)
      991    0.011    0.000   16.882    0.017 /home/kevin/vcs/once/alot/alot/db/utils.py:251(_decrypted_message_from_message)
      230    0.047    0.000   16.110    0.070 /home/kevin/vcs/once/alot/alot/db/utils.py:98(_handle_signatures)
       35    0.001    0.000   11.338    0.324 /usr/lib/python3.8/selectors.py:451(select)
       35   11.337    0.324   11.337    0.324 {method 'poll' of 'select.epoll' objects}
      230    0.016    0.000   10.054    0.044 /home/kevin/vcs/once/alot/alot/db/utils.py:38(add_signature_headers)
      226    0.008    0.000    9.964    0.044 /home/kevin/vcs/once/alot/alot/crypto.py:29(get_key)
      226    0.012    0.000    8.585    0.038 /usr/lib/python3/dist-packages/gpg/core.py:1201(get_key)
      226    0.006    0.000    8.554    0.038 /usr/lib/python3/dist-packages/gpg/gpgme.py:1336(gpgme_get_key)
      226    8.548    0.038    8.548    0.038 {built-in method gpg._gpgme.gpgme_get_key}
     1598    0.022    0.000    6.913    0.004 /usr/lib/python3/dist-packages/gpg/core.py:137(_funcwrap)
      991    0.009    0.000    6.389    0.006 /usr/lib/python3.8/email/__init__.py:40(message_from_bytes)
      991    0.002    0.000    6.377    0.006 /usr/lib/python3.8/email/parser.py:114(parsebytes)
      991    0.096    0.000    6.325    0.006 /usr/lib/python3.8/email/parser.py:59(parsestr)
      991    0.013    0.000    6.229    0.006 /usr/lib/python3.8/email/parser.py:41(parse)
     8301    0.010    0.000    5.830    0.001 /usr/lib/python3.8/email/feedparser.py:173(feed)
      230    0.009    0.000    5.739    0.025 /home/kevin/vcs/once/alot/alot/crypto.py:190(verify_detached)
     9292    0.010    0.000    5.550    0.001 /usr/lib/python3.8/email/feedparser.py:178(_call_parse)
21914/9292    0.444    0.000    5.540    0.001 /usr/lib/python3.8/email/feedparser.py:218(_parsegen)
    20280    0.066    0.000    4.765    0.000 /usr/lib/python3.8/email/message.py:462(get)
        1    0.000    0.000    4.704    4.704 /home/kevin/vcs/once/alot/alot/commands/globals.py:110(apply)
        1    0.000    0.000    4.703    4.703 /home/kevin/vcs/once/alot/alot/ui.py:403(buffer_open)
        1    0.000    0.000    4.703    4.703 /home/kevin/vcs/once/alot/alot/ui.py:460(buffer_focus)
    18233    0.043    0.000    4.675    0.000 /usr/lib/python3.8/email/policy.py:150(header_fetch_parse)
    18693    0.036    0.000    4.653    0.000 /usr/lib/python3.8/email/headerregistry.py:592(__call__)
      230    0.011    0.000    4.344    0.019 /usr/lib/python3/dist-packages/gpg/core.py:501(verify)
    15865    0.038    0.000    4.243    0.000 /usr/lib/python3.8/email/message.py:564(get_content_type)
      230    0.008    0.000    4.242    0.018 /usr/lib/python3/dist-packages/gpg/gpgme.py:1794(gpgme_op_verify)
      230    4.234    0.018    4.234    0.018 {built-in method gpg._gpgme.gpgme_op_verify}
    18693    0.083    0.000    4.153    0.000 /usr/lib/python3.8/email/headerregistry.py:195(__new__)
    17812    0.105    0.000    3.820    0.000 /usr/lib/python3.8/email/headerregistry.py:444(parse)
      456    0.011    0.000    2.758    0.006 /usr/lib/python3/dist-packages/gpg/core.py:185(__init__)
     6384    0.019    0.000    2.730    0.000 /usr/lib/python3/dist-packages/gpg/core.py:164(__setattr__)
      456    0.005    0.000    2.670    0.006 /usr/lib/python3/dist-packages/gpg/core.py:1131(home_dir)
      456    0.006    0.000    2.664    0.006 /usr/lib/python3/dist-packages/gpg/core.py:1349(set_engine_info)
      456    0.015    0.000    2.646    0.006 /usr/lib/python3/dist-packages/gpg/gpgme.py:1069(gpgme_ctx_set_engine_info)
      456    2.631    0.006    2.631    0.006 {built-in method gpg._gpgme.gpgme_ctx_set_engine_info}
        1    0.000    0.000    2.525    2.525 /home/kevin/.config/alot/hooks.py:56(exit)
        1    0.000    0.000    2.525    2.525 /usr/lib/python3.8/subprocess.py:332(call)
        6    0.000    0.000    2.520    0.420 /usr/lib/python3.8/subprocess.py:1074(wait)
        6    0.000    0.000    2.520    0.420 /usr/lib/python3.8/subprocess.py:1772(_wait)
        4    0.000    0.000    2.520    0.630 /usr/lib/python3.8/subprocess.py:1759(_try_wait)
        4    2.520    0.630    2.520    0.630 {built-in method posix.waitpid}
     8195    0.037    0.000    2.106    0.000 /usr/lib/python3.8/email/message.py:588(get_content_maintype)
    17572    0.154    0.000    1.832    0.000 /usr/lib/python3.8/email/_header_value_parser.py:2612(parse_content_type_header)
  1008842    0.274    0.000    1.204    0.000 /usr/lib/python3.8/email/feedparser.py:128(__next__)
    17427    0.044    0.000    1.154    0.000 /usr/lib/python3.8/email/_header_value_parser.py:2544(parse_mime_parameters)
    27655    0.104    0.000    1.088    0.000 /usr/lib/python3.8/email/_header_value_parser.py:2402(get_parameter)

@paretje
Copy link
Contributor

paretje commented Jun 7, 2020

A bisect resulted in 036f30e

@paretje
Copy link
Contributor

paretje commented Jun 7, 2020

I guess this is the issue:

diff --git a/alot/db/message.py b/alot/db/message.py
index 02c059f6..9f5fafb5 100644
--- a/alot/db/message.py
+++ b/alot/db/message.py
@@ -10,7 +10,7 @@ from datetime import datetime
 from notmuch import NullPointerError

 from . import utils
-from .utils import extract_body
+from .utils import get_body_part, extract_body_part
 from .utils import decode_header
 from .attachment import Attachment
 from .. import helper
@@ -68,6 +68,8 @@ class Message:
         else:
             self._from = '"Unknown" <>'

+        self.mime_part = get_body_part(self.get_email())
+
     def __str__(self):
         """prettyprint the message"""
         aname, aaddress = self.get_author()
@@ -263,8 +265,7 @@ class Message:

     def get_body_text(self):
         """ returns bodystring extracted from this mail """
-        # TODO: allow toggle commands to decide which part is considered body
-        return extract_body(self.get_email())
+        return extract_body_part(self.mime_part)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants