Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception: Number of retries reached 0. #270

Closed
andr-04 opened this issue Sep 28, 2017 · 54 comments
Closed

Exception: Number of retries reached 0. #270

andr-04 opened this issue Sep 28, 2017 · 54 comments
Labels
help wanted Details, logs, code is wanted

Comments

@andr-04
Copy link
Contributor

andr-04 commented Sep 28, 2017

I haven't check on new version v0.14, but previous one this exception happened very often on outgoing requests in a random time. I don't know: Telegram servers sometime is slow or it's flood protection. Definitely this exception has to be more informative to understand the reason of it.

@Lonami
Copy link
Member

Lonami commented Sep 28, 2017

If any request has a None result:

if any(x.result is None for x in requests):

It's going to retry:

return self.invoke(
*requests, call_receive=call_receive, retries=(retries - 1)
)

Up to five times before raising an error:

if retries <= 0:
raise ValueError('Number of retries reached 0.')

I don't know why the result of some requests is None. I know it fails for LogOutRequest, need to fix that, forgot to do it before v0.14.1.

@andr-04
Copy link
Contributor Author

andr-04 commented Sep 28, 2017

Maybe the exception on LogOutRequest is related with this one as well?

@Lonami
Copy link
Member

Lonami commented Sep 28, 2017

Yes, the exception there is this one, because logging out is "confirmed" through a message acknowledge and no result is actually read. From my experience, I think ack requests are only used int this case… So no result per se is set.

Is using LogOutRequest the only request with which you encounter this issue?

@Lonami
Copy link
Member

Lonami commented Sep 29, 2017

Seems to work with LogOutRequest but I'm yet to decide what to do with every other request. Maybe reconnect? But what request is it the one which fails? Is it one in specific? Maybe no result is returned ever? Or does it happen with all?

@andr-04
Copy link
Contributor Author

andr-04 commented Sep 29, 2017

In log I see the problem only with ResolveUsernameRequest. Probably other requests have this problem as well, but my use case raises the exception only with this one.

@Lonami
Copy link
Member

Lonami commented Sep 30, 2017

The error may be because .disconnect() was called from somewhere in the code, and it's trying to invoke while not being connected. Thus it fails and doesn't attempt to reconnect. Need to find where though.

@88ee55
Copy link
Contributor

88ee55 commented Oct 2, 2017

I have the same exception.
channels.ReadHistoryRequest(channel=peer, max_id=0)
peer is InputChannel

telethon 0.15.0.3
python 3.5

@88ee55
Copy link
Contributor

88ee55 commented Oct 2, 2017

ReadHistoryRequest
You need to call if unread more than 0
then there will be no exception

@Lonami
Copy link
Member

Lonami commented Oct 2, 2017

So are you saying that with max_id=0 it fails all retries, but works with a non-zero value? I wonder what Telegram is replying or doing for this to happen with only certain parameters.

@88ee55
Copy link
Contributor

88ee55 commented Oct 2, 2017

exception occurs when there are no unread messages

@Lonami
Copy link
Member

Lonami commented Oct 7, 2017

If anyone wants to test, doing this would be really helpful. The only requirement is that you can reproduce this bug.

Add this class to telethon/tl/tlobject.py:

from threading import Event
import traceback
import os
class LoggingEvent:
    def __init__(self, t):
        self.o = open(os.path.expanduser('~/telethon.log'), 'w')
        self.e = Event()
        self.t = type(t).__name__
    def _log(self, k):
        m = '-- '+k+' FLAG ON '+self.t.+' --'
        self.o.write(m+'\n')
        traceback.print_stack(file=self.o)
        self.o.write('-'*len(m)+'\n\n\n')
        self.o.flush()
    def clear(self):
        self._log('CLEARING')
        return self.e.clear()
    def set(self):
        self._log('SETTING')
        return self.e.set()
    def is_set(self):
        return self.e.is_set()
    def wait(self, *a, **k):
        self.e.wait(*a, **k)

Then, inside the TLObject class change:

self.confirm_received = Event()

with

self.confirm_received = LoggingEvent(self)

@alihossein
Copy link
Contributor

alihossein commented Oct 8, 2017

exception occurs when there are no unread messages

how can I check the number of unread messages?

@88ee55
Copy link
Contributor

88ee55 commented Oct 9, 2017

how can I check the number of unread messages?

GetFullChannelRequest(InputChannel()).chat_full.unread_count

@Lonami
Copy link
Member

Lonami commented Oct 9, 2017

exception occurs when there are no unread messages

This is false-y, so it's probably the reason why the exception occured. Can you test again? It likely was fixed on 244a47c.

@Lonami Lonami added the help wanted Details, logs, code is wanted label Oct 9, 2017
@mnvx
Copy link
Contributor

mnvx commented Oct 11, 2017

I has stuck on this Exception too. Currently I skip these requests:

    def my_method(self):
        try:
            dialogs, entities = self._client.get_dialogs()
        except Exception as e:
            time.sleep(1) # I hope what after delay connection will be repaired
            raise MyApiException('Im sorry, try later')
        return dialogs

    def call_my_method(self):
        try:
            dialogs = self.my_method()
        except MyApiException as e:
            self._log(str(e))
            return

        self.process_dialogs(dialogs)

@K900
Copy link
Contributor

K900 commented Oct 11, 2017

I've reproduced the bug successfully. Here's the log:

-- SETTING FLAG ON TLMessage --
  File ".../dump_tg.py", line 38, in <module>
    main()
  File ".../dump_tg.py", line 32, in main
    download_chat(client, b, 'b')
  File ".../dump_tg.py", line 23, in download_chat
    downloader.download_from_message(message)
  File "...\downloader.py", line 92, in download_from_message
    output_path=path
  File "...\downloader.py", line 56, in download_if_needed
    progress_callback=progress_callback
  File "C:\Users\...\AppData\Local\Programs\Python\Python36-32\lib\site-packages\telethon\telegram_bare_client.py", line 701, in download_file
    input_location, offset, part_size
  File "C:\Users\...\AppData\Local\Programs\Python\Python36-32\lib\site-packages\telethon\telegram_bare_client.py", line 448, in __call__
    result = self._invoke(sender, call_receive, *requests)
  File "C:\Users\...\AppData\Local\Programs\Python\Python36-32\lib\site-packages\telethon\telegram_bare_client.py", line 503, in _invoke
    sender.disconnect()
  File "C:\Users\...\AppData\Local\Programs\Python\Python36-32\lib\site-packages\telethon\network\mtproto_sender.py", line 55, in disconnect
    self._clear_all_pending()
  File "C:\Users\...\AppData\Local\Programs\Python\Python36-32\lib\site-packages\telethon\network\mtproto_sender.py", line 247, in _clear_all_pending
    r.confirm_received.set()
  File "C:\Users\...\AppData\Local\Programs\Python\Python36-32\lib\site-packages\telethon\tl\tlobject.py", line 22, in set
    def set(self):
  File "C:\Users\...\AppData\Local\Programs\Python\Python36-32\lib\site-packages\telethon\tl\tlobject.py", line 15, in _log
    self.o.write(m+'\n')
-------------------------------

@K900
Copy link
Contributor

K900 commented Oct 13, 2017

With the latest commits from master, I now get telethon.errors.rpc_errors_420.FloodWaitError: (FloodWaitError(...), 'A wait of 488 seconds is required.'). Looks like the connection drop issue is fixed, but it doesn't always sleep when rate limited.

@Lonami
Copy link
Member

Lonami commented Oct 13, 2017

No, there's a threshold (defaults to 30 seconds). If it's longer than that you don't want the library to just hang, you may want to take other actions. You should be able to modify that threshold to be as big as you need.

@K900
Copy link
Contributor

K900 commented Oct 13, 2017

Oh, I see. How do you set it up? I don't see any relevant parameters on TelegramClient :(

@Lonami
Copy link
Member

Lonami commented Oct 13, 2017

Was mistaken, one minute actually. It's here.

@Lonami
Copy link
Member

Lonami commented Nov 1, 2017

@pashbelg are you willing to help or were you just confirming you have this issue too? If the latter, that's not too helpful.

@k01aty
Copy link

k01aty commented Nov 9, 2017

If I use 'InteractiveTelegramClient' instead of 'TelegramClient' .connect() ,it is not throwing this exception

@vegeta1k95
Copy link
Contributor

I also get this exception randomly after doing some GetDialogsRequest. But after some pause invoking same request doesn't raise error even if dialogs did not change at all.

@andr-04
Copy link
Contributor Author

andr-04 commented Nov 9, 2017

I've fixed many aspects of this issue and the last one I've found for today.

I've found that sometimes Telegram stops send any responses. send returns the bytes which were sent without exceptions, but no answers from Telegram happened anymore. After some time Errno 104: Connection reset by peer happened. The solution -- not just return None, but to check if this is a retry (2nd attempt for example), make a reconnection.

@Lonami
Copy link
Member

Lonami commented Nov 9, 2017

Is it possible to quickly check whether the socket has been closed, so we can do it when retrying? Shouldn't it error with "socket closed" while trying to read already? The background thread should be reconnecting. Maybe retries happen too quick?

@andr-04
Copy link
Contributor Author

andr-04 commented Nov 9, 2017

No, the socket is not closed because of send returns positive number of sent bytes. As a result you just never get the result during retries. And don't detect the connection is broken -- this is the main problem. So just reconnect if no response during first 1 or 2 attempts.

@Lonami
Copy link
Member

Lonami commented Nov 10, 2017

Is the ValueError raised immediatly or does it take long due to the 5s timeout?

@andr-04
Copy link
Contributor Author

andr-04 commented Nov 10, 2017

I don't use the master version last time. I gave you a hint, you can investigate more by yourself. According the source code it can't be raised immediately, just after retries exceeded. Yes, it waits for timeout and timeout is happened.

Lonami added a commit that referenced this issue Nov 12, 2017
@vegeta1k95
Copy link
Contributor

Okey. I've managed to find something about this.

I had one session file and I made two copies of it.
I used FIRST copy in my script and eventually somehow managed to catch this error.
This error made my script stumble and started appearing on every requests (i.e. ResolveUsernameRequest), and even on connection request (GetStateRequest in 'sync_state' function of bare_telegram_client). I could not do anything, even multiple reconnections didn't help.

However, I started the same script in parallel with the SECOND fresh copy of the session and it started working like if nothing happened, while first was struggling to do anything. Then I replaced first copy with the second one and it also started working.

So I can assume that the cause may lie within the session file and not because of connection problems or Telegram server rejecting requests. Maybe, data, i.e. auth_key or salt, somehow becomes damaged and inflicts bad responses from servers. Unfortunately, I forgot to make a backup of "bad" copy to check if there were differences.

That's all I know at the moment. I'm trying to find the way to reproduce this, to investigate it more.

@Lonami
Copy link
Member

Lonami commented Jan 15, 2018

Bump. Does this still happen as of v0.16.1.x?

@ehsanonline
Copy link

ehsanonline commented Jan 20, 2018

@Lonami yes , I have tried to forward a message in 2 megagroups and after about 20 times of forwarding, with 5min sleep between each time, I got this error.
File "../modules\telethon\telegram_bare_client.py", line 459, in __call__ raise RuntimeError('Number of retries reached 0.') RuntimeError: Number of retries reached 0.

@rex101
Copy link

rex101 commented Feb 26, 2018

how can I create client using Telethon because when I am executing this code it says from telethon import TelegramClient

api_id = 00000
api_hash = 'aadasd6a76d7a6d'

client = TelegramClient('session_name', api_id, api_hash)
client.start()

ERROR:

Traceback (most recent call last):
File "test.py", line 4, in
from telethon import TelegramClient
File "/abc/bde/Desktop/Telet/Telethon/telethon/init.py", line 2, in
from .telegram_bare_client import TelegramBareClient
File "/abc/bde/Desktop/Telet/Telethon/telethon/telegram_bare_client.py", line 414
def call(self, *requests, retries=5):
^
SyntaxError: invalid syntax

@Lonami
Copy link
Member

Lonami commented Feb 27, 2018

@rex101 why did you leave #644 empty and comment here‽ You probably are using Python 2 and it states literally everywhere that you need Python 3.

@rex101
Copy link

rex101 commented Mar 2, 2018

@Lonami Can't write it's showing " This conversation has been locked as resolved and limited to collaborators. "

@Lonami
Copy link
Member

Lonami commented Mar 2, 2018

@rex101 stop commenting on completely unrelated issues.

@bikhial
Copy link

bikhial commented Mar 16, 2018

Hi guys

I get This Number of Retries reached 0 message for GetHistoryRequest when i try to fetch channel posts. is this related to this issue or not?
if not what can i do for resolve it?
using lates version 0.18.2

thanks

@ferferga
Copy link
Contributor

Same for me, but still looking for a pattern to reproduce. Will update here whenever I find something useful. However, and I don't know if it's useful or have something to do with this, when this exception is raised and I retry the exact same query, another exception with the message "Telegram's having problems right now, try again later" is raised. I use "limit=None" as others

@Lonami
Copy link
Member

Lonami commented Apr 13, 2018

Yesterday I was having some trouble to iterate over all messages of a channel too, I got a few "internal errors from Telegram", but not "retries reached 0"

@olivier-salmon
Copy link

I have a python script running in a loop, checking new messages every 5 minutes. After a while, the error appears:

2018-04-18 06:46:45
Error: Number of retries reached 0.
2018-04-18 06:52:36
Error: Number of retries reached 0.
2018-04-18 06:58:26
Error: Number of retries reached 0.
2018-04-18 07:04:16
Error: Number of retries reached 0.

If I kill the running script and start again, it's fine for hours. Then the error comes back.

What can I do to help to resolve this?

@Lonami
Copy link
Member

Lonami commented Apr 18, 2018

You can create a Minimal Working Example for me to try and reproduce the issue, with the hope that I can fix it. Otherwise enable DEBUG logging, give me the output, and hopefully I can see something meaningful on it.

@saksham2410
Copy link

Has the issue fixed? Or anyone has found an alternative method? I get these Number of retries reached 0 errors very frequently.

@umaxfun
Copy link

umaxfun commented May 25, 2018

Some new shiny logs:

[2018-05-25 11:27:50,382: INFO/MainProcess] Received task: jobs.save_channel_by_name_async[a4d161c5-8c5a-41e3-a38a-0580b5749a33]
[2018-05-25 11:27:50,399: INFO/ForkPoolWorker-2] ConnectionError "[Errno 32] Broken pipe" while writing data
[2018-05-25 11:27:50,406: WARNING/ForkPoolWorker-2] Connection was reset while invoking
[2018-05-25 11:27:50,406: INFO/ForkPoolWorker-2] Disconnecting MtProtoSender...
[2018-05-25 11:27:50,407: INFO/ForkPoolWorker-2] Abruptly confirming TLMessage
[2018-05-25 11:27:50,408: INFO/ForkPoolWorker-2] Invoking GetFullChannelRequest failed 1 times, connecting again and retrying
[2018-05-25 11:27:51,409: INFO/ForkPoolWorker-2] Attempting reconnection...
[2018-05-25 11:27:51,409: INFO/ForkPoolWorker-2] Connecting to 149.154.167.51:443...
[2018-05-25 11:27:51,466: INFO/ForkPoolWorker-2] Connection success!
[2018-05-25 11:28:01,469: WARNING/ForkPoolWorker-2] Invoking GetFullChannelRequest failed 2 times, connecting again and retrying
[2018-05-25 11:28:02,470: INFO/ForkPoolWorker-2] Reconnection aborted: already connected
[2018-05-25 11:28:12,473: WARNING/ForkPoolWorker-2] Invoking GetFullChannelRequest failed 3 times, connecting again and retrying
[2018-05-25 11:28:13,474: INFO/ForkPoolWorker-2] Reconnection aborted: already connected
[2018-05-25 11:28:23,477: WARNING/ForkPoolWorker-2] Invoking GetFullChannelRequest failed 4 times, connecting again and retrying
[2018-05-25 11:28:24,478: INFO/ForkPoolWorker-2] Reconnection aborted: already connected
[2018-05-25 11:28:34,481: WARNING/ForkPoolWorker-2] Invoking GetFullChannelRequest failed 5 times, connecting again and retrying
[2018-05-25 11:28:35,482: INFO/ForkPoolWorker-2] Reconnection aborted: already connected
[2018-05-25 11:28:35,500: ERROR/ForkPoolWorker-2] Task jobs.save_channel_by_name_async[a4d161c5-8c5a-41e3-a38a-0580b5749a33] raised unexpected: RuntimeError('Number of retries reached 0 for GetFullChannelRequest.',)

Seems that the problem is located here:

[2018-05-25 11:28:01,469: WARNING/ForkPoolWorker-2] Invoking GetFullChannelRequest failed 2 times, connecting again and retrying
[2018-05-25 11:28:02,470: INFO/ForkPoolWorker-2] Reconnection aborted: already connected

@Lonami Lonami mentioned this issue Jun 11, 2018
5 tasks
@Lonami
Copy link
Member

Lonami commented Jun 14, 2018

bb3a564 has been merged, reopen after the next version is published if this error still occurs. Note that master is now running under asyncio so your code won't be compatible with it. To preserve this compatibility, a new branch emulating synchronous behaviour will be created.

@Lonami Lonami closed this as completed Jun 14, 2018
@endor-daniel
Copy link

Still happens.

@Lonami
Copy link
Member

Lonami commented Jun 27, 2018

Really? v1.0-post2? What are the DEBUG logs? What request? How to reproduce the issue?

@endor-daniel
Copy link

@Lonami - apparently, Telegram servers were down for about an hour.

@Lonami
Copy link
Member

Lonami commented Jun 27, 2018

I did notice that. So not an issue.

@endor-daniel
Copy link

Not an issue in Telethon, that's for sure.

@iranpak
Copy link

iranpak commented Nov 12, 2018

Hi.
i have same issue while getting a channel history using iter_messages method. while script starts it works well. but after a while this happens. i am using telethon 0.18.2. And this was not happening until yesterday. this is the log for my issue:

DEBUG:telethon.telegram_bare_client:Invoking GetHistoryRequest DEBUG:telethon.network.mtproto_sender:Sending requests with IDs: GetHistoryRequest: 6622887426673398964 INFO:telethon.telegram_bare_client:Connection success! DEBUG:telethon.network.mtproto_sender:Sending requests with IDs: GetStateRequest: 6622887427520293228 DEBUG:telethon.telegram_bare_client:Receiving items from the network... DEBUG:telethon.extensions.tcp_client:socket.timeout "timed out" while reading data DEBUG:telethon.telegram_bare_client:Receiving items from the network timed out DEBUG:telethon.telegram_bare_client:Receiving items from the network... DEBUG:telethon.extensions.tcp_client:socket.timeout "timed out" while reading data WARNING:telethon.telegram_bare_client:Invoking timed out INFO:telethon.telegram_bare_client:Invoking ['GetHistoryRequest(peer=InputPeerChannel(channel_id=1450340365, access_hash=6837884551796214524), offset_id=48, offset_date=None, add_offset=0, limit=30, max_id=0, min_id=17, hash=0)'] failed 1 times, connecting again and retrying INFO:telethon.telegram_bare_client:Reconnection aborted: already connected DEBUG:telethon.network.mtproto_sender:Sending requests with IDs: GetHistoryRequest: 6622887474814562696 DEBUG:telethon.extensions.tcp_client:socket.timeout "timed out" while reading data DEBUG:telethon.telegram_bare_client:Receiving items from the network timed out DEBUG:telethon.telegram_bare_client:Receiving items from the network... WARNING:telethon.extensions.tcp_client:socket.timeout "timed out" when 5/8 had been received WARNING:telethon.telegram_bare_client:Invoking timed out WARNING:telethon.telegram_bare_client:Invoking ['GetHistoryRequest(peer=InputPeerChannel(channel_id=1450340365, access_hash=6837884551796214524), offset_id=48, offset_date=None, add_offset=0, limit=30, max_id=0, min_id=17, hash=0)'] failed 2 times, connecting again and retrying INFO:telethon.telegram_bare_client:Reconnection aborted: already connected DEBUG:telethon.network.mtproto_sender:Sending requests with IDs: GetHistoryRequest: 6622887522986685556 DEBUG:telethon.extensions.tcp_client:socket.timeout "timed out" while reading data DEBUG:telethon.telegram_bare_client:Receiving items from the network timed out DEBUG:telethon.network.mtproto_sender:Sending requests with IDs: PingRequest: 6622887556435838520 DEBUG:telethon.telegram_bare_client:Receiving items from the network... DEBUG:telethon.extensions.tcp_client:socket.timeout "timed out" while reading data WARNING:telethon.telegram_bare_client:Invoking timed out WARNING:telethon.telegram_bare_client:Invoking ['GetHistoryRequest(peer=InputPeerChannel(channel_id=1450340365, access_hash=6837884551796214524), offset_id=48, offset_date=None, add_offset=0, limit=30, max_id=0, min_id=17, hash=0)'] failed 3 times, connecting again and retrying INFO:telethon.telegram_bare_client:Reconnection aborted: already connected DEBUG:telethon.network.mtproto_sender:Sending requests with IDs: GetHistoryRequest: 6622887570290757844 DEBUG:telethon.extensions.tcp_client:socket.timeout "timed out" while reading data DEBUG:telethon.telegram_bare_client:Receiving items from the network timed out DEBUG:telethon.telegram_bare_client:Receiving items from the network... WARNING:telethon.extensions.tcp_client:socket.timeout "timed out" when 5/8 had been received WARNING:telethon.telegram_bare_client:Invoking timed out WARNING:telethon.telegram_bare_client:Invoking ['GetHistoryRequest(peer=InputPeerChannel(channel_id=1450340365, access_hash=6837884551796214524), offset_id=48, offset_date=None, add_offset=0, limit=30, max_id=0, min_id=17, hash=0)'] failed 4 times, connecting again and retrying INFO:telethon.telegram_bare_client:Reconnection aborted: already connected DEBUG:telethon.network.mtproto_sender:Sending requests with IDs: GetHistoryRequest: 6622887618782004568 DEBUG:telethon.extensions.tcp_client:socket.timeout "timed out" while reading data DEBUG:telethon.telegram_bare_client:Receiving items from the network timed out DEBUG:telethon.telegram_bare_client:Receiving items from the network... DEBUG:telethon.extensions.tcp_client:socket.timeout "timed out" while reading data WARNING:telethon.telegram_bare_client:Invoking timed out WARNING:telethon.telegram_bare_client:Invoking ['GetHistoryRequest(peer=InputPeerChannel(channel_id=1450340365, access_hash=6837884551796214524), offset_id=48, offset_date=None, add_offset=0, limit=30, max_id=0, min_id=17, hash=0)'] failed 5 times, connecting again and retrying INFO:telethon.telegram_bare_client:Reconnection aborted: already connected Traceback (most recent call last): File "/root/vitabot/history_transfer.py", line 313, in check_channels channel_messages = get_channel_messages(single_channel, messsages_count) File "/root/vitabot/history_transfer.py", line 282, in get_channel_messages for message in reversed_channel_messages: File "/usr/local/lib/python3.6/dist-packages/telethon/telegram_client.py", line 1036, in iter_messages hash=0 File "/usr/local/lib/python3.6/dist-packages/telethon/telegram_bare_client.py", line 467, in __call__ [type(x).__name__ for x in requests] RuntimeError: Number of retries reached 0 for ['GetHistoryRequest'].

Is this because of my old version of telethon?

@Lonami
Copy link
Member

Lonami commented Nov 12, 2018

Is this because of my old version of telethon?

Quite possibly. Old versions are not supported at all.

@hzexe
Copy link

hzexe commented Nov 12, 2018

The same too.

telethon.version
'1.4.1'
python 3.5

@Lonami
Copy link
Member

Lonami commented Nov 13, 2018

It's impossible 1.4.1 shows "the same" because that error is gone. Please show a DEBUG log with 1.4.1 if it really happens "the same".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Details, logs, code is wanted
Projects
None yet
Development

No branches or pull requests