idle function not returning any message when new mail comes #258

syedajmal1998 · 2022-08-18T23:15:12Z

idle function not returning any message when new mail comes

Config:

        'custom' => [
            'host' => 'outlook.office365.com',
            'port' => 993,
            'encryption' => "ssl",
            'validate_cert' => true,
            'username' => 'email@my_domain.com',
            'password' => get_access_token(),
            'authentication' => "oauth",
        ],
// halper_functions.php
function get_access_token()
    {
        $tenantId = env("TENANT_ID");
        $clientId = env("CLIENT_ID");
        $clientSecret = env("CLIENT_SECRET");
        $guzzle = new \GuzzleHttp\Client();
        $url = 'https://login.microsoftonline.com/' . $tenantId . '/oauth2/v2.0/token';
        $token = json_decode($guzzle->post($url, [
            'form_params' => [
                'client_id' => $clientId,
                'client_secret' => $clientSecret,
                'scope' => 'https://outlook.office.com/IMAP.AccessAsUser.All',
                'grant_type' => 'password',
                'username' => 'email@my_domain.com',
                'password' => 'password',
            ],
        ])->getBody()->getContents());
        $accessToken = $token->access_token;
        return $accessToken;
    }

custom command:

public function handle()
    {
        $client = Client::account("custom");
        try {
            $client->connect();
        } catch (ConnectionFailedException $e) {
            Log::error($e->getMessage());
            return 1;
        }
        try {
            /** @var Folder $folder */
            $folder = $client->getFolder("INBOX");
        } catch (ConnectionFailedException $e) {
            Log::error($e->getMessage());
            return 1;
        } catch (FolderFetchingException $e) {
            Log::error($e->getMessage());
            return 1;
        }
        try {
            dump("listening...");
            $folder->idle(function ($message) {
            dump($message); // this doesn't run
            }, 1200, true);
        } catch (ConnectionFailedException $e) {
            Log::error($e->getMessage());
            return 1;
        }

but when i get messages manually it works

Route::get("test", function () {
    $client = Client::account("custom");
    try {
        $client->connect();
    } catch (ConnectionFailedException $e) {
        throw $e;
    }
    try {
        /** @var Folder $folder */
        $folder = $client->getFolder("INBOX");
    } catch (ConnectionFailedException $e) {
        throw $e;
    } catch (FolderFetchingException $e) {
        throw $e;
    }
    
    dump($client->getFolder("INBOX")->examine());
    dump($client->getFolder("INBOX")->messages()->all()->limit(5)->get());
    dd("done");
});

OS: CentOs (prod), win 11 (dev)
PHP: 8.1
webklex/laravel-imap : 2.4
Provider: Outlook

Webklex · 2022-08-19T10:33:46Z

Hi @syedajmal1998 ,
thanks for the detailed report. I believe to have determined and fixed the issue. I'm currently testing the patch to see if it actually works as expected. I'll let you know in several hours.

It turns out the idle connection timeout / lifetime depends on the server settings and might just last for several minutes before it terminates.

The ImapProtocol class used to use the php function fgets to fetch the next line - unfortunately this method ignores the timeout set with stream_set_timeout. To fix this, I switched to reading the data stream byte by byte via fread.

I also improved the Folder::idle method to process * OK Still here callbacks received while idling. It should also "survive" temporary connection issues and always reestablish a connection.

Its currently running for about 2,5h (applied timeout: 300sec) without any issues.

Best regards,

syedajmal1998 · 2022-08-19T15:51:07Z

thank you so much for your time I'll wait

Webklex · 2022-08-19T19:07:38Z

Hi @syedajmal1998 ,
looks like its fixed :)

Please update to the latest release: https://github.com/Webklex/php-imap/releases/tag/4.0.0
(might take a few minutes, until the release gets available via composer).

Best regards & happy coding,

syedajmal1998 · 2022-08-19T20:04:45Z

thank you so much for time it means alot, but also can you please update webklex/laravel-imap dependencies, i cant install it

Webklex · 2022-08-19T21:11:43Z

You are right, its long overdue.
A new release is now available: https://github.com/Webklex/laravel-imap/releases/tag/4.0.0

Best regards,

syedajmal1998 · 2022-08-19T22:48:33Z

sorry for asking, but its still the same for me.

syedajmal1998 · 2022-08-19T22:56:46Z

also a new bug:

Illegal offset type

at vendor\webklex\php-imap\src\Connection\Protocols\ImapProtocol.php:646
642▕
643▕ return $data;
644▕ }
645▕ if ($uid) {
➜ 646▕ $result[$tokens[2][$uidKey]] = $data;
647▕ }else{
648▕ $result[$tokens[0]] = $data;
649▕ }
650▕ }

1 vendor\webklex\php-imap\src\Connection\Protocols\ImapProtocol.php:697
Webklex\PHPIMAP\Connection\Protocols\ImapProtocol::fetch()

2 \webklex\php-imap\src\Query\Query.php:228
Webklex\PHPIMAP\Connection\Protocols\ImapProtocol::flags()

when:

$client->getFolder("INBOX")->messages()->all()->limit(5)->get()

Webklex · 2022-08-19T23:00:21Z

Please enable the debug mode inside your config/imap.php config file:
https://github.com/Webklex/laravel-imap/blob/4.0.0/src/config/imap.php#L152

The debug output should look something like this, during the "startup" process:

>> TAG1 LOGIN "[email protected]" "some password"
<< TAG1 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY LITERAL+ NOTIFY SPECIAL-USE QUOTA] Logged in
>> TAG2 CAPABILITY
<< * CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY LITERAL+ NOTIFY SPECIAL-USE QUOTA
<< TAG2 OK Capability completed (0.001 + 0.000 secs).
>> TAG3 SELECT "INBOX"
<< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk unknown-1)
<< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk unknown-1 \*)] Flags permitted.
<< * 136 EXISTS
<< * 0 RECENT
<< * OK [UNSEEN 94] First unseen.
<< * OK [UIDVALIDITY 1488899637] UIDs valid
<< * OK [UIDNEXT 275] Predicted next UID
<< TAG3 OK [READ-WRITE] Select completed (0.010 + 0.000 + 0.009 secs).
>> TAG4 IDLE
<< + idling

You could also try to reduce the timeout from 1200 to 300 or some other lower number.

syedajmal1998 · 2022-08-19T23:09:07Z

<< TAG1 OK AUTHENTICATE completed.

TAG2 CAPABILITY
<< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CLIENTACCESSRULES CLIENTNETWORKPRESENCELOCATION BACKENDAUTHENTICATE CHILDREN IDLE NAMESPACE LITERAL+
<< TAG2 OK CAPABILITY completed.
TAG3 SELECT "INBOX"
<< * 1516 EXISTS
<< * 0 RECENT
<< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
<< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
<< * OK [UNSEEN 21] Is the first unseen message
<< * OK [UIDVALIDITY 14] UIDVALIDITY value
<< * OK [UIDNEXT 3977] The next unique identifier value
<< TAG3 OK [READ-WRITE] SELECT completed.
TAG4 IDLE
<< + IDLE accepted, awaiting DONE command.

syedajmal1998 · 2022-08-19T23:13:38Z

here is the full one

<< * OK The Microsoft Exchange IMAP4 service is ready. [some long code]

TAG1 AUTHENTICATE XOAUTH2 [some too long code]
<< TAG1 OK AUTHENTICATE completed.
TAG2 LIST "" "*"
<< * LIST (\Marked \HasNoChildren) "/" Archive
<< * LIST (\Marked \HasChildren) "/" Calendar
<< * LIST (\HasNoChildren) "/" Calendar/Birthdays
<< * LIST (\HasNoChildren) "/" "Calendar/United States holidays"
<< * LIST (\HasChildren) "/" Contacts
<< * LIST (\HasChildren) "/" "Conversation History"
<< * LIST (\Marked \HasNoChildren \Trash) "/" "Deleted Items"
<< * LIST (\Marked \HasNoChildren \Drafts) "/" Drafts
<< * LIST (\Marked \HasNoChildren) "/" INBOX
<< * LIST (\HasNoChildren) "/" Journal
<< * LIST (\HasNoChildren \Junk) "/" "Junk Email"
<< * LIST (\HasNoChildren) "/" Notes
<< * LIST (\HasNoChildren) "/" Outbox
<< * LIST (\HasNoChildren) "/" "RSS Subscriptions"
<< * LIST (\HasNoChildren \Sent) "/" "Sent Items"
<< * LIST (\HasChildren) "/" "Sync Issues"
<< * LIST (\HasNoChildren) "/" "Sync Issues/Conflicts"
<< * LIST (\HasNoChildren) "/" "Sync Issues/Local Failures"
<< * LIST (\HasNoChildren) "/" "Sync Issues/Server Failures"
<< * LIST (\HasNoChildren) "/" Tasks
<< TAG2 OK LIST completed.
^ "listening..."
TAG3 LOGOUT
<< * BYE Microsoft Exchange Server IMAP4 server signing off.
<< TAG3 OK LOGOUT completed.
<< * OK The Microsoft Exchange IMAP4 service is ready. [again some long code]
TAG1 AUTHENTICATE XOAUTH2 [again some too long code]
<< TAG1 OK AUTHENTICATE completed.
TAG2 CAPABILITY
<< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CLIENTACCESSRULES CLIENTNETWORKPRESENCELOCATION BACKENDAUTHENTICATE CHILDREN IDLE NAMESPACE LITERAL+
<< TAG2 OK CAPABILITY completed.
TAG3 SELECT "INBOX"
<< * 1516 EXISTS
<< * 0 RECENT
<< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
<< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
<< * OK [UNSEEN 21] Is the first unseen message
<< * OK [UIDVALIDITY 14] UIDVALIDITY value
<< * OK [UIDNEXT 3977] The next unique identifier value
<< TAG3 OK [READ-WRITE] SELECT completed.
TAG4 IDLE
<< + IDLE accepted, awaiting DONE command.

Webklex · 2022-08-19T23:14:04Z

That looks exactly as expected. Now send an email to that mailbox and check the debug output. You should see something. Please let me know what happens next :)

syedajmal1998 · 2022-08-19T23:16:30Z

i sent multiple messages but no response no new log

Webklex · 2022-08-19T23:19:29Z

But you did start the idle command before sending the messages correct? Which folder are you idling on? Is it "INBOX" or another folder? If you check the mailbox with another client, where are the new messages located? Inside "INBOX" or another folder?

You might also have to wait a little bit until the server has processed the mail and decides to notify all active sessions.

syedajmal1998 · 2022-08-19T23:25:15Z

yes i started idle command before sending new mails and waited and in outlook new mails were in inbox folder

Webklex · 2022-08-19T23:25:26Z

Another thought:
try to set the timeout to 60 and let the command run at least 120 seconds after you've send the test mail. The timeout should "kick" in during that time and try to fetch a response again. Would be interesting if that changes anything.

syedajmal1998 · 2022-08-19T23:36:43Z

i think it worked but got this message
DONE
<< * 1 RECENT

TAG5 IDLE
<< * 1521 EXISTS

Webklex\PHPIMAP\Exceptions\RuntimeException

idle failed

at vendor\webklex\php-imap\src\Connection\Protocols\ImapProtocol.php:1043
1039▕ /
1040▕ public function idle() {
1041▕ $this->sendRequest("IDLE");
1042▕ if (!$this->assumedNextLine('+ ')) {
➜ 1043▕ throw new RuntimeException('idle failed');
1044▕ }
1045▕ }
1046▕
1047▕ /*

1 vendor\webklex\php-imap\src\Folder.php:401
Webklex\PHPIMAP\Connection\Protocols\ImapProtocol::idle()

2 app\Console\Commands\FetchIdleCommand.php:250
Webklex\PHPIMAP\Folder::idle(Object(Closure))

TAG6 LOGOUT
<< TAG4 OK IDLE completed.
<< + IDLE accepted, awaiting DONE command.
<< TAG6 BAD Command received in Invalid state.

Webklex · 2022-08-19T23:56:01Z

That's indeed interesting!
I would have expected to see something like this:

>> TAG4 IDLE
<< + IDLE accepted, awaiting DONE command.
<< * 1521 EXISTS
>> DONE

1521 EXISTS was sent to early by the server - before IDLE was actually acknowledged to have started. I haven't seen this before. I wonder what could cause this. Perhaps this is an outlook specific feature related to "recent" messages? But this would still be strange - anyway, do you get the same error if you have no recent messages?

Btw you have uid_cache enabled or disabled inside your config? If it is disabled, please enable it.
Ref.: https://github.com/Webklex/php-imap/blob/master/src/config/imap.php#L153

Best regards & thanks for helping me understand this "feature",

syedajmal1998 · 2022-08-20T00:03:35Z

'uid_cache' was not in config

syedajmal1998 · 2022-08-20T00:23:20Z

these are all new, Microsoft dropped support for logging in using username & password for imap and everything was working fine till 3-4 days ago.
im using idle command for around 3-4 months this never happened

Webklex · 2022-08-20T00:37:15Z

I hacked together an alternative idle function, which doesn't check if the IDLE command got acknowledged.

/** @var \Webklex\PHPIMAP\Client $client */
$client->connect();
/** @var \Webklex\PHPIMAP\Folder $folder */
$folder = $client->getFolder("INBOX");
$timeout = 300;
$callback = function($message){
    dump("New message received: ".$message->subject);
};

/**
 * @param \Webklex\PHPIMAP\Connection\Protocols\ProtocolInterface $connection
 * @return void
 */
function idle($connection) {
    $connection->sendRequest("IDLE");
}

$client->setTimeout($timeout);
if (!in_array("IDLE", $client->getConnection()->getCapabilities())) {
    throw new NotSupportedCapabilityException("IMAP server does not support IDLE");
}
$client->openFolder($folder->path, true);
$connection = $client->getConnection();
idle($connection);

$sequence = ClientManager::get('options.sequence', IMAP::ST_MSGN);

while (true) {
    try {
        $line = $connection->nextLine();

        if (($pos = strpos($line, "EXISTS")) !== false) {
            $connection->done();
            $msgn = (int) substr($line, 2, $pos -2);

            $client->openFolder($folder->path, true);
            $message = $folder->query()->getMessageByMsgn($msgn);
            $message->setSequence($sequence);
            $callback($message);

            $event = $folder->getEvent("message", "new");
            $event::dispatch($message);
            idle($connection);
        }
    }catch (\Webklex\PHPIMAP\Exceptions\RuntimeException $e) {
        if(strpos($e->getMessage(), "empty response") >= 0 && $connection->connected()) {
            $connection->done();
            idle($connection);
            continue;
        }
        if(strpos($e->getMessage(), "connection closed") === false) {
            throw $e;
        }

        $client->reconnect();
        $client->openFolder($folder->path, true);

        $connection = $client->getConnection();
        idle($connection);
    }
}

In this snipped I could see $connection->done(); causing problems, because it might get called to early (?) But that exception should be catchable if it does indeed causes problems.

Best regards,

syedajmal1998 · 2022-08-20T08:02:56Z

thank you so much this worked pretty well. also i set timeout to 20s bcs it waits till timeout ends for new mail to fetch not like before

syedajmal1998 · 2022-08-20T08:06:53Z

that block of code worked and fixed my issue, if this is fixed in package you can close this issue

syedajmal1998 · 2022-08-20T08:08:36Z

and sorry if i said or did something wrong this was my first ever issued a bug in github :)

Webklex · 2022-08-20T09:37:58Z

Hi @syedajmal1998 ,
nah you've did everything right :) Thanks for bringing this issue up 👍

At the moment I would like to wait a little bit to see if this "feature" gets fixed by outlook / Microsoft or if other providers behave the same. I don't think they intended it to work this way. Also removing the "idle" check in general isn't great. So for now I wont modify the idle methods. However you can use the snipped from above to build your own custom idle implementation.

I hope you're having a great weekend :)

Best regards,

Webklex · 2023-03-16T00:51:43Z

Please update to v5.1 and give it another try.
If you are currently using an older version below v5.0, please read the breaking changes leading up to v5.1 before upgrading.

Best regards,

repo-dev-1 · 2024-04-08T10:50:01Z

Hello Weblex
Thanks for your work. But this problem is still relevant. I can’t figure out what exactly is the reason, but idle always crashes with an “empty response” after the time set in the timeout. (Same situation on different mail servers)

Webklex · 2024-05-16T12:21:11Z

Hi @repo-dev-1 ,
please create a new issue and provide as much information as possible.

Best regards & happy coding,

codeway · 2024-12-10T13:09:39Z

Same issue here. Solution in my case :
I have updated to v5.3 :
"webklex/laravel-imap": "^5.3"
and change php 7.4 to 8.1
It's ok now :

The part of code :

syedajmal1998 mentioned this issue Aug 18, 2022

no message when new message comes while running $folder->idle() #257

Closed

Webklex added bug Something isn't working validating validated and removed validating labels Aug 19, 2022

Webklex added a commit that referenced this issue Aug 19, 2022

Idle command reworked and several issues fixed #170 #229 #237 #249 #258

f0f6d61

syedajmal1998 closed this as completed Aug 20, 2022

syedajmal1998 reopened this Aug 20, 2022

Webklex added the Outlook Related to Outlook label Aug 20, 2022

Webklex closed this as completed Jun 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

idle function not returning any message when new mail comes #258

idle function not returning any message when new mail comes #258

syedajmal1998 commented Aug 18, 2022 •

edited

Loading

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022 •

edited

Loading

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022 •

edited

Loading

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022 •

edited

Loading

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 20, 2022

syedajmal1998 commented Aug 20, 2022

Webklex commented Aug 20, 2022

syedajmal1998 commented Aug 20, 2022 •

edited

Loading

syedajmal1998 commented Aug 20, 2022

syedajmal1998 commented Aug 20, 2022

Webklex commented Aug 20, 2022

Webklex commented Mar 16, 2023

repo-dev-1 commented Apr 8, 2024

Webklex commented May 16, 2024

codeway commented Dec 10, 2024 •

edited

Loading

idle function not returning any message when new mail comes #258

idle function not returning any message when new mail comes #258

Comments

syedajmal1998 commented Aug 18, 2022 • edited Loading

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022 • edited Loading

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022 • edited Loading

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 19, 2022 • edited Loading

Webklex commented Aug 19, 2022

syedajmal1998 commented Aug 20, 2022

syedajmal1998 commented Aug 20, 2022

Webklex commented Aug 20, 2022

syedajmal1998 commented Aug 20, 2022 • edited Loading

syedajmal1998 commented Aug 20, 2022

syedajmal1998 commented Aug 20, 2022

Webklex commented Aug 20, 2022

Webklex commented Mar 16, 2023

repo-dev-1 commented Apr 8, 2024

Webklex commented May 16, 2024

codeway commented Dec 10, 2024 • edited Loading

syedajmal1998 commented Aug 18, 2022 •

edited

Loading

syedajmal1998 commented Aug 19, 2022 •

edited

Loading

syedajmal1998 commented Aug 19, 2022 •

edited

Loading

syedajmal1998 commented Aug 19, 2022 •

edited

Loading

syedajmal1998 commented Aug 20, 2022 •

edited

Loading

codeway commented Dec 10, 2024 •

edited

Loading