Only last message in the query has attachments within IMAP::ST_UID on Gmail #108

EthraZa · 2021-02-17T22:03:44Z

Problem:
When passing 'sequence' => \Webklex\PHPIMAP\IMAP::ST_UID to ClientManager, we can "see" the attachments only for the last message in the query.
I'm on this problem all the day long trying to figure it out, but it is just passing over the top of my head. For now, I just found that the ST_UID is the one to blame, but don't know why.

It works ok on Outlook.com.

How I set it up:
On Gmail (ordinary free account), I set a folder "Test" with 3 messages coming from the same source.
1st - 1 PDF
2nd - 1 PDF, 1 XML
3rd - 2 XML

What happens:
'sequence' => \Webklex\PHPIMAP\IMAP::ST_UID
I get only the 2 XML attachments from the 3rd message. 1st and 2nd are reported as no attachments, be it with hasAttachment or getAttachments method.
If I do with only the first 2 messages, it get the PDF and XML attachments of the 2nd.
So it always get only the last message attachments.

'sequence' => \Webklex\PHPIMAP\IMAP::ST_MSGN
Everything is "seen" ok, I can download all attachments of all messages.

Test code:

$mesgs = $query->get(); // UNSEEN or ALL on [Gmail]/Teste

var_dump(['mesgs count' => $mesgs->count()]);

$mesgs->each(function($m){
            $attachs = $m->getAttachments();
            var_dump(['attachs count' => $attachs->count()]);

            $attachs->each(function($a){
                var_dump($a->getAttributes());
            });
});

Dumps:

'sequence' => \Webklex\PHPIMAP\IMAP::ST_UID

>> TAG3 SELECT "GHSix/Teste"
<< FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)

<< OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing \*)] Flags permitted.

<< OK [UIDVALIDITY 71] UIDs valid.

<< 3 EXISTS

<< 0 RECENT

<< OK [UIDNEXT 140] Predicted next UID.

<< OK [HIGHESTMODSEQ 48445442]

<< OK [READ-WRITE] GHSix/Teste selected. (Success)

>> TAG4 UID SEARCH UNSEEN
<< SEARCH 136 138 139

<< OK SEARCH completed (Success)

>> TAG5 UID FETCH 136,138,139 (FLAGS)
<< 1 FETCH (UID 136 FLAGS ())

<< 2 FETCH (UID 138 FLAGS ())

<< 3 FETCH (UID 139 FLAGS ())

<< OK Success

>> TAG6 UID FETCH 136,138,139 (RFC822.HEADER)
<< 1 FETCH (UID 136 RFC822.HEADER {3577}

<< 2 FETCH (UID 138 RFC822.HEADER {3580}

<< 3 FETCH (UID 139 RFC822.HEADER {3581}

<< OK Success

>> TAG7 UID FETCH 136,138,139 (RFC822.TEXT)
<< 1 FETCH (UID 136 RFC822.TEXT {72578}

<< 2 FETCH (UID 138 RFC822.TEXT {69196}

<< 3 FETCH (UID 139 RFC822.TEXT {24681}

<< 1 FETCH (UID 136 FLAGS (\Seen))

<< 2 FETCH (UID 138 FLAGS (\Seen))

<< 3 FETCH (UID 139 FLAGS (\Seen))

<< OK Success

>> TAG8 FETCH 1:* (UID)
<< 1 FETCH (UID 136)

<< 2 FETCH (UID 138)

<< 3 FETCH (UID 139)

<< OK Success

>> TAG9 UID STORE 136:136 -FLAGS.SILENT (\Seen)
<< 1 FETCH (UID 136 FLAGS ())

<< OK Success

>> TAG10 UID FETCH 136 (FLAGS)
<< 1 FETCH (UID 136 FLAGS ())

<< OK Success

>> TAG11 FETCH 1:* (UID)
<< 1 FETCH (UID 136)

<< 2 FETCH (UID 138)

<< 3 FETCH (UID 139)

<< OK Success

>> TAG12 UID STORE 138:138 -FLAGS.SILENT (\Seen)
<< 2 FETCH (UID 138 FLAGS ())

<< OK Success

>> TAG13 UID FETCH 138 (FLAGS)
<< 2 FETCH (UID 138 FLAGS ())

<< OK Success

>> TAG14 FETCH 1:* (UID)
<< 1 FETCH (UID 136)

<< 2 FETCH (UID 138)

<< 3 FETCH (UID 139)

<< OK Success

>> TAG15 UID STORE 139:139 -FLAGS.SILENT (\Seen)
<< 3 FETCH (UID 139 FLAGS ())

<< OK Success

>> TAG16 UID FETCH 139 (FLAGS)
<< 3 FETCH (UID 139 FLAGS ())

<< OK Success

/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:281:
array(1) {
  'mesgs count' =>
  int(3)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:285:
array(1) {
  'attachs count' =>
  int(0)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:285:
array(1) {
  'attachs count' =>
  int(0)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:285:
array(1) {
  'attachs count' =>
  int(2)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:289:
array(9) {
  'content' =>
  string(8630) "<?xml version="1.0" encoding="utf-8"?>
<cteProc xmlns="http://www.portalfiscal.inf.br/cte" versao="3.00"><CTe xmlns="http://www.portalfiscal.inf.br/cte"><infCte versao="3.00" Id="CTe41210209090999000424570000000383431000419408"><ide><cUF>41</cUF><cCT>00041940</cCT><CFOP>6351</CFOP><natOp>PRESTACAO DE SERVICO DE TRANSPORTE</natOp><mod>57</mod><serie>0</serie><nCT>38343</nCT><dhEmi>2021-02-17T16:24:00-03:00</dhEmi><tpImp>1</tpImp><tpEmis>1</tpEmis><cDV>8</cDV><tpAmb>1</tpAmb><tpCTe>0</tpCTe><procEmi>0</procEm"...
  'type' =>
  string(4) "text"
  'part_number' =>
  int(1)
  'content_type' =>
  string(8) "text/xml"
  'id' =>
  NULL
  'name' =>
  string(51) "CTe41210209090999000424570000000383431000419408.xml"
  'disposition' =>
  class Webklex\PHPIMAP\Attribute#406 (2) {
    protected $name =>
    string(19) "content_disposition"
    protected $values =>
    array(1) {
      [0] =>
      string(10) "attachment"
    }
  }
  'img_src' =>
  NULL
  'size' =>
  int(11810)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:289:
array(9) {
  'content' =>
  string(8398) "<?xml version="1.0" encoding="utf-8"?>
<cteProc xmlns="http://www.portalfiscal.inf.br/cte" versao="3.00"><CTe xmlns="http://www.portalfiscal.inf.br/cte"><infCte versao="3.00" Id="CTe35210204882803000178570010002832921000431904"><ide><cUF>35</cUF><cCT>00043190</cCT><CFOP>5353</CFOP><natOp>PRESTACAO DE SERVICO DE TRANSPORTE</natOp><mod>57</mod><serie>1</serie><nCT>283292</nCT><dhEmi>2021-02-17T16:22:00-03:00</dhEmi><tpImp>1</tpImp><tpEmis>1</tpEmis><cDV>4</cDV><tpAmb>1</tpAmb><tpCTe>0</tpCTe><procEmi>0</procE"...
  'type' =>
  string(4) "text"
  'part_number' =>
  int(2)
  'content_type' =>
  string(8) "text/xml"
  'id' =>
  NULL
  'name' =>
  string(51) "CTe35210204882803000178570010002832921000431904.xml"
  'disposition' =>
  class Webklex\PHPIMAP\Attribute#445 (2) {
    protected $name =>
    string(19) "content_disposition"
    protected $values =>
    array(1) {
      [0] =>
      string(10) "attachment"
    }
  }
  'img_src' =>
  NULL
  'size' =>
  int(11494)
}
>> TAG17 LOGOUT
<< BYE LOGOUT Requested

<< OK 73 good day (Success)

'sequence' => \Webklex\PHPIMAP\IMAP::ST_MSGN

>> TAG3 SELECT "GHSix/Teste"
<< FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)

<< OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing \*)] Flags permitted.

<< OK [UIDVALIDITY 71] UIDs valid.

<< 3 EXISTS

<< 0 RECENT

<< OK [UIDNEXT 140] Predicted next UID.

<< OK [HIGHESTMODSEQ 48445520]

<< OK [READ-WRITE] GHSix/Teste selected. (Success)

>> TAG4 SEARCH UNSEEN
<< SEARCH 1 2 3

<< OK SEARCH completed (Success)

>> TAG5 FETCH 1,2,3 (FLAGS)
<< 1 FETCH (FLAGS ())

<< 2 FETCH (FLAGS ())

<< 3 FETCH (FLAGS ())

<< OK Success

>> TAG6 FETCH 1,2,3 (RFC822.HEADER)
<< 1 FETCH (RFC822.HEADER {3577}

<< 2 FETCH (RFC822.HEADER {3580}

<< 3 FETCH (RFC822.HEADER {3581}

<< OK Success

>> TAG7 FETCH 1,2,3 (RFC822.TEXT)
<< 1 FETCH (RFC822.TEXT {72578}

<< 2 FETCH (RFC822.TEXT {69196}

<< 3 FETCH (RFC822.TEXT {24681}

<< OK Success

>> TAG8 FETCH 1:* (UID)
<< 1 FETCH (UID 136)

<< 2 FETCH (UID 138)

<< 3 FETCH (UID 139)

<< OK Success

>> TAG9 STORE 1:1 -FLAGS.SILENT (\Seen)
<< OK Success

>> TAG10 FETCH 1 (FLAGS)
<< 1 FETCH (FLAGS ())

<< OK Success

>> TAG11 FETCH 1:* (UID)
<< 1 FETCH (UID 136)

<< 2 FETCH (UID 138)

<< 3 FETCH (UID 139)

<< OK Success

>> TAG12 STORE 2:2 -FLAGS.SILENT (\Seen)
<< OK Success

>> TAG13 FETCH 2 (FLAGS)
<< 2 FETCH (FLAGS ())

<< OK Success

>> TAG14 FETCH 1:* (UID)
<< 1 FETCH (UID 136)

<< 2 FETCH (UID 138)

<< 3 FETCH (UID 139)

<< OK Success

>> TAG15 STORE 3:3 -FLAGS.SILENT (\Seen)
<< OK Success

>> TAG16 FETCH 3 (FLAGS)
<< 3 FETCH (FLAGS ())

<< OK Success

/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:281:
array(1) {
  'mesgs count' =>
  int(3)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:285:
array(1) {
  'attachs count' =>
  int(2)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:289:
array(9) {
  'content' =>
  string(42778) "%PDF-1.3
3 0 obj
<</Type /Page
/Parent 1 0 R
/Resources 2 0 R
/Contents 4 0 R>>
endobj
4 0 obj
<</Filter /FlateDecode /Length 8089>>
stream
x��]Ks$�q�ϯ����3�*�5�bw�VM6Տ�Z�j�ZQAW�J�_g�
t��7��}���P�������3_*�@�G[���d�����W�$��UJ���^�Ҥ�,IMe�Y���Wy���D�-`;+J�y��j��UE@�,K���?��$��'3W��3�▒͈|,�0�}��3��n��*Ge�
&-n=$E
k�)*|��:8ݙ�w�)�j���f�{�;��o
                           <��Y��7��(�����b9_5����~i6����5�֜/7�f�޴���r�[�������=_^�����6���r��W�ss�ܼ[����w���,�{��|~�ik�6����'a��SA?5�߂��Ol�uv�*+��*͊ۅ"...
  'type' =>
  string(4) "text"
  'part_number' =>
  int(1)
  'content_type' =>
  string(17) ""application/pdf""
  'id' =>
  NULL
  'name' =>
  string(51) "CTe41210202058135000990570010000527061000531112.pdf"
  'disposition' =>
  class Webklex\PHPIMAP\Attribute#300 (2) {
    protected $name =>
    string(19) "content_disposition"
    protected $values =>
    array(1) {
      [0] =>
      string(10) "attachment"
    }
  }
  'img_src' =>
  NULL
  'size' =>
  int(58540)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:289:
array(9) {
  'content' =>
  string(9245) "<?xml version="1.0" encoding="utf-8"?>
<cteProc xmlns="http://www.portalfiscal.inf.br/cte" versao="3.00"><CTe xmlns="http://www.portalfiscal.inf.br/cte"><infCte versao="3.00" Id="CTe41210202058135000990570010000527061000531112"><ide><cUF>41</cUF><cCT>00053111</cCT><CFOP>6353</CFOP><natOp>PRESTACAO DE SERVICO DE TRANSPORTE</natOp><mod>57</mod><serie>1</serie><nCT>52706</nCT><dhEmi>2021-02-17T16:22:00-03:00</dhEmi><tpImp>1</tpImp><tpEmis>1</tpEmis><cDV>2</cDV><tpAmb>1</tpAmb><tpCTe>0</tpCTe><procEmi>0</procEm"...
  'type' =>
  string(4) "text"
  'part_number' =>
  int(2)
  'content_type' =>
  string(8) "text/xml"
  'id' =>
  NULL
  'name' =>
  string(51) "CTe41210202058135000990570010000527061000531112.xml"
  'disposition' =>
  class Webklex\PHPIMAP\Attribute#339 (2) {
    protected $name =>
    string(19) "content_disposition"
    protected $values =>
    array(1) {
      [0] =>
      string(10) "attachment"
    }
  }
  'img_src' =>
  NULL
  'size' =>
  int(12652)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:285:
array(1) {
  'attachs count' =>
  int(1)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:289:
array(9) {
  'content' =>
  string(49688) "%PDF-1.3
3 0 obj
<</Type /Page
/Parent 1 0 R
/Resources 2 0 R
/Contents 4 0 R>>
endobj
4 0 obj
<</Filter /FlateDecode /Length 9745>>
stream
x��]Ks9r��W����uH�£^���MM�I6��Tl8��Z:HQ�83���a�}�����w�|2�\000��P]��g׎����      $�!�B}��o^䓢�~v���<{�B���|��n*�~��I��Z7e��E�&*g|ﰞ���6��:TM�2�rb��C�B.�I^��      �[glJ5�2�{�A|�Bk*;Q�a�
                                                                                                                                                                                      � {/�z�
zߩ�{p�m��*{}���5Mv��lv���<���릙��=�&e�]}�~u2����E���,[Φ����]e����=Y,���Of��Y6]\|7���gW
                                                                                      w)�Z�����ݚ�g����l���ߺ�^f��dq�������r9?Yd��r9[���1����_gW��;_�%��"...
  'type' =>
  string(4) "text"
  'part_number' =>
  int(1)
  'content_type' =>
  string(17) ""application/pdf""
  'id' =>
  NULL
  'name' =>
  string(51) "CTe35210204882803000178570010002832951000431876.pdf"
  'disposition' =>
  class Webklex\PHPIMAP\Attribute#369 (2) {
    protected $name =>
    string(19) "content_disposition"
    protected $values =>
    array(1) {
      [0] =>
      string(10) "attachment"
    }
  }
  'img_src' =>
  NULL
  'size' =>
  int(67994)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:285:
array(1) {
  'attachs count' =>
  int(2)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:289:
array(9) {
  'content' =>
  string(8630) "<?xml version="1.0" encoding="utf-8"?>
<cteProc xmlns="http://www.portalfiscal.inf.br/cte" versao="3.00"><CTe xmlns="http://www.portalfiscal.inf.br/cte"><infCte versao="3.00" Id="CTe41210209090999000424570000000383431000419408"><ide><cUF>41</cUF><cCT>00041940</cCT><CFOP>6351</CFOP><natOp>PRESTACAO DE SERVICO DE TRANSPORTE</natOp><mod>57</mod><serie>0</serie><nCT>38343</nCT><dhEmi>2021-02-17T16:24:00-03:00</dhEmi><tpImp>1</tpImp><tpEmis>1</tpEmis><cDV>8</cDV><tpAmb>1</tpAmb><tpCTe>0</tpCTe><procEmi>0</procEm"...
  'type' =>
  string(4) "text"
  'part_number' =>
  int(1)
  'content_type' =>
  string(8) "text/xml"
  'id' =>
  NULL
  'name' =>
  string(51) "CTe41210209090999000424570000000383431000419408.xml"
  'disposition' =>
  class Webklex\PHPIMAP\Attribute#431 (2) {
    protected $name =>
    string(19) "content_disposition"
    protected $values =>
    array(1) {
      [0] =>
      string(10) "attachment"
    }
  }
  'img_src' =>
  NULL
  'size' =>
  int(11810)
}
/www/dev/eguarda/php/script/fetchattach/app/Src/ImapFetchAttach.php:289:
array(9) {
  'content' =>
  string(8398) "<?xml version="1.0" encoding="utf-8"?>
<cteProc xmlns="http://www.portalfiscal.inf.br/cte" versao="3.00"><CTe xmlns="http://www.portalfiscal.inf.br/cte"><infCte versao="3.00" Id="CTe35210204882803000178570010002832921000431904"><ide><cUF>35</cUF><cCT>00043190</cCT><CFOP>5353</CFOP><natOp>PRESTACAO DE SERVICO DE TRANSPORTE</natOp><mod>57</mod><serie>1</serie><nCT>283292</nCT><dhEmi>2021-02-17T16:22:00-03:00</dhEmi><tpImp>1</tpImp><tpEmis>1</tpEmis><cDV>4</cDV><tpAmb>1</tpAmb><tpCTe>0</tpCTe><procEmi>0</procE"...
  'type' =>
  string(4) "text"
  'part_number' =>
  int(2)
  'content_type' =>
  string(8) "text/xml"
  'id' =>
  NULL
  'name' =>
  string(51) "CTe35210204882803000178570010002832921000431904.xml"
  'disposition' =>
  class Webklex\PHPIMAP\Attribute#470 (2) {
    protected $name =>
    string(19) "content_disposition"
    protected $values =>
    array(1) {
      [0] =>
      string(10) "attachment"
    }
  }
  'img_src' =>
  NULL
  'size' =>
  int(11494)
}
>> TAG17 LOGOUT
<< BYE LOGOUT Requested

<< OK 73 good day (Success)

If dumping of anything at any moment in the code can help, please, just let me know.

Updated with ImapProtocol debug enabled.

The text was updated successfully, but these errors were encountered:

EthraZa · 2021-10-28T17:24:31Z

Hi, I see that version 2.7.0 implemented the moveManyMessages method, but I can't find this on the docs.
Will this method help me in Gmail using ST_MSGN?

Also, can we hope to see the php-imap evolving or is it stalled?

Webklex · 2021-10-28T20:26:39Z

Hi @EthraZa ,
is there a certain reason you have to use ST_MSGN instead of ST_UID?
What happens if you use the moveManyMessages() method? :)

If you are missing a certain feature, feel free to create a new feature or pull request and lets talk or discuss 👍

Best regards,

EthraZa · 2021-10-28T20:41:23Z

Hi.

This issue itself, if I use ST_UID with Gmail, I can't get the attachments.
About moveManyMessages I don't know, I just saw it in the releases description - I'm not updating php-imap anymore if not extremely necessary, mainly because of pending #150 PR.

Webklex · 2021-10-28T20:48:40Z

I see. I'm scared to touch the boundary. Every time its changed, it breaks something else..

What do you think if the boundary regex gets moved to the config - so everyone who likes to play with it can do so? Perhaps we find a solid one (perhaps your regex is perfect - I don't know).

Best regards,

EthraZa · 2021-10-28T20:56:27Z

I myself, like to put in the config everything that can be in the config. The boundary regexp there is indead a great idea.

Right now I'm using php-imap in a project that is reading about 2000 email acconts that send 100's to 1000's each everyday with XML, ZIP and PDFs attacheds, till now, my regexp is doing great.

Webklex added bug Something isn't working validating labels Feb 24, 2021

Webklex added a commit that referenced this issue Oct 28, 2021

Boundary regex config doc added #169 #150 #126 #121 #111 #152 #108

8a8321c

Webklex added the Gmail Related to Google mail / GMAIL label Nov 4, 2021

Webklex closed this as completed Nov 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Only last message in the query has attachments within IMAP::ST_UID on Gmail #108

Only last message in the query has attachments within IMAP::ST_UID on Gmail #108

EthraZa commented Feb 17, 2021 •

edited

Loading

EthraZa commented Oct 28, 2021

Webklex commented Oct 28, 2021

EthraZa commented Oct 28, 2021 •

edited

Loading

Webklex commented Oct 28, 2021

EthraZa commented Oct 28, 2021

Only last message in the query has attachments within IMAP::ST_UID on Gmail #108

Only last message in the query has attachments within IMAP::ST_UID on Gmail #108

Comments

EthraZa commented Feb 17, 2021 • edited Loading

EthraZa commented Oct 28, 2021

Webklex commented Oct 28, 2021

EthraZa commented Oct 28, 2021 • edited Loading

Webklex commented Oct 28, 2021

EthraZa commented Oct 28, 2021

EthraZa commented Feb 17, 2021 •

edited

Loading

EthraZa commented Oct 28, 2021 •

edited

Loading