[1LP][RFR] Fix ftp failures due to year change

[valaparthvi]

Purpose or Intent

• Fixing FTPClient.ls() which in my opinion is failing due to the change in year.

Normal lines
-rwxrwxrwx 1 ftp ftp 1602 Jul 25 05:02 filter_report.yaml

Changed lines
-rwxrwxrwx 1 ftp ftp 1361 Jun 10 2019 import_report.yaml

Error:

ValueError                                Traceback (most recent call last)
<ipython-input-13-eec0aba7fc42> in <module>
----> 1 fs.ls()

~/code/cfme_workspace/integration_tests/cfme/utils/ftp.py in ls(self)
    343             result.append((is_dir, fields[-1], date))
    344 
--> 345         self.ftp.dir(_callback)
    346         return result
    347 

/usr/lib64/python3.7/ftplib.py in dir(self, *args)
    573             if arg:
    574                 cmd = cmd + (' ' + arg)
--> 575         self.retrlines(cmd, func)
    576 
    577     def mlsd(self, path="", facts=[]):

/usr/lib64/python3.7/ftplib.py in retrlines(self, cmd, callback)
    480                 elif line[-1:] == '\n':
    481                     line = line[:-1]
--> 482                 callback(line)
    483             # shutdown ssl layer
    484             if _SSLSocket is not None and isinstance(conn, _SSLSocket):

~/code/cfme_workspace/integration_tests/cfme/utils/ftp.py in _callback(line)
    336             # Nov 11 12:34 filename (from the end)
    337             import ipdb; ipdb.set_trace()
--> 338             date = strptime(str(datetime.now().year) + " " + fields[-4] + " " + fields[-3] + " " +
    339                             fields[-2],
    340                             "%Y %b %d %H:%M")

/usr/lib64/python3.7/_strptime.py in _strptime_time(data_string, format)
    569     """Return a time struct based on the input string and the
    570     format string."""
--> 571     tt = _strptime(data_string, format)[0]
    572     return time.struct_time(tt[:time._STRUCT_TM_ITEMS])
    573 

/usr/lib64/python3.7/_strptime.py in _strptime(data_string, format)
    357     if not found:
    358         raise ValueError("time data %r does not match format %r" %
--> 359                          (data_string, format))
    360     if len(data_string) != found.end():
    361         raise ValueError("unconverted data remains: %s" %

ValueError: time data '2020 Jun 10 2019' does not match format '%Y %b %d %H:%M'

[digitronik]

@valaparthvi thank for quick catch.
I think lets check first its back year file or not as per that we will put year

yr = datetime.now().year if ":" in fields[-2] else fields[-2]
time = fields[-2] if ":" in fields[-2] else "00:00"
date = strptime(f"{yr} {fields[-4]} {fields[-3]} {time}", "%Y %b %d %H:%M")

[digitronik]

I don't this ValueError handling is better way.
As per my observation. when ftp file/dir created in back year then it will remove time and put year.

[digitronik]

Your taking current year here i.e 2020 for last year; which is wrong.

[ganeshhubale]

Hi @digitronik Just curious :)

• I can see 2020 year for all the files uploaded as below:

In [9]: fs.ls()                                                                                                                                                              
Out[9]: 
[(False, 'bz_1364407.zip', datetime.datetime(2020, 12, 19, 7, 30)),
 (False, 'bz_1440226.zip', datetime.datetime(2020, 10, 9, 4, 34)),
 (False, 'bz_1558926.yml', datetime.datetime(2020, 11, 7, 3, 11)),
 (False, 'bz_1558926.zip', datetime.datetime(2020, 11, 7, 1, 30)),
 (False, 'bz_1559999.zip', datetime.datetime(2020, 12, 19, 2, 40)),
 (False, 'bz_1570152.yml', datetime.datetime(2020, 9, 13, 5, 18)),
 (False, 'bz_1570152.zip', datetime.datetime(2020, 9, 13, 3, 30)),
 (False, 'bz_1571000.yml', datetime.datetime(2020, 10, 24, 4, 24)),
 (False, 'bz_1571000.zip', datetime.datetime(2020, 10, 24, 4, 23)),
 (False, 'bz_1576107.zip', datetime.datetime(2020, 12, 19, 8, 27)),
 (False, 'bz_1613443.yml', datetime.datetime(2020, 11, 21, 0, 6)),
 (False, 'bz_1613443.zip', datetime.datetime(2020, 11, 21, 0, 7)),
 (False, 'bz_1614436.yml', datetime.datetime(2020, 9, 18, 6, 42)),
.....

• But most of these added in 2019. So why it is not showing exact date of that file uploaded(only about old files otherwise current files are taking expected datetime)?
• Otherwise looks good to me :)

[digitronik]

@ganeshhubale this is due to time is available for those file and no year.

In [6]: fs.ftp.dir()                                                                                                                              
-rw-rw-r--   1 root     ftp        985195 Dec 19 07:30 bz_1364407.zip
-rwxrwxrwx   1 ftp      ftp          1607 Oct  9 04:34 bz_1440226.zip
-rw-rw-r--   1 root     ftp          1512 Nov  7 03:11 bz_1558926.yml
-rw-rw-r--   1 root     ftp          3902 Nov  7 01:30 bz_1558926.zip
-rw-rw-r--   1 root     ftp        980499 Dec 19 02:40 bz_1559999.zip
-rwxrwxrwx   1 ftp      ftp          2577 Sep 13 05:18 bz_1570152.yml
-rwxrwxrwx   1 ftp      ftp        977247 Sep 13 03:30 bz_1570152.zip
-rw-rw-r--   1 root     ftp          3503 Oct 24 04:24 bz_1571000.yml
-rw-rw-r--   1 root     ftp          8843 Oct 24 04:23 bz_1571000.zip