Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vcs.plot(iso) creates text object that are not removed after clear #1424

Closed
durack1 opened this issue Jun 24, 2015 · 56 comments · Fixed by #1449
Closed

vcs.plot(iso) creates text object that are not removed after clear #1424

durack1 opened this issue Jun 24, 2015 · 56 comments · Fixed by #1449
Assignees
Milestone

Comments

@durack1
Copy link
Member

durack1 commented Jun 24, 2015

Not related to #1397 (bg=1 memory leak)

@durack1
Copy link
Member Author

durack1 commented Jun 24, 2015

@doutriaux1 it seems you've nailed where the slow down is happening, which is text* (and maybe some other vcs) objects accumulating throughout a loop.

The following code:

bg = False
counter = 1
for var in ['sic','sst']:
    ...
    for data in ['obs','bcs']:
    ...
        x = vcs.init()
        for count,y in enumerate(range(1870,2013)):
        ...
            for m in range(12):
                startTime                   = time.time()
                ...
                title.string                = '%i-%.2i' % (y,m+1)
                x.plot(title,bg=bg)
                x.plot(s1,t1,iso1,bg=bg)
                x.plot(diff,t2,iso2,bg=bg)
                x.plot(s2,t3,iso1,bg=bg)
                ...
                x.png(fileName)
                x.clear()
                del(vcs.elements["isofill"][iso1.name])
                del(vcs.elements["isofill"][iso2.name])
                del(vcs.elements["textcombined"][title.name])                
                del(vcs.elements["template"][t1.name])
                del(vcs.elements["template"][t2.name])
                del(vcs.elements["template"][t3.name])
                for nm in vcs.elements["texttable"].keys():
                    if not nm in basic_tt:
                        del(vcs.elements["texttable"][nm])
                for nm in vcs.elements["textorientation"].keys():
                    if not nm in basic_to:
                        del(vcs.elements["textorientation"][nm])
                for nm in vcs.elements["textcombined"].keys():
                    if not nm in basic_tc:
                        del(vcs.elements["textcombined"][nm])
                endTime                 = time.time()
                print counterStr,printStr,varName.ljust(6),BC,timeStr,memStr
                counter                 = counter+1
            gc.collect() ; # Attempt to force a memory flush
        x.close()     

Gets this output:

00001 processing: 1870-01 sic     Time: 04.401 secs; Max mem: 0.237 GB
00150 processing: 1882-06 sic     Time: 03.026 secs; Max mem: 0.244 GB
00300 processing: 1894-12 sic     Time: 03.048 secs; Max mem: 0.256 GB
00500 processing: 1911-08 sic     Time: 03.061 secs; Max mem: 0.262 GB
01000 processing: 1953-04 sic     Time: 03.079 secs; Max mem: 0.286 GB
01500 processing: 1994-12 sic     Time: 03.149 secs; Max mem: 0.306 GB
02000 processing: 1893-08 sic  bc Time: 03.640 secs; Max mem: 0.351 GB
02500 processing: 1935-04 sic  bc Time: 03.333 secs; Max mem: 0.374 GB
03000 processing: 1976-12 sic  bc Time: 03.465 secs; Max mem: 0.393 GB
03500 processing: 1875-08 tos     Time: 03.354 secs; Max mem: 0.418 GB
04000 processing: 1917-04 tos     Time: 03.679 secs; Max mem: 0.429 GB
05000 processing: 2000-08 tos     Time: 03.349 secs; Max mem: 0.496 GB
06000 processing: 1940-12 tos  bc Time: 03.656 secs; Max mem: 0.529 GB
06864 processing: 2012-12 tos  bc Time: 03.588 secs; Max mem: 0.567 GB

For comparison, the timings and resource usage (which were erroneously listed in #1397 - doesn't have much to do with the bg=True issue title) without these explicit del() were:

00001 processing: 1870-01 sic     Time: 04.093 secs; Max mem: 0.238 GB
00150 processing: 1882-06 sic     Time: 06.176 secs; Max mem: 0.255 GB
00300 processing: 1894-12 sic     Time: 10.063 secs; Max mem: 0.261 GB
00500 processing: 1911-08 sic     Time: 15.334 secs; Max mem: 0.280 GB
01000 processing: 1953-04 sic     Time: 31.086 secs; Max mem: 0.320 GB
01500 processing: 1994-12 sic     Time: 48.595 secs; Max mem: 0.352 GB
02000 processing: 1893-08 sic  bc Time: 68.674 secs; Max mem: 0.389 GB
02500 processing: 1935-04 sic  bc Time: 90.528 secs; Max mem: 0.480 GB
03000 processing: 1976-12 sic  bc Time: 111.397 secs; Max mem: 0.480 GB
03500 processing: 1875-08 tos     Time: 134.129 secs; Max mem: 0.498 GB
04000 processing: 1917-04 tos     Time: 163.216 secs; Max mem: 0.517 GB
04236 processing: 1936-12 tos     Time: 174.415 secs; Max mem: 0.525 GB

It would be great if vcs clear and vcs close did a better job of cleaning up after themselves..

@durack1
Copy link
Member Author

durack1 commented Jun 24, 2015

@doutriaux1 as an aside, it also appears that the vcs ffmpeg call is also not releasing objects:

x = vcs.init()
x.ffmpeg('out.mp4',outFileList,rate=5,bitrate=2048)
x.close()

The first time it's called it seems to jump the memory usage a little:

01716 processing: 2012-12 sic     Time: 03.285 secs; Max mem: 0.319 GB
Processing:  AMIPBCS_newVsOld_sic.mp4
01717 processing: 1870-01 sic  bc Time: 03.798 secs; Max mem: 0.333 GB

Whereas subsequent calls don't appear to do the same:

03432 processing: 2012-12 sic  bc Time: 03.587 secs; Max mem: 0.418 GB
Processing:  AMIPBCS_newVsOld_sicbcs.mp4
03433 processing: 1870-01 tos     Time: 04.039 secs; Max mem: 0.418 GB
05148 processing: 2012-12 tos     Time: 03.354 secs; Max mem: 0.501 GB
Processing:  AMIPBCS_newVsOld_tos.mp4
05149 processing: 1870-01 tos  bc Time: 04.334 secs; Max mem: 0.501 GB

@durack1
Copy link
Member Author

durack1 commented Jun 25, 2015

@doutriaux1 this will get you there - it's the script which reports the time and mem usage above:
https://github.com/PCMDI/amipbcs/blob/master/make_newVsOldDiffs.py

@durack1
Copy link
Member Author

durack1 commented Jul 16, 2015

@doutriaux1 seems like there are some more things which aren't being purged, reopening..

Code that includes:

del(vcs.elements["isofill"][iso1.name])
del(vcs.elements["isofill"][iso2.name])
del(vcs.elements["textcombined"][title.name])                
del(vcs.elements["template"][t1.name])
del(vcs.elements["template"][t2.name])
del(vcs.elements["template"][t3.name])
for nm in vcs.elements["texttable"].keys():
  if not nm in basic_tt:
    del(vcs.elements["texttable"][nm])
for nm in vcs.elements["textorientation"].keys():
  if not nm in basic_to:
    del(vcs.elements["textorientation"][nm])
for nm in vcs.elements["textcombined"].keys():
  if not nm in basic_tc:
    del(vcs.elements["textcombined"][nm])

Yields the following timings/memory usage in a loop:

UV-CDAT version:  2.2.0
UV-CDAT prefix:  /usr/local/uvcdat/2.2.0
delFudge:  True
count                             frameTime          residentMemory
00001 processing: 1870-01 sic     Time: 03.241 secs; Max mem: 0.241 GB
...
00500 processing: 1911-08 sic     Time: 03.179 secs; Max mem: 0.274 GB
...
01000 processing: 1953-04 sic     Time: 03.169 secs; Max mem: 0.295 GB

And without:

UV-CDAT version:  2.2.0-204-geef3c60
UV-CDAT prefix:  /usr/local/uvcdat/2015-07-14-full
delFudge:  False
count                             frameTime          residentMemory
00001 processing: 1870-01 sic     Time: 03.225 secs; Max mem: 0.237 GB
...
00010 processing: 1870-10 sic     Time: 05.558 secs; Max mem: 0.238 GB
...
00020 processing: 1871-08 sic     Time: 09.972 secs; Max mem: 0.240 GB
...
00030 processing: 1872-06 sic     Time: 15.952 secs; Max mem: 0.241 GB
...
00040 processing: 1873-04 sic     Time: 23.081 secs; Max mem: 0.243 GB
...
00050 processing: 1874-02 sic     Time: 31.757 secs; Max mem: 0.246 GB
...
00060 processing: 1874-12 sic     Time: 42.138 secs; Max mem: 0.246 GB
...
00070 processing: 1875-10 sic     Time: 53.792 secs; Max mem: 0.250 GB
...
00080 processing: 1876-08 sic     Time: 68.210 secs; Max mem: 0.256 GB
...
00090 processing: 1877-06 sic     Time: 78.360 secs; Max mem: 0.263 GB
...
00100 processing: 1878-04 sic     Time: 93.911 secs; Max mem: 0.266 GB
...
00110 processing: 1879-02 sic     Time: 111.486 secs; Max mem: 0.267 GB

And just for completeness, adding back in the del statements, we get:

UV-CDAT version:  2.2.0-204-geef3c60
UV-CDAT prefix:  /usr/local/uvcdat/2015-07-14-full
delFudge:  True
count                             frameTime          residentMemory
00001 processing: 1870-01 sic     Time: 03.795 secs; Max mem: 0.241 GB
...
00020 processing: 1871-08 sic     Time: 03.140 secs; Max mem: 0.242 GB
...
00050 processing: 1874-02 sic     Time: 03.164 secs; Max mem: 0.242 GB
...
00100 processing: 1878-04 sic     Time: 03.131 secs; Max mem: 0.243 GB
...
00200 processing: 1886-08 sic     Time: 03.139 secs; Max mem: 0.245 GB
...
00500 processing: 1911-08 sic     Time: 03.179 secs; Max mem: 0.255 GB
...
01000 processing: 1953-04 sic     Time: 03.244 secs; Max mem: 0.271 GB
...
01500 processing: 1994-12 sic     Time: 03.384 secs; Max mem: 0.276 GB
...
06864 processing: 2012-12 tos  bc Time: 07.218 secs; Max mem: 0.376 GB

@durack1 durack1 reopened this Jul 16, 2015
@durack1
Copy link
Member Author

durack1 commented Jul 17, 2015

@doutriaux1 looks to me like even the del command above is not getting everything.. The memory footprint and time taken for step 6864 above shows another leak somewhere..

@doutriaux1
Copy link
Contributor

@durack1 we know there's a tiny tiniy leak in cdtime (8bits I believe) . Lots of relative/component conversion could lead to big leak eventually. But that might not be it.

@durack1
Copy link
Member Author

durack1 commented Jul 20, 2015

@doutriaux1, nope cdtime is never loaded.. So doubtful that's it..

@doutriaux1
Copy link
Contributor

@durack1 if you use cdms2 and have a time axis, you are using cdtime and trigger that leak.

@durack1
Copy link
Member Author

durack1 commented Jul 20, 2015

@doutriaux1 not sure this is enough to trigger it:

https://github.com/PCMDI/amipbcs/blob/master/make_newVsOldDiffs.py#L171-177

There's no explicit time axis manipulation..

@doutriaux1
Copy link
Contributor

@dnadeau4 we should take a look at the tiniy cdtime/cdms leak

@durack1
Copy link
Member Author

durack1 commented Sep 8, 2015

@doutriaux1 did you want me to run my test on current master - are there changes in there that should solve this issue?

@durack1
Copy link
Member Author

durack1 commented Nov 19, 2015

@doutriaux1 to see where 2.4.0rc1 is at here's the log (it's another test using Xvfb to get around the existing bg=True problem #1397):

[duro@ocean 150219_AMIPForcingData]$ make_newVsOldDiffs.py
UV-CDAT version:  2.4.0rc1-8-gd23939e
UV-CDAT prefix:  /usr/local/uvcdat/2.4.0rc1
delFudge:  True
00001 processing: 1870-01 sic     Time: 03.731 secs; Max mem: 0.239 GB
...
00020 processing: 1871-08 sic     Time: 02.857 secs; Max mem: 0.241 GB
...
00050 processing: 1874-02 sic     Time: 02.892 secs; Max mem: 0.245 GB
...
00100 processing: 1878-04 sic     Time: 02.865 secs; Max mem: 0.246 GB
...
00200 processing: 1886-08 sic     Time: 02.934 secs; Max mem: 0.248 GB
...
00500 processing: 1911-08 sic     Time: 02.887 secs; Max mem: 0.257 GB
...
00750 processing: 1932-06 sic     Time: 02.998 secs; Max mem: 0.267 GB
...
01000 processing: 1953-04 sic     Time: 03.074 secs; Max mem: 0.274 GB
...
01500 processing: 1994-12 sic     Time: 05.771 secs; Max mem: 0.289 GB
...
02000 processing: 1893-08 sic    bc Time: 03.263 secs; Max mem: 0.291 GB
...
02500 processing: 1935-04 sic    bc Time: 03.494 secs; Max mem: 0.306 GB
...
05000 processing: 2000-08 tos     Time: 05.025 secs; Max mem: 0.345 GB
...
06000 processing: 1940-12 tos    bc Time: 06.134 secs; Max mem: 0.356 GB
...
06864 processing: 2012-12 tos    bc Time: 06.850 secs; Max mem: 0.373 GB

So long story short, it's better than previously, but could do with another look before closing..

@chaosphere2112 here are the numbers I mentioned earlier..

@durack1
Copy link
Member Author

durack1 commented Nov 19, 2015

@chaosphere2112 I was curious about your python objects count in #1397, so added an object count to the code above, this does appear to be a cause of my problem:

[duro@ocean 150219_AMIPForcingData]$ more make_newVsOldDiffs7.txt 
UV-CDAT version:  2.4.0rc1-8-gd23939e
UV-CDAT prefix:  /usr/local/uvcdat/2.4.0rc1
delFudge:  True
00001 processing: 1870-01 sic     Time: 03.313 secs; Max mem: 0.239 GB PyObj#: 0050634;
...
00005 processing: 1870-05 sic     Time: 03.096 secs; Max mem: 0.240 GB PyObj#: 0050839;
...
00010 processing: 1870-10 sic     Time: 02.913 secs; Max mem: 0.242 GB PyObj#: 0051099;
...
00020 processing: 1871-08 sic     Time: 02.928 secs; Max mem: 0.244 GB PyObj#: 0051621;
...
00030 processing: 1872-06 sic     Time: 02.920 secs; Max mem: 0.244 GB PyObj#: 0052143;
...
00040 processing: 1873-04 sic     Time: 02.925 secs; Max mem: 0.244 GB PyObj#: 0052665;
...
00050 processing: 1874-02 sic     Time: 02.944 secs; Max mem: 0.244 GB PyObj#: 0053187;
...
00500 processing: 1911-08 sic     Time: 02.863 secs; Max mem: 0.253 GB PyObj#: 0076581;
...
01000 processing: 1953-04 sic     Time: 03.011 secs; Max mem: 0.269 GB PyObj#: 0102585;
...
02000 processing: 1893-08 sic    bc Time: 03.354 secs; Max mem: 0.287 GB PyObj#: 0154686;
...
03000 processing: 1976-12 sic    bc Time: 03.819 secs; Max mem: 0.308 GB PyObj#: 0206692;
...
04000 processing: 1917-04 tos     Time: 04.239 secs; Max mem: 0.328 GB PyObj#: 0258811;
...
05000 processing: 2000-08 tos     Time: 05.023 secs; Max mem: 0.346 GB PyObj#: 0310817;
...
06000 processing: 1940-12 tos    bc Time: 06.097 secs; Max mem: 0.362 GB PyObj#: 0362918;
...
06864 processing: 2012-12 tos    bc Time: 07.090 secs; Max mem: 0.379 GB PyObj#: 0407830;

So a further cleanup of these objects would be a very good idea indeed. Not sure where things stand in 2.4.0rc2+

@chaosphere2112
Copy link
Contributor

@durack1 Nope, not yet. I'll try and get a branch up this afternoon, I've been on other projects for the last month or so.

@chaosphere2112
Copy link
Contributor

@durack1 Also, if you want to eliminate your delFudge stuff, you can plot your labels using "donotstoredisplay=True"

@durack1
Copy link
Member Author

durack1 commented Jun 30, 2016

@chaosphere2112 great, I'm just running with that additional argument now in 2.6.0 (and my delFudge=0) I'll let you know what it looks like.. I'm also going to be interested to see how this impacts the plot speed - should there be any changes there with your 2.6.0 tweaks?

@durack1
Copy link
Member Author

durack1 commented Jul 5, 2016

@chaosphere2112 I just ran the script with donotstoredisplay=True and this is the result (also turned off the delFudge tweak) - it seems things are still very broken, 2315+ seconds and 24+ GB to plot a 3 panel figure seems a little extreme to me:

UV-CDAT version: v2.5.0-388-g9d06e08
UV-CDAT prefix: /export/durack1/anaconda2/envs/uvcdat
delFudge: False
Background graphics: True
donotstoredisplay: True
00001 processing: 1870-01 sic     Time: 07.361 secs; Max mem: 1.118 GB PyObj#: 0092480;
00002 processing: 1870-02 sic     Time: 05.179 secs; Max mem: 1.551 GB PyObj#: 0093276;
00003 processing: 1870-03 sic     Time: 04.296 secs; Max mem: 1.595 GB PyObj#: 0093771;
00004 processing: 1870-04 sic     Time: 05.091 secs; Max mem: 1.638 GB PyObj#: 0094265;
00005 processing: 1870-05 sic     Time: 04.540 secs; Max mem: 1.680 GB PyObj#: 0094760;
00006 processing: 1870-06 sic     Time: 06.039 secs; Max mem: 1.723 GB PyObj#: 0095255;
00007 processing: 1870-07 sic     Time: 05.725 secs; Max mem: 1.766 GB PyObj#: 0095750;
00008 processing: 1870-08 sic     Time: 06.299 secs; Max mem: 1.809 GB PyObj#: 0096245;
00009 processing: 1870-09 sic     Time: 05.634 secs; Max mem: 1.852 GB PyObj#: 0096740;
00010 processing: 1870-10 sic     Time: 05.728 secs; Max mem: 1.895 GB PyObj#: 0097235;
...
00015 processing: 1871-03 sic     Time: 09.083 secs; Max mem: 2.110 GB PyObj#: 0099710;
...
00020 processing: 1871-08 sic     Time: 14.018 secs; Max mem: 2.325 GB PyObj#: 0102185;
...
00025 processing: 1872-01 sic     Time: 16.569 secs; Max mem: 2.539 GB PyObj#: 0104360;
...
00050 processing: 1874-02 sic     Time: 37.991 secs; Max mem: 3.614 GB PyObj#: 0116882;
...
00075 processing: 1876-03 sic     Time: 69.513 secs; Max mem: 4.691 GB PyObj#: 0128791;
...
00100 processing: 1878-04 sic     Time: 115.786 secs; Max mem: 5.766 GB PyObj#: 0140544;
...
00150 processing: 1882-06 sic     Time: 210.337 secs; Max mem: 7.910 GB PyObj#: 0164830;
...
00200 processing: 1886-08 sic     Time: 352.287 secs; Max mem: 10.059 GB PyObj#: 0188492;
...
00250 processing: 1890-10 sic     Time: 535.705 secs; Max mem: 12.187 GB PyObj#: 0211198;
...
00300 processing: 1894-12 sic     Time: 762.082 secs; Max mem: 14.338 GB PyObj#: 0234080;
...
00350 processing: 1899-02 sic     Time: 1055.304 secs; Max mem: 16.458 GB PyObj#: 0256807;
...
00400 processing: 1903-04 sic     Time: 1338.400 secs; Max mem: 18.611 GB PyObj#: 0280157;
...
00450 processing: 1907-06 sic     Time: 1887.651 secs; Max mem: 20.769 GB PyObj#: 0303351;
...
00500 processing: 1911-08 sic     Time: 2166.998 secs; Max mem: 22.929 GB PyObj#: 0326837;
...
00525 processing: 1913-09 sic     Time: 2315.047 secs; Max mem: 24.007 GB PyObj#: 0337186;

@chaosphere2112
Copy link
Contributor

@durack1 Yeah, that's a bit much. I'm working on some other stuff right now, but I should be able to sit down with you and your script this week and see if anything leaps out to me.

@durack1
Copy link
Member Author

durack1 commented Jul 5, 2016

@chaosphere2112 no probs - I'm in this week so let me know..

@durack1
Copy link
Member Author

durack1 commented Jul 5, 2016

@doutriaux1 this issue along with a couple of others (#1770, #1931 and #1944) should probably be pinned to a 2.8 or similar milestone..?

@durack1
Copy link
Member Author

durack1 commented Jul 6, 2016

@chaosphere2112 just to see changes between 2.4.1 and 2.6.0 I just ran the same code with my delFudge=True and this is a snippet of the result - same story, there are some real problems remaining ~100GB is VERY extreme:

UV-CDAT version:      v2.5.0-388-g9d06e08
UV-CDAT prefix:       /export/durack1/anaconda2/envs/uvcdat
delFudge:             True
Background graphics:  True
donotstoredisplay:    True
00001 processing: 1870-01 sic     Time: 08.520 secs; Max mem: 1.118 GB PyObj#: 0092207;
00002 processing: 1870-02 sic     Time: 08.911 secs; Max mem: 1.551 GB PyObj#: 0092560;
00003 processing: 1870-03 sic     Time: 06.186 secs; Max mem: 1.595 GB PyObj#: 0092612;
00004 processing: 1870-04 sic     Time: 05.566 secs; Max mem: 1.637 GB PyObj#: 0092663;
00005 processing: 1870-05 sic     Time: 10.375 secs; Max mem: 1.680 GB PyObj#: 0092715;
00006 processing: 1870-06 sic     Time: 07.093 secs; Max mem: 1.723 GB PyObj#: 0092767;
00007 processing: 1870-07 sic     Time: 06.441 secs; Max mem: 1.766 GB PyObj#: 0092819;
00008 processing: 1870-08 sic     Time: 06.363 secs; Max mem: 1.809 GB PyObj#: 0092871;
00009 processing: 1870-09 sic     Time: 06.340 secs; Max mem: 1.852 GB PyObj#: 0092923;
00010 processing: 1870-10 sic     Time: 07.140 secs; Max mem: 1.895 GB PyObj#: 0092975;
...
00200 processing: 1886-08 sic     Time: 03.534 secs; Max mem: 10.039 GB PyObj#: 0102870;
...
00500 processing: 1911-08 sic     Time: 10.599 secs; Max mem: 22.875 GB PyObj#: 0118455;
...
01716 processing: 2012-12 sic     Time: 08.618 secs; Max mem: 75.644 GB PyObj#: 0181688;
Processing:  pngs/AMIPBCS_newVsOld_sic.mp4
01717 processing: 1870-01 sic    bc Time: 15.458 secs; Max mem: 76.091 GB PyObj#: 0181716;
...
03313 processing: 2003-01 sic    bc Time: 08.411 secs; Max mem: 76.091 GB PyObj#: 0264608;
03314 processing: 2003-02 sic    bc Time: 08.455 secs; Max mem: 76.091 GB PyObj#: 0264961;
03315 processing: 2003-03 sic    bc Time: 09.625 secs; Max mem: 76.091 GB PyObj#: 0265013;
03316 processing: 2003-04 sic    bc Time: 08.775 secs; Max mem: 76.126 GB PyObj#: 0265065;
03317 processing: 2003-05 sic    bc Time: 08.323 secs; Max mem: 76.175 GB PyObj#: 0265117;
03318 processing: 2003-06 sic    bc Time: 10.960 secs; Max mem: 76.224 GB PyObj#: 0265169;
03319 processing: 2003-07 sic    bc Time: 11.191 secs; Max mem: 76.273 GB PyObj#: 0265221;
...
06860 processing: 2012-08 tos    bc Time: 12.369 secs; Max mem: 100.068 GB PyObj#: 0449385;
06861 processing: 2012-09 tos    bc Time: 21.967 secs; Max mem: 100.117 GB PyObj#: 0449437;
06862 processing: 2012-10 tos    bc Time: 12.716 secs; Max mem: 100.165 GB PyObj#: 0449489;
06863 processing: 2012-11 tos    bc Time: 13.690 secs; Max mem: 100.216 GB PyObj#: 0449541;
06864 processing: 2012-12 tos    bc Time: 16.863 secs; Max mem: 100.271 GB PyObj#: 0449593;
Processing:  pngs/AMIPBCS_newVsOld_tosbcs.mp4

make_newVOldDiffs_donotstoredisplay_delFudge1_bg1.txt

@durack1
Copy link
Member Author

durack1 commented Jul 11, 2016

@doutriaux1 @danlipsa @chaosphere2112 just pinging you folks here.. Wishful thinking for 2.6.1??

@durack1 durack1 modified the milestones: 2.12, 2.6 Aug 23, 2017
@durack1 durack1 removed the 1 - Ready label Aug 23, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants