v3.00.57rc1 - closer to fine

Summary

"The less I seek my source for some definitive, the closer I am to fine." -- Indigo Girls.

The last version (3.0.56) was the first one where we thought sr3 could replace any
v2 configuration. As we did so, a few new regressions were found in some senders.
This release addresses all of those. We are close to there being none left.

There are also improvements in this release:

• #1350 Add “down” and “disconnected” run states to sr3 status
• #1323 DESTFN=NONE::TIME now works.
• #1299 after_gather entry point added.
• #1289 accept statements now provide arguments to plugins
• change in post_exchangeSplit… now based on relPath, not checksum. (work on HPC.)
• fix #1241 sr3 status more compact.
• fix #1206 exclude retries from lag calculation.
• #1301 sr3 status display fractional message rates when (< 1 / second.)
• user mode systemd unit file now tested and working.
• The https://dd.weather.gc.ca web site was revised. Example subscriptions revised to match.
• #1346 warn that queue parameters should be earlier in configs now.

Sender regressions compared to v2 that are fixed:

• #1337 mirror per umask, not global
• #1340 rename not working.
• #1327 inflight NONE in sender causes messages to be retried.
• #1322 sr3 sender fails to post message when no identity field in message.

As well as many other bug fixes.

Full Changelog: v3.00.56...v3.00.57rc1

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.57rc1/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.56 Recursion is Back Baby, Subtopics After Queues

What's new

• CHANGE: queueName and/or queueShare must be above subtopic in configuration files. (order did not matter in earlier versions)
• CHANGE: in API breakage: replace sarracenia.credentials with sarracenia.config.credentials
• New: standby mode, triggered by attempts 0 in the configuration file, allows clean local queueing of transfers during long outages.
• New: recursion option for polls and watches (actually it was removed before, but added back.)
• New: logDuplicates option to show only files rejected by the duplicate suppression mechanisms.
• rest is fixes of many things. It just behaves better.
  • http download performance regression fixed.
  • ack failure recovery regression fixed.
  • HPC mirroring rename/symlink fix (requires metpx-sr3c 3.24.11 )

What's Changed

• Issue1199 - directory / not interpreted properly. by @petersilva in #1231
• just reducing bogus messages by @petersilva in #1230
• Unquote secret_access_key in s3CloudSender plugin by @reidsunderland in #1228
• sr3 sanity behaves badly with instances > 100. by @petersilva in #1226
• ANOTHER AM U18.04 adaptation + another AM sender problem found in satnet testings by @andreleblanc11 in #1221
• fix crash on sr3c pidfiles (i01.pid) by @reidsunderland in #1234
• Issue35 try 2 stateful binding part1 by @petersilva in #1232
• Switch from global QoS to per-consumer QoS by @reidsunderland in #1235
• Doc sec zoning multihop by @petersilva in #1236
• Fix bug from moving credentials into config subdir by @reidsunderland in #1239
• fix #1242 sftp making subdirs when it shouldn't by @reidsunderland in #1243
• Fix for #1156 by @mshak2 in #1193
• QoS should apply to all instances by @reidsunderland in #1245
• add ncftp client to basic vm config... might fix some flow tests. by @petersilva in #1249
• Json logging (EXPERIMENTAL) by @petersilva in #1238
• Add recursion back by @petersilva in #1237
• fix #1246 delete source when no options given by @petersilva in #1250
• Fixing unit tests by @petersilva in #1251
• testing: add pyftpdlib to apt packages to try avoiding version incompatibility by @reidsunderland in #1252
• suppress error message about missing subscriptions.json by @petersilva in #1256
• Update paho-mqtt package from pip by @reidsunderland in #1260
• Improve Redis Flow Tests on GitHub by @reidsunderland in #1259
• Refactor the interruptible_sleep test by @reidsunderland in #1257
• more information about doing post releases by @petersilva in #1253
• tweaks to flowcb/gather/file.py after experiments with noticing large files. by @petersilva in #1258
• Add cpuS and wVip to status list - CommandLineGuide.rst by @andreleblanc11 in #1263
• Fix HTTP performance regression by @reidsunderland in #1265
• fix date check in messages. by @petersilva in #1264
• fix #1255 remove error message with undeclared value in it by @petersilva in #1268
• Issue1271 - Change raw2bulletin renamer to an after_accept entry point by @andreleblanc11 in #1273
• Issue1270 - fixing relative symlinks with (inappropriate) ../../ in them... by @petersilva in #1272
• Issue1267 queueName not working as expected because of upcoming feature by @petersilva in #1276
• close #1278 we learned that acks should never be retried by @petersilva in #1281
• Fix #1266 - Add a check for any disabled configs when using sr3 start by @mshak2 in #1282
• Issue1271 - Fix accomodation to sarra component by @andreleblanc11 in #1279
• for #1275 standby mode can be implemented with attempts=0 by @petersilva in #1280

Full Changelog: v3.00.55...v3.00.56

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.56/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.56rc3 back to recursion, subtopics after queues

What's Changed

• close #1278 we learned that acks should never be retried by @petersilva in #1281
• Fix #1266 - Add a check for any disabled configs when using sr3 start by @mshak2 in #1282
• Issue1271 - Fix accomodation to sarra component by @andreleblanc11 in #1279
• for #1275 standby mode can be implemented with attempts=0 by @petersilva in #1280

Full Changelog: v3.00.56rc2...v3.00.56rc3

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.56rc3/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.56rc2 back to recursion, subtopics after queues

Regression

• known stability regression, when ack fails subscriber must be restarted #1278 (work-around: use v3.00.54.post1 )
  (stable version will not be released without a fix.)

What's New

• need to put queueName settings before the subtopic settings in configuration files, for this and all future versions.
• fixed #1261 performance regression for https download
• HPC rename/symlink mirroring fix: needs to be combined with an upcoming C (metpx-sr3c) package (relative symbolic links and renames when done from a directory that is symlinked to the tree being mirrored.)

What's Changed

• suppress error message about missing subscriptions.json by @petersilva in #1256
• Update paho-mqtt package from pip by @reidsunderland in #1260
• Improve Redis Flow Tests on GitHub by @reidsunderland in #1259
• Refactor the interruptible_sleep test by @reidsunderland in #1257
• more information about doing post releases by @petersilva in #1253
• tweaks to flowcb/gather/file.py after experiments with noticing large files. by @petersilva in #1258
• Add cpuS and wVip to status list - CommandLineGuide.rst by @andreleblanc11 in #1263
• Fix HTTP performance regression by @reidsunderland in #1265
• fix date check in messages. by @petersilva in #1264
• fix #1255 remove error message with undeclared value in it by @petersilva in #1268
• Issue1271 - Change raw2bulletin renamer to an after_accept entry point by @andreleblanc11 in #1273
• Issue1270 - fixing relative symlinks with (inappropriate) ../../ in them... by @petersilva in #1272
• Issue1267 queueName not working as expected because of upcoming feature by @petersilva in #1276

Full Changelog: v3.00.56rc1...v3.00.56rc2

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.56rc2/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.56rc1 back to recursion

REGRESSION

• known performance regression on http downloads #1261 (work-around: use v3.00.54post1 )
• known stability regression, when ack fails subscriber must be restarted #1278 (work-around: use v3.00.54.post1 )
  (stable version will not be released without fix.)

What's New

• New: recursion option for polls and watches (actually it was removed before, but added back.)
• New: logDuplicates option to show only files rejected by the duplicate suppression mechanisms.
• rest is fixes of many things. It just behaves better.

What's Changed

• Issue1199 - directory / not interpreted properly. by @petersilva in #1231
• just reducing bogus messages by @petersilva in #1230
• Unquote secret_access_key in s3CloudSender plugin by @reidsunderland in #1228
• sr3 sanity behaves badly with instances > 100. by @petersilva in #1226
• ANOTHER AM U18.04 adaptation + another AM sender problem found in satnet testings by @andreleblanc11 in #1221
• fix crash on sr3c pidfiles (i01.pid) by @reidsunderland in #1234
• Issue35 try 2 stateful binding part1 by @petersilva in #1232
• Switch from global QoS to per-consumer QoS by @reidsunderland in #1235
• Doc sec zoning multihop by @petersilva in #1236
• Fix bug from moving credentials into config subdir by @reidsunderland in #1239
• fix #1242 sftp making subdirs when it shouldn't by @reidsunderland in #1243
• Fix for #1156 by @mshak2 in #1193
• QoS should apply to all instances by @reidsunderland in #1245
• add ncftp client to basic vm config... might fix some flow tests. by @petersilva in #1249
• Json logging (EXPERIMENTAL) by @petersilva in #1238
• Add recursion back by @petersilva in #1237
• fix #1246 delete source when no options given by @petersilva in #1250
• Fixing unit tests by @petersilva in #1251
• testing: add pyftpdlib to apt packages to try avoiding version incompatibility by @reidsunderland in #1252

Full Changelog: v3.00.55...v3.00.56rc1

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.56rc1/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.55post1

REGRESSION

• known performance regression on http downloads #1261 (work-around: use v3.00.56 )
• known stability regression, when ack fails subscriber must be restarted #1278 (work-around: use v3.00.56 )

What's Changed

• fix #1242 regression.

Full Changelog: v3.00.55...o3.00.55post1

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.55.post1/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.55 Good Timing

REGRESSION

• known performance regression on http downloads #1261 (work-around: use v3.00.56 )
• known stability regression, when ack fails subscriber must be restarted #1278 (work-around: use v3.00.56 )

Briefly:

Working through deployment issues on core operational data pumps:

• fixing more timers expiring during transfers. many reliability improvements.
• many more polls deployed, and some improvements for them.
• Add scheduled_time option for scheduled flows to run at the given time.
• management UI improvements (sr3 status display of lag, queue sr3 stop works better.)
• fix for citypage download issues (and other datasets that rapidly change upstream.)
• scheduled flow is simplified and more accurate.
• fix #1219 time offsets in configuration files now correctly converted

What's Changed

• Add new BC Trans API to sr3 by @andreleblanc11 in #1150
• Fix #1151 by @andreleblanc11 in #1152
• Fix #953 re-enable and fix passive queue declare by @reidsunderland in #1160
• AM adaptation for U18.04 servers by @andreleblanc11 in #1164
• The Hearty eXpert Issue1138 - documenting messageAgeMax by @petersilva in #1147
• Issue1159 by @petersilva in #1162
• Later pip needs convincing by @petersilva in #1171
• Issue1155 AMQP ack refactor by @reidsunderland in #1174
• #1167 fix clash in class names with a directory prefix. inelegant, bu… by @petersilva in #1172
• Fix #1173: problems stopping instances by @reidsunderland in #1178
• fix #1166 improved download decision criteria by @petersilva in #1175
• fix Ubuntu 20.04 flow tests by @reidsunderland in #1186
• Implement HTTP(S) stat using urllib, and HTTPS transfer class minor refactor by @reidsunderland in #1202
• Change stop output to show the number of processes still running (like v2) by @reidsunderland in #1204
• fix rejectPercent maybe other stuff? by @petersilva in #1192
• maybe fix #1195 more log fields by default by @petersilva in #1197
• Fix badly expiring timers by @petersilva in #1208
• Add scheduled_time for #1209 by @andreleblanc11 in #1210
• Issue1198 - charset option for ftp. by @petersilva in #1200
• Add more wiggle room in scheduled_flow for accumulated wait overhead by @andreleblanc11 in #1217
• Fix for Issue1218 - Permission problems in directory creations on FTP/SFTP transfers by @andreleblanc11 in #1220
• Issue1214 another scheduled refactor by @petersilva in #1223
• fix #1219 enough to convert ops by @petersilva in #1224

Full Changelog: v3.00.54...v3.00.55

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.55/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.55rc2

What's Changed

• Issue1214 another scheduled refactor by @reidsunderland in #1223
• fix #1219 enough to convert ops by @petersilva in #1224

Full Changelog: v3.00.55rc1...v3.00.55rc2

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.55rc2/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.55rc1

Briefly:

• Working through deployment issues on core operational data pumps.
• fixing more timers expiring during transfers. many reliability improvements.
• many more polls deployed, and some improvements for them.
• management UI improvements (sr3 status display of lag, queue sr3 stop works better.)
• fix for citypage download issues (and other datasets that rapidly change upstream.)

What's Changed

• Add new BC Trans API to sr3 by @andreleblanc11 in #1150
• Fix #1151 by @andreleblanc11 in #1152
• Fix #953 re-enable and fix passive queue declare by @reidsunderland in #1160
• AM adaptation for U18.04 servers by @andreleblanc11 in #1164
• The Hearty eXpert Issue1138 - documenting messageAgeMax by @petersilva in #1147
• Issue1159 by @petersilva in #1162
• Later pip needs convincing by @petersilva in #1171
• Issue1155 AMQP ack refactor by @reidsunderland in #1174
• #1167 fix clash in class names with a directory prefix. inelegant, bu… by @petersilva in #1172
• Fix #1173: problems stopping instances by @reidsunderland in #1178
• fix #1166 improved download decision criteria by @petersilva in #1175
• fix Ubuntu 20.04 flow tests by @reidsunderland in #1186
• Implement HTTP(S) stat using urllib, and HTTPS transfer class minor refactor by @reidsunderland in #1202
• Change stop output to show the number of processes still running (like v2) by @reidsunderland in #1204
• fix rejectPercent maybe other stuff? by @petersilva in #1192
• maybe fix #1195 more log fields by default by @petersilva in #1197
• Fix badly expiring timers by @petersilva in #1208
• Add scheduled_time for #1209 by @andreleblanc11 in #1210
• Issue1198 - charset option for ftp. by @petersilva in #1200
• Add more wiggle room in scheduled_flow for accumulated wait overhead by @andreleblanc11 in #1217
• Fix for Issue1218 - Permission problems in directory creations on FTP/SFTP transfers by @andreleblanc11 in #1220

Full Changelog: v3.00.54rc3...v3.00.55rc1

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.55rc1/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v2.24.08post2

• Version 2 is now in maintenance mode. Only install if you already have a version 2 configuration to maintain.
• There is nothing in v2 that is not working as well or better in sr3.
• It is recommended that users of v2 upgrade their configurations and begin transition to sr3. (although v2 will remain available for the foreseeable future.)
• Sarracenia version 3 is recommended for new installations and is a super-set of version 2.
• known issues with v2 that we do not plan to address ( https://github.com/MetPX/sarracenia/issues?q=is%3Aopen+is%3Aissue+label%3Av2only+label%3Awontfix )
  • does not support MQTT.

This release includes cleanups of issues encountered upgrading main data pumps with legacy v2 stream. This release has good compatibility with sr3 as well as older v2 configurations, and crashes less that older v2 versions.

What's Changed

• revert ftp charset change.
• V2 fix the problems found during v2 upgrade of main pump. by @petersilva in #1182

Full Changelog: v2.24.08...v2.24.08post2

Installation

Guide: https://github.com/MetPX/sarracenia/blob/v2_stable/doc/Install.rst

packages available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
python: https://pypi.org/project/metpx-sarracenia/