1.7.4 (#274)

* locked maestro version

* incremented to 1.5.3

* Update CHANGELOG.md

* Feature/koning/vhost no leading slash (#217)

* The broker name can now be amqps (with ssl) or amqp (without ssl).
The default rabbitmq vhost is now <user> instead of /<user>.

* Undo fix typos.

* Fix spacing.

* Add amqps conn for self doc.

* Add inital running workers check to monitor. This should eliminate any (#218)

race conditions.

* updated timeout (#223)

* updated default task timeout

* Added to CHANGELOG

* Feature/koning/create encrypt key init (#219)

* Add init_key to create the encryption key when merlin config is run.

* Move import to point of use for encryption.

* Feature/koning/monitor upgrade (#221)

* Fix sleep variable.

* Change another sleep variable.

* Change brackets.

* Fix clarity for jobs but no consumers condition.

* Fix comment.

* Update comment.

* Move check_merlin_status to router.

* Run fix-style.

* Bugfix/ben/expand name (#220)

* working on bugfix

* achieved desired behavior

* made expansion anf filewriting conditional

* fixed style

* updated CHANGELOG

* fixed samples and restart

* fixed style

* used variable

* added name expansion test

* temporarily commented out one test

* fixed style

* added debug block

* changes

* corrected bug

* updated example; scripts are now copied into merlin_info

* Feature/ben/expose visibility timeout (#224)

* exposed visibility timeout in config file

* added visibility timeouts to default config files

* fixed style

* added seconds to clarify name

* fixed attribute name

* locked celery version at 4.4.2

* updated CHANGELOG

* v1.5.4

* adjusted CHANGELOG

* adjusted CHANGELOG

* Feature/koning/info conn timeout (#226)

* Add a timeout check for the merlin info kombu connection test. The redis
server will not comply with the connection_timeout config.

* Update docs.

* Load default timeout if non in config file (#229)

* 1.6.1

* bugfix (#232)

* bugfix

* removed repetition in logic

* Log sample generation to merlin_info (#234)

* sample generation is now logged to merlin_info

* Added to CHANGELOG

* fixed style

* Maestro 1.1.8+ update (#233)

* working on adding maestro's schema validation to merlin

* updated schemas

* adding merlin_schema to spec logic

* added debug prints

* added print

* fixing vestigial Maestro calls

* adjusted merlin_schemas

* fixed json file

* validation progress

* added from_dict replacement logic

* adjusted maestrowf version req

* fixed string

* schema improvements

* merlinsection schema updates

* merlin section error parsing appears to work

* updated CHANGELOG

* fixed merge error

* Added walltime to json schema

* Add the bank and walltime keywords to the batch slurm launch, these (#236)

will not alter the lsf launch.

* Bugfix for new celery versions (#231)

* Change the expand_tasks_with_samples task to use an immutable signature,
no return code is neccessary.

* Change only expand_tasks_with_samples to si()

* Increase celery version requirement to the latest (4.4.5).

* Update CHANGELOG.

Co-authored-by: Benjamin Bay <[email protected]>

* hidden test specs (#237)

* added tests to run_tests.py

* updated merlinsection schema

* added full_spec test

* added to CHANGELOG

* merlinspec change

* use regex to find variable tokens (#239)

* now using regex to find variable tokens

* added shell ref function

* added to CHANGELOG

* Revert maestro 1.1.8 support (#241)

* reverted support of Maestro 1.1.8

* removed tests that use maestro validation

* fixed style

* fixed cli tests, reactivated 3 old ones (#242)

* fixed cli tests, reactivated 3 old ones

* added to CHANGELOG

* More provenance specs + logic improvements (#240)

* added orig.yaml and expanded.yaml

* added partial provenance spec

* fixed style

* added to CHANGELOG

* removed comments

* made env replacement more precise

* fixed restart bug

* changed provenance file names

* logic update

* fixed style

* expansion now working

* added cli test logic to test all 3 provenance file types

* updated CHANGELOG

* expansion sans io (#243)

* reverted support of Maestro 1.1.8

* removed tests that use maestro validation

* fixed style

* added sections property to spec

* reducing number of file writes and reads

* provenance specs are correct

* fixed style

* removed unused imports

* cleaning up logic

* fixed style

* final logic improvements

* removed unused property

* updated CHANGELOG

* v1.6.2

* debugging study

* added provenance tests

* fixed path in test

* added check for workflow name expanding to invalid filename

* Docs/ben/level max dirs (#250)

* added to documentation

* added to CHANGELOG

* added default

* updated docs to reflect recent provenance spec update (#251)

* Feature/ben/pgen (#248)

* added basic pgen support

* added to CHANGELOG

* pgen, pargs, and env appear to work

* added to CHANGELOG

* fixed typo

* added pgen cli test

* added faq entry on pgen

* Update Dockerfile (#254)

* Update CHANGELOG.md

* pretty yaml (#252)

* starting

* building pretty_dump function

* updates

* fixed style

* tweaks

* valid yaml

* added yaml_sections

* added to CHANGELOG

* removed old yaml representer code

* added cli test for equality of provenance specs

* fixed pgen test

* disabled test while not working

* fixed yaml lists for source, paths, git

* tweaks / fixes

* env variable bugfix (#247)

* removed expansion of env variables in provenance specs

* added to CHANGELOG

* corrected unit tests

* added todos

* working on expand_env_var function

* fixed type bug

* removed comments, prints, fixed style

* added to CHANGELOG

* tweaks

* fixed output path

* allowed study name to use env vars

* fixed style

* added to CHANGELOG, added support for 'restart'

* Update CHANGELOG.md

* added docs strings

* added to docs

* updated comment

* fixed CHANGELOG

* removed debug print

* flux examples fix (#253)

* Update flux example workflows to use new kvs query methods.

* Update CHANGELOG with fix info.

* Add coker fix comment.

* Update CHANGELOG.md

* added gitlab ci file

* improved gitlab ci setup

* renamed file

* hotfix to allow distributed cli tests to run

* redis update (#255)

* Update redis to new version with TLS support.

* Add changelog.

* Bugfix for rediss cert_reqs keyword. (#256)

* expose celery (#245)

* added default app.yaml, still in progress...

* added celery defaults

* Update app.yaml

* config override appears to be working

* corrected value name

* shifted some properties to be optional defaults

* removed unneeded file

* improvements

* fixed style

* fixed style

* hid 'no override' message

* tweaked var names

* added 2 tests for celery app (#246)

* fixed test

* fixed workers bug

* fixed unit tests for running offline

* removed 2 unit tests

* fixed CHANGELOG

* added info print of all celery configurations being overridden

* 1.7.0

* 1.7.0

* copy samplesfile to merlin_info (#259)

* copy samplesfile to merlin_info

* added to CHANGELOG

* Bugfix/koning/process exceptions (#260)

* Add bugfix log.

* Add an exception catcher for the multiprocessing Process class.

* fixed style

* 1.7.1

* hotfix

* fix

* 1.7.2

* Update CHANGELOG.md

* Update CHANGELOG.md

* 1.7.3

* 1.7.3

* Refactor/ben/spec logic (#267)

* improved lots of logic

* added minimum format test

* added to CHANGELOG

* added to CHANGELOG

* added 3 cli tests

* added shortcut var

* sorted imports

* removed erroneous note from doc

* null spec updates (for paper) (#270)

* made samples generation actually null; now uses touch, once only

* improving samples

* updated job launcher

* modifications

* added echo

* added echo

* added times

* fixed slurm command logic

* script improvements to accomodate multiple node logs

* read output script updates

* read output script fixes

* updated times

* updated times

* script updates

* script changes

* launcher script improvement: added run id to queue names

* tweak

* added account logic for quartz and pascal

* path

* fixed bug that mixed queues at w=32 and w=64

* bugfix

* tweak

* added new launcher specs

* added 24 hour cap

* fixed sample vals

* script update

* launcher changes

* updated merlin monitor call

* updates

* fixed style

* Feature/koning/maestro remove submit message (#269)

* Add a MerlinStepRecord to remove Maestro message when tasks resubmit.

* Add the dependent objects for MerlinStepRecord.

* Add CHANGELOG entry for bugfix.

* Revert quick fix changes.

* fixed style

* preparing run

* script updates

* updated launcher script: automated node list vals

* removed predefined node lists

* Update CHANGELOG.md

* Update CHANGELOG.md

* Feature/ben/null chain (#272)

* added null chain scripts

* spec improvements

* removed sim spec, as null spec can now be customized to wait longer

* added to merlin monitor

* specified copied file name

* added time to null_chain

* max sample ramp

* added to CHANGELOG

* fixed style

* 1.7.4

Co-authored-by: Joe Koning <[email protected]>
Co-authored-by: Luc Peterson <[email protected]>
Co-authored-by: fixdocker <[email protected]>

CHANGELOG.md

@@ -4,6 +4,19 @@ All notable changes to Merlin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.7.4]
+
+### Fixed
+- Users will no longer see the message, "Cannot set the submission time of '<step name>'
+  because it has already been set", when tasks are restarted.
+- Bug causing `merlin restart` to break.
+
+### Changed
+- Improved internal logic beyond the crude fixes of the prior 2 patches.
+- Added a developer cli test for the minimum valid spec format.
+- Improvements to the `null_spec` example, used for measuring overhead in merlin. Includes
+  a new `null_chain` and removes the now-redundant `sim_spec`.
+
 ## [1.7.3]
 
 ### Fixed

Makefile

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

docs/source/merlin_specification.rst

@@ -85,16 +85,13 @@ see :doc:`./merlin_variables`.
     # Do not define a variable named "shell" here.
     #-------------------------------
     variables:
-      # set a custom output path for the study workspace. This path is where
+      # Set a custom output path for the study workspace. This path is where
       # Merlin will place all temporary files, state files, and any output.
       # The resulting path is usually a timestamped folder within OUTPUT_PATH
       # and in this case would be
       # './sample_output/merlin/merlin_sample1_<timestamp>'.
       # NOTE: If not specified,
       # OUTPUT_PATH is assumed to be the path where Merlin was launched from.
-      # NOTE: If the '-o' flag is specified for the run subcommand, OUTPUT_PATH
-      # the output path will be taken from there and will not generate a
-      # timestamped path.
       OUTPUT_PATH: ./sample_output/merlin # OUTPUT_PATH is a keyword
                                           # variable that Merlin looks for
                                           # to replace with the study

merlin/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -38,7 +38,7 @@
 import sys
 
 
-__version__ = "1.7.3"
+__version__ = "1.7.4"
 VERSION = __version__
 PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "")
 

merlin/ascii_art.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/celery.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/abstracts/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/abstracts/enums/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/openfilelist.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/opennpylib.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/sample_index.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/sample_index_factory.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/security/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/security/encrypt.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/security/encrypt_backend_traffic.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/tasks.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/util_sampling.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/config/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/config/broker.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/config/configfile.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/config/results_backend.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/data/celery/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/display.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/examples/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/examples/dev_workflows/full_format.yaml

@@ -1,6 +1,6 @@
 description:
     name: full_format
-    description: a working spec including every possible field
+    description: a correctly-formatted but non-functioning spec including every possible field
     extra_key: this should be allowed for metadata
 
 env:

merlin/examples/dev_workflows/minimum_format.yaml

@@ -0,0 +1,9 @@
+description:
+    name: minimum_format
+    description: a spec that should not break
+
+study:
+    - name: step1
+      description: step 1
+      run:
+         cmd: echo "test"

merlin/examples/dev_workflows/no_description.yaml

@@ -0,0 +1,5 @@
+study:
+    - name: step1
+      description: step 1
+      run:
+         cmd: echo "test"

merlin/examples/dev_workflows/no_steps.yaml

@@ -0,0 +1,5 @@
+description:
+    name: no_steps
+    description: a spec that should break because there are no steps
+
+study:

merlin/examples/dev_workflows/no_study.yaml

@@ -0,0 +1,3 @@
+description:
+    name: no_study
+    description: a spec without the required 'study' section

merlin/examples/examples.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/examples/generator.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.3.
+# This file is part of Merlin, Version: 1.7.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/examples/workflows/null_spec/null_chain.yaml

@@ -0,0 +1,64 @@
+description:
+    name: null_chain
+    description: |
+        Run N_SAMPLES steps of TIME seconds each at CONC concurrency.
+        May be used to measure overhead in merlin.
+        Iterates thru a chain of workflows.
+
+batch:
+    type: slurm
+
+env:
+    variables:
+        OUTPUT_PATH: ./
+        TIME: 1
+        N_SAMPLES: 1
+        MAX_SAMPLES: 100000
+        CONC: 1
+        RUN_ID: 0
+        N_WORK: 0
+        QUEUE: queue_s$(N_SAMPLES)_c$(N_WORK)_r$(RUN_ID)
+        WORKER: worker_s$(N_SAMPLES)_c$(N_WORK)_r$(RUN_ID)
+
+study:
+    - name: null_step
+      description: exit immediately
+      run:
+        cmd: |
+            #echo $(SAMPLE)
+            sleep $(TIME)
+            exit $(MERLIN_SUCCESS)
+        task_queue: $(QUEUE)
+
+    - name: verify
+      description: echo done
+      run:
+        cmd: echo "Done"
+        depends: [null_step_*]
+        task_queue: $(QUEUE)
+
+    - name: launch_next
+      description: launch this workflow with different args
+      run:
+        cmd: |
+            NEXT_SAMPLES=$(($(N_SAMPLES)*10))
+            if ((NEXT_SAMPLES <= $(MAX_SAMPLES))) ; then
+                echo "Starting next iteration"
+                cd $(MERLIN_WORKSPACE)/../
+                merlin run $(SPECROOT)/null_chain.yaml --vars N_SAMPLES=$NEXT_SAMPLES CONC=$(CONC) RUN_ID=$(RUN_ID) N_WORK=$(N_WORK) QUEUE=$(QUEUE) TIME=$(TIME) MAX_SAMPLES=$(MAX_SAMPLES)
+            else
+                echo "Chain complete"
+            fi
+        depends: [verify]
+        task_queue: $(QUEUE)
+
+merlin:
+    resources:
+        workers:
+            $(WORKER):
+                args: -O fair --prefetch-multiplier 1 -E -l info --concurrency $(CONC) --logfile=%%p.log
+    samples:
+        generate:
+            cmd: python3 $(SPECROOT)/scripts/make_samples.py --number $(N_SAMPLES) --filepath $(MERLIN_INFO)/samples_file.npy
+        file: $(MERLIN_INFO)/samples_file.npy
+        column_labels: [SAMPLE]