Initiating Nautobot jobs, next-2.0 rebase #270

MeganerdDev · 2023-09-26T06:23:18Z

Closes: #223

#224 closed in favor of this rebase

What's Changed

Adds the following nautobot subcommands:

init_job() as init-job
- Adds init_job subccomand which initiates a Nautobot job by job name
- Update init_jobs for new pattern, tested and working
- Adds kwargs JSON-string dictionary to init_jobs (untested but looks good)
get_jobs() as get-jobs
- Adds get_jobs which returns all Nautobot jobs viewable to user
- Adds kwargs JSON-string list to get_jobs export header items (tested as working)
filter_jobs() as filter-jobs
- Adds filter_jobs which returns filtered set of Nautobot jobs viewable to user
  - eg: /nautobot filter-jobs "enabled" or /nautobot filter-jobs "enabled,installed"
    - this could be changed to JSON-string dictionary/list? eg: /nautobot filter-jobs '{"enabled": true}'

Remaining work

The render markdown for init_job should presen thet job form items
- With init_job method we can pass kwargs as JSON-string (untested)
Add to documentation

TODO

Explanation of Change(s)
Added change log fragment(s) (for more information see the documentation)
Attached Screenshots, Payload Example
Unit, Integration Tests
Documentation Updates (when adding/changing features)
Outline Remaining Work, Constraints from Design

Screenshots

/nautobot init-job

/nautobot init-jobs for job that is not enabled

if there is a failed job, we will return the job_result.result message

/nautobot get-jobs

by default we will get some head items

/nautobot get-jobs '["id", "name", "installed", "tags"]'

we can specify the header items we want with a JSON formatted list

if there is a bad key, eg: /nautobot get-jobs '["id", "name", "invalid key demo"]'

/nautobot filter-jobs "enabled"

Notes

I was having some GIT troubles rebasing to next-2.0 the exact way I wanted, which is why I opened this PR. The other PR may serve use for any 1.x hold-overs by using the static service account?
Thanks for the hard work on the user permission mapping @smk4664 , the feature looks solid

Update init_jobs for new pattern, tested and working Add kwargs to init_jobs, untested but looks good Add get_jobs which returns all Nautobot jobs viewable to user Add filter_jobs which returns filtered set of Nautobot jobs viewable to user

…loses nautobot#223 Run black formatter

…be exported for jobs.

Update markdown to URL

MeganerdDev · 2023-09-26T07:34:24Z

Got some edge cases left I think, but pretty much everything but making CI happy is left

MeganerdDev · 2023-09-26T07:49:38Z

I left a few TODO entries, but nothing is a blocker I think

MeganerdDev · 2023-09-26T08:17:54Z

Not sure if making pylint happy for generator is mandatory for merge or if we can add an ignore for this case
# pylint: disable=use-a-generator

smk4664 · 2023-09-26T13:36:57Z

nautobot_chatops/workers/nautobot.py

+        profile = True
+
+    # Get instance of the user who will run the job
+    user = get_user_model()


You do not need to get the user model, or get the user, as dispatcher.user is already a user instance.

thanks, pushed d7e4ccc

smk4664 · 2023-09-26T13:42:43Z

nautobot_chatops/workers/nautobot.py

+        return (CommandStatusChoices.STATUS_FAILED, f'Job "{job_name}" failed to initiate. Result: {job_result.result}')
+
+    # TODO: need base-domain, this yields: /extras/job-results/<job_id>/
+    job_url = job_result.get_absolute_url()


We can use dispatcher.context['request_schema'] and dispatcher.context['request_host'] to get the base-domain. But this won't work in Slack Socket Mode. I added the SLACK_SOCKET_STATIC_HOST to solve that issue for static urls, but since static could point to S3, we would need another solution here.

noted, will explore this

smk4664 · 2023-09-26T13:45:41Z

nautobot_chatops/workers/nautobot.py

+
+    job_class_path = job_model.class_path
+
+    # TODO: Check if json_args keys are valid for this job model


It might also be a good idea to ensure the user has permissions to run the job.

does the restrict() a few lines up do this? I can test to confirm

job_model = Job.objects.restrict(dispatcher.user, "view").get(name=job_name)```

MeganerdDev · 2023-09-26T19:34:19Z

Thanks for the feedback, Ill get this updated!

MeganerdDev · 2023-09-29T18:59:38Z

My current action/to-do items for this PR:

have a job's user-input widgets prompt for chat user input
job to provide link to job (base url + path)
testing to confirm features working as expected
documentation tasks

MeganerdDev · 2023-10-12T06:49:44Z

status update on TODO item -> "have a job's user-input widgets prompt for chat user input"
I have a working concept, will update PR with that soon!

Bug: job forms with only 1-item

MeganerdDev · 2023-10-31T04:40:10Z

This now provides a separate nautobot subcommand init-job-form

When /nautobot init-job-form is called without a job name it presents a drop-down dialog
Otherwise you can specify the job name, eg: /nautobot init-job-form "demo-job"

Caveats:

Currently there is a blocking issues I am stuck on for this. Whenever I initiate the job it goes from Running then Pending status, although the job is actually running.
init-job-form could be merged to a single subcommand with init-job, have the input form call itself with the additional args

prompt_for_job added to init-job subcommand removed testing code blocks todo: bug: single job forms wont submit properly with multi_input_dialog notes: merge init-job and init-job-form seems best

…attern Added wait for job to initiate using refresh_from_db Corrected the initiated job url hyperlink message

MeganerdDev · 2024-02-06T00:46:52Z

I removed this helper function, noting in case we want to re-add (eg: enable_job subcommand would use this)

def enable_job(module, name, source="local"):
    """
    Helper function to look up a job by class and job model and enable it.

    Args:
        module (str): Job module name
        name (str): Job class name
        source (str): Job grouping (default: "local")
    """
    job_class = get_job(f"{module}.{name}")
    job_model = Job.objects.get(module_name=module, job_class_name=name) # .restrict() may be preferred if we pass user
    if not job_model.enabled:
        job_model.enabled = True
        job_model.validated_save()
    return

MeganerdDev · 2024-02-06T01:26:52Z

Noting some things for the current status of this PR:

I explored the idea of pagination of the job's drop-down item. It seemed to me that widget in each dispatcher already supported a large enough volume for these items
Combining init_job and init_job_form subcommands? Not required for PR, but may be the better pattern
Add an enable_job subcommand using the helper above? Not required for PR, can be its own PR
Numeric input mask for numeric only text input widgets on client-side? Unexplored
Some remaining TODO comments can be removed from the PR

Fixed items:

The correct Job URL hyperlink is now properly presented to user in chat response
The job is properly initiating, corrected by enqueue_job and refresh_from_db

Follow existing pylint ignore pattern on repo

MeganerdDev · 2024-02-06T02:02:06Z

nautobot_chatops/workers/nautobot.py

+    )
+
+    # Wait on the job to finish
+    while job_result.status not in JobResultStatusChoices.READY_STATES:


We may want something to drop out of this loop after enough time?

We may want something to drop out of this loop after enough time?

Addressed in 7470ec8

…abase

jvanderaa · 2024-02-06T03:19:48Z

nautobot_chatops/workers/nautobot.py

+
+
+@subcommand_of("nautobot")
+def init_job(dispatcher, *args, job_name: str = "", json_string_kwargs: str = ""):  # pylint: disable=too-many-locals


Would this make more sense to be named start_job or maybe launch_job? And then as I put that into suggestion, I then looked at Nautobot itself. Probably run_job would make sense to line up with the Run Job button.

run_job should be fine for the namespace. Initially I did not use run_job namespace since there was a similar import nautobot.extras.jobs import run_job, but this has been replaced by enqueue_job anyways in the PR

Rename `init_job_form`subcommand to `run_job_form`

smk4664 · 2024-02-06T23:30:36Z

nautobot_chatops/workers/nautobot.py

+
+    if job_result and job_result.status == "FAILURE":
+        dispatcher.send_error(f"The requested job {job_name} failed to initiate. Result: {job_result.result}")
+        return (CommandStatusChoices.STATUS_FAILED, f'Job "{job_name}" failed to initiate. Result: {job_result.result}')


This is checking if the task failed, but then sending a message that the task failed to initiate. The task was initiated but failed.

Good point, I adjusted the phrasing in 875ca64, but let me know if you have anything specific in mind maybe

qduk · 2024-03-01T00:46:05Z

Awesome work @MeganerdDev! The only thing I'd note is the addition of some documentation in the /nautobot commands table.

meganerd added 2 commits September 25, 2023 20:04

copy init-jobs changes fom nautobot#224 with rebase to next-2.0 branch

c17166c

MeganerdDev requested review from glennmatthews, jvanderaa, smk4664 and whitej6 as code owners September 26, 2023 06:23

Correct change fragment, nautobot#270 will replace nautobot#224, and c…

e01dbe6

…loses nautobot#223 Run black formatter

MeganerdDev mentioned this pull request Sep 26, 2023

Initiating Nautobot jobs #224

Closed

6 tasks

meganerd added 3 commits September 26, 2023 03:05

Add kwargs for get_jobs() that allows user-specified header items to …

99e4512

…be exported for jobs.

Minor update

0d623ef

Catch job_result.result == "FAILURE"

0f28144

Update markdown to URL

meganerd added 3 commits September 26, 2023 03:35

run black

9d99ca7

pydocstyle, flake8 fixes

4d0d877

black formatter

fe07fc3

meganerd added 2 commits September 26, 2023 03:53

pylint

1c8da1a

remove runnable references

3fc453a

smk4664 reviewed Sep 26, 2023

View reviewed changes

meganerd and others added 3 commits September 27, 2023 00:46

reuse existing user instance (dispatch.user)

d7e4ccc

typo correction for dispatcher.user

1423b8c

Replace list comp with generator

50725bb

meganerd added 2 commits October 30, 2023 23:17

Add init-job-form nautobot subcommand

42e1931

Bug: job forms with only 1-item

flake8 fix, catch explicit exception

f074f92

Remove block response, leaving testing commented

6910196

meganerd and others added 6 commits October 31, 2023 00:54

pylint8 fixes

98c2db3

prompt_for_job added to init-job subcommand removed testing code blocks todo: bug: single job forms wont submit properly with multi_input_dialog notes: merge init-job and init-job-form seems best

CI history expired, re-running checks

9a259a5

pylint CI fixes: W1113, R0914, C0301

8dd56fe

Black code formatter

195c888

pylint ignore too-many-locals for init_job

28c1d95

Replace execute_job with enqueue_job, which is correct Nautobot 2.x p…

cb26441

…attern Added wait for job to initiate using refresh_from_db Corrected the initiated job url hyperlink message

Fix missing variable

64ed936

Update breadcrumb changes

00eb275

Follow existing pylint ignore pattern on repo

MeganerdDev commented Feb 6, 2024

View reviewed changes

Add max iterations for waiting on job to enter ready state in the dat…

7470ec8

…abase

jvanderaa reviewed Feb 6, 2024

View reviewed changes

Rename init_job subcommand to run_job

0ab7e78

Rename `init_job_form`subcommand to `run_job_form`

jvanderaa approved these changes Feb 6, 2024

View reviewed changes

smk4664 reviewed Feb 6, 2024

View reviewed changes

meganerddev added 2 commits February 7, 2024 02:22

Update error for initiated job status failure state

875ca64

Rub Black formatter

e6c14c6

smk4664 merged commit eb31930 into nautobot:next-2.0 Mar 13, 2024
14 checks passed

smk4664 mentioned this pull request Mar 28, 2024

v3.0.2 #310

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initiating Nautobot jobs, next-2.0 rebase #270

Initiating Nautobot jobs, next-2.0 rebase #270

MeganerdDev commented Sep 26, 2023 •

edited

Loading

MeganerdDev commented Sep 26, 2023

MeganerdDev commented Sep 26, 2023

MeganerdDev commented Sep 26, 2023

smk4664 Sep 26, 2023

MeganerdDev Sep 27, 2023

smk4664 Sep 26, 2023

MeganerdDev Sep 27, 2023

smk4664 Sep 26, 2023

MeganerdDev Sep 27, 2023

MeganerdDev commented Sep 26, 2023

MeganerdDev commented Sep 29, 2023 •

edited

Loading

MeganerdDev commented Oct 12, 2023

MeganerdDev commented Oct 31, 2023

MeganerdDev commented Feb 6, 2024

MeganerdDev commented Feb 6, 2024

MeganerdDev Feb 6, 2024

MeganerdDev Feb 6, 2024

jvanderaa Feb 6, 2024

MeganerdDev Feb 6, 2024

smk4664 Feb 6, 2024

MeganerdDev Feb 7, 2024 •

edited

Loading

qduk commented Mar 1, 2024 •

edited

Loading


		job_class_path = job_model.class_path

		# TODO: Check if json_args keys are valid for this job model



		@subcommand_of("nautobot")
		def init_job(dispatcher, *args, job_name: str = "", json_string_kwargs: str = ""): # pylint: disable=too-many-locals

Initiating Nautobot jobs, next-2.0 rebase #270

Initiating Nautobot jobs, next-2.0 rebase #270

Conversation

MeganerdDev commented Sep 26, 2023 • edited Loading

Closes: #223

#224 closed in favor of this rebase

What's Changed

Remaining work

TODO

Screenshots

/nautobot init-job

/nautobot init-jobs for job that is not enabled

/nautobot get-jobs

/nautobot get-jobs '["id", "name", "installed", "tags"]'

/nautobot filter-jobs "enabled"

Notes

MeganerdDev commented Sep 26, 2023

MeganerdDev commented Sep 26, 2023

MeganerdDev commented Sep 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MeganerdDev commented Sep 26, 2023

MeganerdDev commented Sep 29, 2023 • edited Loading

MeganerdDev commented Oct 12, 2023

MeganerdDev commented Oct 31, 2023

MeganerdDev commented Feb 6, 2024

MeganerdDev commented Feb 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MeganerdDev Feb 7, 2024 • edited Loading

Choose a reason for hiding this comment

qduk commented Mar 1, 2024 • edited Loading

MeganerdDev commented Sep 26, 2023 •

edited

Loading

MeganerdDev commented Sep 29, 2023 •

edited

Loading

MeganerdDev Feb 7, 2024 •

edited

Loading

qduk commented Mar 1, 2024 •

edited

Loading