Closes #10851: New staging mechanism (#10890)

* WIP

* Convert checkout() context manager to a class

* Misc cleanup

* Drop unique constraint from Change model

* Extend staging tests

* Misc cleanup

* Incorporate M2M changes

* Don't cancel wipe out creation records when an object is deleted

* Rename Change to StagedChange

* Add documentation for change staging

docs/models/extras/branch.md

@@ -0,0 +1,13 @@
+# Branches
+
+A branch is a collection of related [staged changes](./stagedchange.md) that have been prepared for merging into the active database. A branch can be mered by executing its `commit()` method. Deleting a branch will delete all its related changes.
+
+## Fields
+
+### Name
+
+The branch's name.
+
+### User
+
+The user to which the branch belongs (optional).

docs/models/extras/stagedchange.md

@@ -0,0 +1,26 @@
+# Staged Changes
+
+A staged change represents the creation of a new object or the modification or deletion of an existing object to be performed at some future point. Each change must be assigned to a [branch](./branch.md).
+
+Changes can be applied individually via the `apply()` method, however it is recommended to apply changes in bulk using the parent branch's `commit()` method.
+
+## Fields
+
+!!! warning
+    Staged changes are not typically created or manipulated directly, but rather effected through the use of the [`checkout()`](../../plugins/development/staged-changes.md) context manager.
+
+### Branch
+
+The [branch](./branch.md) to which this change belongs.
+
+### Action
+
+The type of action this change represents: `create`, `update`, or `delete`.
+
+### Object
+
+A generic foreign key referencing the existing object to which this change applies.
+
+### Data
+
+JSON representation of the changes being made to the object (not applicable for deletions).

docs/plugins/development/staged-changes.md

@@ -0,0 +1,42 @@
+# Staged Changes
+
+!!! danger "Experimental Feature"
+    This feature is still under active development and considered experimental in nature. Its use in production is strongly discouraged at this time.
+
+!!! note
+    This feature was introduced in NetBox v3.4.
+
+NetBox provides a programmatic API to stage the creation, modification, and deletion of objects without actually committing those changes to the active database. This can be useful for performing a "dry run" of bulk operations, or preparing a set of changes for administrative approval, for example.
+
+To begin staging changes, first create a [branch](../../models/extras/branch.md):
+
+```python
+from extras.models import Branch
+
+branch1 = Branch.objects.create(name='branch1')
+```
+
+Then, activate the branch using the `checkout()` context manager and begin making your changes. This initiates a new database transaction.
+
+```python
+from extras.models import Branch
+from netbox.staging import checkout
+
+branch1 = Branch.objects.get(name='branch1')
+with checkout(branch1):
+    Site.objects.create(name='New Site', slug='new-site')
+    # ...
+```
+
+Upon exiting the context, the database transaction is automatically rolled back and your changes recorded as [staged changes](../../models/extras/stagedchange.md). Re-entering a branch will trigger a new database transaction and automatically apply any staged changes associated with the branch.
+
+To apply the changes within a branch, call the branch's `commit()` method:
+
+```python
+from extras.models import Branch
+
+branch1 = Branch.objects.get(name='branch1')
+branch1.commit()
+```
+
+Committing a branch is an all-or-none operation: Any exceptions will revert the entire set of changes. After successfully committing a branch, all its associated StagedChange objects are automatically deleted (however the branch itself will remain and can be reused).

mkdocs.yml

@@ -131,6 +131,7 @@ nav:
             - REST API: 'plugins/development/rest-api.md'
             - GraphQL API: 'plugins/development/graphql-api.md'
             - Background Tasks: 'plugins/development/background-tasks.md'
+            - Staged Changes: 'plugins/development/staged-changes.md'
             - Exceptions: 'plugins/development/exceptions.md'
             - Search: 'plugins/development/search.md'
     - Administration:
@@ -191,12 +192,14 @@ nav:
             - SiteGroup: 'models/dcim/sitegroup.md'
             - VirtualChassis: 'models/dcim/virtualchassis.md'
         - Extras:
+            - Branch: 'models/extras/branch.md'
             - ConfigContext: 'models/extras/configcontext.md'
             - CustomField: 'models/extras/customfield.md'
             - CustomLink: 'models/extras/customlink.md'
             - ExportTemplate: 'models/extras/exporttemplate.md'
             - ImageAttachment: 'models/extras/imageattachment.md'
             - JournalEntry: 'models/extras/journalentry.md'
+            - StagedChange: 'models/extras/stagedchange.md'
             - Tag: 'models/extras/tag.md'
             - Webhook: 'models/extras/webhook.md'
         - IPAM:

netbox/extras/choices.py

@@ -182,3 +182,20 @@ class WebhookHttpMethodChoices(ChoiceSet):
         (METHOD_PATCH, 'PATCH'),
         (METHOD_DELETE, 'DELETE'),
     )
+
+
+#
+# Staging
+#
+
+class ChangeActionChoices(ChoiceSet):
+
+    ACTION_CREATE = 'create'
+    ACTION_UPDATE = 'update'
+    ACTION_DELETE = 'delete'
+
+    CHOICES = (
+        (ACTION_CREATE, 'Create'),
+        (ACTION_UPDATE, 'Update'),
+        (ACTION_DELETE, 'Delete'),
+    )

netbox/extras/migrations/0084_staging.py

@@ -0,0 +1,45 @@
+from django.conf import settings
+from django.db import migrations, models
+import django.db.models.deletion
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('contenttypes', '0002_remove_content_type_name'),
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+        ('extras', '0083_savedfilter'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='Branch',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False)),
+                ('created', models.DateTimeField(auto_now_add=True, null=True)),
+                ('last_updated', models.DateTimeField(auto_now=True, null=True)),
+                ('name', models.CharField(max_length=100, unique=True)),
+                ('description', models.CharField(blank=True, max_length=200)),
+                ('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, to=settings.AUTH_USER_MODEL)),
+            ],
+            options={
+                'ordering': ('name',),
+            },
+        ),
+        migrations.CreateModel(
+            name='StagedChange',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False)),
+                ('created', models.DateTimeField(auto_now_add=True, null=True)),
+                ('last_updated', models.DateTimeField(auto_now=True, null=True)),
+                ('action', models.CharField(max_length=20)),
+                ('object_id', models.PositiveBigIntegerField(blank=True, null=True)),
+                ('data', models.JSONField(blank=True, null=True)),
+                ('branch', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='staged_changes', to='extras.branch')),
+                ('object_type', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='+', to='contenttypes.contenttype')),
+            ],
+            options={
+                'ordering': ('pk',),
+            },
+        ),
+    ]

netbox/extras/models/__init__.py

@@ -3,9 +3,11 @@
 from .customfields import CustomField
 from .models import *
 from .search import *
+from .staging import *
 from .tags import Tag, TaggedItem
 
 __all__ = (
+    'Branch',
     'CachedValue',
     'ConfigContext',
     'ConfigContextModel',
@@ -20,6 +22,7 @@
     'Report',
     'SavedFilter',
     'Script',
+    'StagedChange',
     'Tag',
     'TaggedItem',
     'Webhook',

netbox/extras/models/staging.py

@@ -0,0 +1,114 @@
+import logging
+
+from django.contrib.auth import get_user_model
+from django.contrib.contenttypes.fields import GenericForeignKey
+from django.contrib.contenttypes.models import ContentType
+from django.db import models, transaction
+
+from extras.choices import ChangeActionChoices
+from netbox.models import ChangeLoggedModel
+from utilities.utils import deserialize_object
+
+__all__ = (
+    'Branch',
+    'StagedChange',
+)
+
+logger = logging.getLogger('netbox.staging')
+
+
+class Branch(ChangeLoggedModel):
+    """
+    A collection of related StagedChanges.
+    """
+    name = models.CharField(
+        max_length=100,
+        unique=True
+    )
+    description = models.CharField(
+        max_length=200,
+        blank=True
+    )
+    user = models.ForeignKey(
+        to=get_user_model(),
+        on_delete=models.SET_NULL,
+        blank=True,
+        null=True
+    )
+
+    class Meta:
+        ordering = ('name',)
+
+    def __str__(self):
+        return f'{self.name} ({self.pk})'
+
+    def merge(self):
+        logger.info(f'Merging changes in branch {self}')
+        with transaction.atomic():
+            for change in self.staged_changes.all():
+                change.apply()
+        self.staged_changes.all().delete()
+
+
+class StagedChange(ChangeLoggedModel):
+    """
+    The prepared creation, modification, or deletion of an object to be applied to the active database at a
+    future point.
+    """
+    branch = models.ForeignKey(
+        to=Branch,
+        on_delete=models.CASCADE,
+        related_name='staged_changes'
+    )
+    action = models.CharField(
+        max_length=20,
+        choices=ChangeActionChoices
+    )
+    object_type = models.ForeignKey(
+        to=ContentType,
+        on_delete=models.CASCADE,
+        related_name='+'
+    )
+    object_id = models.PositiveBigIntegerField(
+        blank=True,
+        null=True
+    )
+    object = GenericForeignKey(
+        ct_field='object_type',
+        fk_field='object_id'
+    )
+    data = models.JSONField(
+        blank=True,
+        null=True
+    )
+
+    class Meta:
+        ordering = ('pk',)
+
+    def __str__(self):
+        action = self.get_action_display()
+        app_label, model_name = self.object_type.natural_key()
+        return f"{action} {app_label}.{model_name} ({self.object_id})"
+
+    @property
+    def model(self):
+        return self.object_type.model_class()
+
+    def apply(self):
+        """
+        Apply the staged create/update/delete action to the database.
+        """
+        if self.action == ChangeActionChoices.ACTION_CREATE:
+            instance = deserialize_object(self.model, self.data, pk=self.object_id)
+            logger.info(f'Creating {self.model._meta.verbose_name} {instance}')
+            instance.save()
+
+        if self.action == ChangeActionChoices.ACTION_UPDATE:
+            instance = deserialize_object(self.model, self.data, pk=self.object_id)
+            logger.info(f'Updating {self.model._meta.verbose_name} {instance}')
+            instance.save()
+
+        if self.action == ChangeActionChoices.ACTION_DELETE:
+            instance = self.model.objects.get(pk=self.object_id)
+            logger.info(f'Deleting {self.model._meta.verbose_name} {instance}')
+            instance.delete()