Merge branch 'bootstrap_ssot_integration' of github.com:nautobot/naut…

…obot-app-ssot into bootstrap_ssot_integration

changes/479.fixed

@@ -0,0 +1 @@
+Corrected the attribute used to reference the ControllerManagedDeviceGroup off a Controller object.

changes/542.documentation

@@ -0,0 +1 @@
+Correct documentation for ACI integration and add missing DNA Center installation documentation.

changes/546.added

@@ -0,0 +1,2 @@
+Added support for specifying LocationType for Areas, Buildings, and Floors in DNA Center integration.
+Added support for specifying LocationType for Buildings in Device42 integration.

changes/546.documentation

@@ -0,0 +1,2 @@
+Added documentation on how to use DNA Center integration along with screenshots of the steps.
+Updated documentation for Device42 integration and updated Job form screenshot to update for Building LocationType Job form change.

changes/548.fixed

@@ -0,0 +1 @@
+Fixed SSoT jobs not respecting DryRun variable.

changes/558.fixed

@@ -0,0 +1 @@
+Fixed VRF attribute for Prefix create() to be ids instead of attrs.

changes/561.fixed

@@ -0,0 +1 @@
+Bug in IP Fabric that causes some network columns to return host bits set; changed `ip_network` to use `strict=False`.

docs/admin/integrations/aci_setup.md

@@ -56,7 +56,7 @@ PLUGINS_CONFIG = {
 
 All APIC specific settings have been updated to use the Controller and related ExternalIntegration objects. The ExternalIntegration object that is assigned to the Controller will define the APIC base URL, user credentials, and SSL verification. It will also have a `tenant_prefix` key in the `extra_config` section of the ExternalIntegration to define the Tenant prefix.
 
-The `aci_apics` setting from the `nautobot_config.py` file is no longer used and any configuration found for it will be automatically migrated into a Controller and an ExternalIntegration object.
+The `aci_apics` setting from the `nautobot_config.py` file is no longer used. Any configuration found for the APICs that were defined in `aci_apics` will need to be manually input into the Nautobot UI to create the required ExternalIntegration and Controller objects.
 
 ## Nautobot Objects Affected by Settings
 
@@ -82,14 +82,19 @@ There are example YAML files for a few common switch models in `nautobot_ssot/in
     When upgrading from `nautobot-plugin-ssot-aci` app, it's necessary to [avoid conflicts](../upgrade.md#potential-apps-conflicts).
 
 - Uninstall the old app:
+
     ```shell
     pip uninstall nautobot-plugin-ssot-aci
     ```
+
 - Upgrade the app with required extras:
+
     ```shell
     pip install --upgrade nautobot-ssot[aci]
     ```
+
 - Fix `nautobot_config.py` by removing `nautobot_ssot_aci` from `PLUGINS` and merging app configuration into `nautobot_ssot`:
+
     ```python
     PLUGINS = [
         "nautobot_ssot",

docs/admin/integrations/dna_center_setup.md

@@ -12,7 +12,7 @@ pip install nautobot-ssot[dna_center]
 
 ## Configuration
 
-Connecting to a DNA Center instance is handled through the Nautobot [Controller](https://docs.nautobot.com/projects/core/en/stable/development/core/controllers/) object. There is an expectation that you will create an [ExternalIntegration](https://docs.nautobot.com/projects/core/en/stable/user-guide/platform-functionality/externalintegration/) with the requisite connection information for your DNA Center instance attached to that Controller object. All imported Devices will be associated to a [ControllerManagedDeviceGroup](https://docs.nautobot.com/projects/core/en/stable/user-guide/core-data-model/dcim/controllermanageddevicegroup/) that is found or created during each Job run. It will update the group name to be "\<Controller name\> Managed Devices" if it exists. When running the Sync Job you will specify which DNA Center Controller instance you wish to synchronize with.  Other behaviors for the integration can be controlled with the following settings:
+Connecting to a DNA Center instance is handled through the Nautobot [Controller](https://docs.nautobot.com/projects/core/en/stable/development/core/controllers/) object. There is an expectation that you will create an [ExternalIntegration](https://docs.nautobot.com/projects/core/en/stable/user-guide/platform-functionality/externalintegration/) with the requisite connection information for your DNA Center instance attached to that Controller object. All imported Devices will be associated to a [ControllerManagedDeviceGroup](https://docs.nautobot.com/projects/core/en/stable/user-guide/core-data-model/dcim/controllermanageddevicegroup/) that is found or created during each Job run. It will update the group name to be "<Controller name\> Managed Devices" if it exists. When running the Sync Job you will specify which DNA Center Controller instance you wish to synchronize with.  Other behaviors for the integration can be controlled with the following settings:
 
 | Configuration Variable                              | Type    | Usage                                                      | Default              |
 | --------------------------------------------------- | ------- | ---------------------------------------------------------- | -------------------- |

docs/admin/integrations/index.md

@@ -6,6 +6,7 @@ This Nautobot app supports the following integrations:
 - [Bootstrap](./bootstrap_setup.md)
 - [Arista CloudVision](./aristacv_setup.md)
 - [Device42](./device42_setup.md)
+- [Cisco DNA Center](./dna_center_setup.md)
 - [Infoblox](./infoblox_setup.md)
 - [IPFabric](./ipfabric_setup.md)
 - [Itential](./itential_setup.md)

docs/user/integrations/device42.md

@@ -6,7 +6,7 @@ From Device42 into Nautobot, it synchronizes the following objects:
 
 | Device42                | Nautobot                     |
 | ----------------------- | ---------------------------- |
-| Buildings               | Sites                        |
+| Buildings               | Locations                    |
 | Rooms                   | RackGroups                   |
 | Racks                   | Racks                        |
 | Vendors                 | Manufacturers                |
@@ -36,7 +36,9 @@ To start the synchronization, simply select the ExternalIntegration that corresp
 
 ![Job Form](../../images/device42_job-form.png)
 
-If you wish to just test the synchronization but not have any data created in Nautobot you'll want to select the `Dry run` toggle. Clicking the `Debug` toggle will enable more verbose logging to inform you of what is occuring behind the scenes. Finally, the `Bulk import` option will enable bulk create and update operations to be used when the synchronization is complete. This can improve performance times for the App by forsaking validation of the imported data. Be aware that this could potentially cause bad data to be pushed into Nautobot.
+> As of SSoT 3.2.0 you now have the option to define the LocationType to use for imported Buildings. If unspecifed in the Job form it will resort to using the Site LocationType as it did previously.
+
+If you wish to just test the synchronization but not have any data created in Nautobot you'll want to select the `Dryrun` toggle. Clicking the `Debug` toggle will enable more verbose logging to inform you of what is occuring behind the scenes. Finally, the `Bulk import` option will enable bulk create and update operations to be used when the synchronization is complete. This can improve performance times for the App by forsaking validation of the imported data. Be aware that this could potentially cause bad data to be pushed into Nautobot.
 
 Running this Job will redirect you to a `Nautobot Job Result` view.
 

docs/user/integrations/dna_center.md

@@ -0,0 +1,85 @@
+# Cisco DNA Center SSoT Integration
+
+The Cisco DNA Center SSoT integration is built as part of the [Nautobot Single Source of Truth (SSoT)](https://github.com/nautobot/nautobot-app-ssot) app. The SSoT app enables Nautobot to be the aggregation point for data coming from multiple systems of record (SoR).
+
+From Cisco DNA Center into Nautobot, it synchronizes the following objects:
+
+| DNA Center              | Nautobot                     |
+| ----------------------- | ---------------------------- |
+| Areas                   | Location*                    |
+| Buildings               | Location*                    |
+| Floors                  | Location*                    |
+| Devices                 | Devices**                    |
+| Ports                   | Interfaces                   |
+| Prefixes                | Prefixes                     |
+| IP Addresses            | IP Addresses                 |
+
+`*` As of SSoT 3.2.0 the LocationType for Areas, Buildings, and Floors can be defined in the Job form. Prior to SSoT 3.2.0 the DNA Center integration creates a Region, Site, and Floor LocationType and imports Areas as Region Locations, Buildings as Site Locations, and Floors as Floor Locations.
+
+`**` If the [Device Lifecycle Nautobot app](https://github.com/nautobot/nautobot-app-device-lifecycle-mgmt) is found to be installed, a matching Version will be created with a RelationshipAssociation connecting the Device and that Version.
+
+## Usage
+
+Once the app is installed and configured, you will be able to perform an inventory ingestion from DNA Center into Nautobot. From the Nautobot SSoT Dashboard view (`/plugins/ssot/`), DNA Center will show as a Data Source.
+
+![Dashboard View](../../images/dnac_dashboard.png)
+
+From the Dashboard, you can also view more information about the App by clicking on the `DNA Center to Nautobot` link and see the Detail view. This view will show the mappings of DNA Center objects to Nautobot objects, the sync history, and other configuration details for the App:
+
+![Detail View](../../images/dnac_detail-view.png)
+
+In order to utilize this integration you must first enable the Job. You can find the available installed Jobs under Jobs -> Jobs:
+
+![Job List](../../images/dnac_job_list.png)
+
+To enable the Job you must click on the orange pencil icon to the right of the `DNA Center to Nautobot` Job. You will be presented with the settings for the Job as shown below:
+
+![Job Settings](../../images/dnac_job_settings.png)
+
+You'll need to check the `Enabled` checkbox and then the `Update` button at the bottom of the page. You will then see that the play button next to the Job changes to blue and becomes functional, linking to the Job form.
+
+![Enabled Job](../../images/dnac_enabled_job.png)
+
+Once the Job is enabled, you'll need to manually create a few objects in Nautobot to use with the Job. First, you'll need to create a Secret that contains the username and password for authenticating to your desired DNA Center instance:
+
+![Username Secret](../../images/dnac_username_secret.png)
+
+![Password Secret](../../images/dnac_password_secret.png)
+
+Once the required Secrets are created, you'll need to create a SecretsGroup that pairs them together and defines the Access Type of HTTP(S) like shown below:
+
+![DNAC SecretsGroup](../../images/dnac_secretsgroup.png)
+
+With the SecretsGroup defined containing your instance credentials you'll then need to create an ExternalIntegration object to store the information about the DNA Center instance you wish to synchronize with.
+
+![DNAC ExternalIntegration](../../images/dnac_external_integration.png)
+
+> The only required portions are the Name, Remote URL, Verify SSL, and Secrets Group. The `Extra Config` section allows you to specify the port that DNA Center is running on. It will default to 443 if unspecified.
+
+![DNAC ExternalIntegration](../../images/dnac_external_integration_adv.png)
+
+The final step before running the Job is to create a Controller that references the ExternalIntegration that you just created. You can attach a `Managed Device Group` to the Controller for all imported Devices to be placed in. If you don't create a Managed Device Group, one will be created automatically and associated to the specified Controller with the name of `<Controller name> Managed Devices`.
+
+![DNAC Controller](../../images/dnac_controller.png)
+
+> You can utilize multiple DNA Center Controllers with this integration as long as you specify a unique Tenant per Controller. The failure to use differing Tenants will have the Devices, Prefixes, and IPAddresses potentially removed if they are non-existent on the additional Controller. Locations should remain unaffected.
+
+With those configured, you will then need to define a LocationType to use for each DNA Center location type of Areas, Buildings, and Floors. With those created, you can run the Job to start the synchronization:
+
+> When creating the Area LocationType you must check the "Nestable" option.
+
+![Job Form](../../images/dnac_job_form.png)
+
+If you wish to just test the synchronization but not have any data created in Nautobot you'll want to select the `Dryrun` toggle. Clicking the `Debug` toggle will enable more verbose logging to inform you of what is occuring behind the scenes. Finally, the `Bulk import` option will enable bulk create and update operations to be used when the synchronization is complete. This can improve performance times for the integration by forsaking validation of the imported data. Be aware that this could potentially cause bad data to be pushed into Nautobot. After those toggles there are also dropdowns that allow you to specify the DNA Center Controller to synchronize with and to define the LocationTypes to use for the imported Areas, Buildings, and Floors from DNA Center. In addition, there are also some optional settings on the Job form:
+
+- The Location Mapping allows you to define a dictionary of Location mappings. This feature is intended for specifying parent Locations for the Areas and Building locations in DNA Center. This is useful if this information is missing from DNA Center but required for Nautobot or to allow you to change the information as it's imported to match information from another System of Record. The expected pattern for this field is `{"<Location Name>": {"parent": "<Parent location Name>"}}`.
+
+- Finally there is an option to specify a Tenant to be assigned to the imported Devices, Prefixes, and IPAddreses. This is handy for cases where you have multiple DNA Center instances that are used by differing business units.
+
+Running this Job will redirect you to a `Nautobot Job Result` view.
+
+![JobResult View](../../images/dnac_jobresult.png)
+
+Once the Job has finished you can click on the `SSoT Sync Details` button at the top right of the Job Result page to see detailed information about the data that was synchronized from DNA Center and the outcome of the sync Job.
+
+![SSoT Sync Details](../../images/dnac_ssot-sync-details.png)

docs/user/integrations/index.md

@@ -6,6 +6,7 @@ This Nautobot app supports the following integrations:
 - [Arista CloudVision](./aristacv.md)
 - [Bootstrap](./bootstrap.md)
 - [Device42](./device42.md)
+- [Cisco DNA Center](./dna_center.md)
 - [Infoblox](./infoblox.md)
 - [IPFabric](./ipfabric.md)
 - [Itential](./itential.md)

mkdocs.yml

@@ -112,6 +112,7 @@ nav:
           - Arista CloudVision: "user/integrations/aristacv.md"
           - Bootstrap: "user/integrations/bootstrap.md"
           - Device42: "user/integrations/device42.md"
+          - DNA Center: "user/integrations/dna_center.md"
           - Infoblox: "user/integrations/infoblox.md"
           - IPFabric: "user/integrations/ipfabric.md"
           - Itential: "user/integrations/itential.md"
@@ -128,6 +129,7 @@ nav:
           - Arista CloudVision: "admin/integrations/aristacv_setup.md"
           - Bootstrap: "admin/integrations/bootstrap_setup.md"
           - Device42: "admin/integrations/device42_setup.md"
+          - DNA Center: "admin/integrations/dna_center_setup.md"
           - Infoblox: "admin/integrations/infoblox_setup.md"
           - IPFabric: "admin/integrations/ipfabric_setup.md"
           - Itential: "admin/integrations/itential_setup.md"

nautobot_ssot/exceptions.py

@@ -0,0 +1,67 @@
+"""Custom Exceptions to be used with SSoT integrations."""
+
+
+class AdapterLoadException(Exception):
+    """Raised when there's an error while loading data."""
+
+
+class AuthFailure(Exception):
+    """Exception raised when authenticating to endpoint fails."""
+
+    def __init__(self, error_code, message):
+        """Populate exception information."""
+        self.expression = error_code
+        self.message = message
+        super().__init__(self.message)
+
+
+class ConfigurationError(Exception):
+    """Exception thrown when Job configuration is wrong."""
+
+
+class JobException(Exception):
+    """Exception raised when failure loading integration Job."""
+
+    def __init__(self, message):
+        """Populate exception information."""
+        self.message = message
+        super().__init__(self.message)
+
+
+class InvalidUrlScheme(Exception):
+    """Exception raised for wrong scheme being passed for URL.
+
+    Attributes:
+        message (str): Returned explanation of Error.
+    """
+
+    def __init__(self, scheme):
+        """Initialize Exception with wrong scheme in message."""
+        self.message = f"Invalid URL scheme '{scheme}' found!"
+        super().__init__(self.message)
+
+
+class MissingConfigSetting(Exception):
+    """Exception raised for missing configuration settings.
+
+    Attributes:
+        message (str): Returned explanation of Error.
+    """
+
+    def __init__(self, setting):
+        """Initialize Exception with Setting that is missing and message."""
+        self.setting = setting
+        self.message = f"Missing configuration setting - {setting}!"
+        super().__init__(self.message)
+
+
+class MissingSecretsGroupException(Exception):
+    """Custom Exception in case SecretsGroup is not found on ExternalIntegration."""
+
+
+class RequestConnectError(Exception):
+    """Exception class to be raised upon requests module connection errors."""
+
+
+class RequestHTTPError(Exception):
+    """Exception class to be raised upon requests module HTTP errors."""

nautobot_ssot/integrations/aci/diffsync/adapters/aci.py

@@ -435,8 +435,8 @@ def load_devices(self):
                 site=self.site,
                 site_tag=self.site,
                 controller_group=(
-                    self.job.apic.controller_managed_device_group.name
-                    if self.job.apic.controller_managed_device_group
+                    self.job.apic.controller_managed_device_groups.name
+                    if self.job.apic.controller_managed_device_groups
                     else ""
                 ),
             )

nautobot_ssot/integrations/aci/diffsync/client.py

@@ -11,6 +11,8 @@
 import requests
 import urllib3
 
+from nautobot_ssot.exceptions import RequestConnectError, RequestHTTPError
+
 from .utils import (
     ap_from_dn,
     bd_from_dn,
@@ -26,14 +28,6 @@
 logger = logging.getLogger(__name__)
 
 
-class RequestConnectError(Exception):
-    """Exception class to be raised upon requests module connection errors."""
-
-
-class RequestHTTPError(Exception):
-    """Exception class to be raised upon requests module HTTP errors."""
-
-
 class AciApi:
     """Representation and methods for interacting with aci."""
 

nautobot_ssot/integrations/aci/diffsync/models/nautobot.py

@@ -449,7 +449,7 @@ def create(cls, adapter, ids, attrs):
         try:
             vrf_tenant = OrmTenant.objects.get(name=attrs["vrf_tenant"])
         except OrmTenant.DoesNotExist:
-            adapter.job.logger.warning(f"Tenant {attrs['vrf_tenant']} not found for VRF {attrs['vrf']}")
+            adapter.job.logger.warning(f"Tenant {attrs['vrf_tenant']} not found for VRF {ids['vrf']}")
             vrf_tenant = None
             return None
 

nautobot_ssot/integrations/aci/jobs.py

@@ -5,6 +5,7 @@
 from nautobot.dcim.models import Controller, Location
 from nautobot.extras.jobs import BooleanVar, Job, ObjectVar
 
+from nautobot_ssot.exceptions import ConfigurationError
 from nautobot_ssot.integrations.aci.diffsync.adapters.aci import AciAdapter
 from nautobot_ssot.integrations.aci.diffsync.adapters.nautobot import NautobotAdapter
 from nautobot_ssot.integrations.aci.diffsync.client import AciApi
@@ -14,10 +15,6 @@
 name = "Cisco ACI SSoT"  # pylint: disable=invalid-name, abstract-method
 
 
-class ConfigurationError(Exception):
-    """Exception thrown when Job configuration is wrong."""
-
-
 class AciDataSource(DataSource, Job):  # pylint: disable=abstract-method, too-many-instance-attributes
     """ACI SSoT Data Source."""