Drop dynamic ID lookup

[stawii]

This discussion is question/proposal. TL/DR at bottom.

I found out that this provider uses .name as unique identifier of all(?) resources. The name is indeed unique, however for all API operations requires .id (like *2D1) which is internal RouterOS thing. To make this work, before update/delete operation functions uses dynamicIdLookup() to map name to ID. This is still fine, but... why it's like that?

The only scenario I found it helps if when you manually dropped and recreated resource, then Terraform can still manage it. This should be mitigated by removing resource from state and importing new one - most providers works like that, because underlying APIs requires this flow. Anyways that kind of "swap" is bad for consistency.
I'm not sure what happens during backup/restore of configuration - if ID stays the same - it's perfect! if not.. either you shouldn't restore configuration but just use Terraform to provision your device again (That's why you're using TF in first place, right?), or just import all resources again.
What terrifies me, is case when you just renamed resource outside Terraform (intentionally or by mistake) which I guess happens more often. Then provider will just ignore it and try to create it again with unexpected consequences rather than renaming it back.
Even worse, same thing might happens when you rename resource INSIDE Terraform, the ID (which is name) isn't updating causing Terraform planing creating new resource on next run (tested this with routeros_interface_bridge) - this is definitely a bug, and should be fixed.

TL/DR: Why not use RouterOS .id as Terraform resource id?

[vaerh]

Yes, I could write a lot about the attempts to connect Mikrotik and Teraform, but there are a number of problems which were described above. Unfortunately, the approach of managing only through TF (which was assumed by me at least) is not applicable. Only recently I made changes to be able to react to changes that were made from outside.
A large number of resources use .id as a key value. The type of keyword field is selected during the development of the resource. There is some compromise between convenience for machine-generated TF configuration and human scripting (the resource name is perceived better than the .id).
Using dynamicIdLookup is an attempt to preserve functionality in the context of the existing possibility to change resources from outside and using the name as a key field.
It seems to me that even in the cases you mentioned it is possible (maybe even necessary) to force recreation of resources when you change the name (whom it is a key field).
Please publish the script data used in the routeros_interface_bridge experiment and I will see what I can do about it.

[vaerh]

Yes, it's a big problem. Probably have to give up support for resources with a key field in the form of "name". But this requires a lot of consideration. As a solution, which would preserve the integrity of the logic when managing through TF, I added recreation of resources when they are renamed.

[stawii]

Hey, thanks for your response and patience. Here are my test cases.

Test: rename manually (outside Terraform)

First, lets create brand new bridge:

resource "routeros_interface_bridge" "test" {
  name = "terraform-test"
}

Then apply, everything is fine, clean plans - no additional changes. But after I've changed bridge name with:

/interface bridge set [ find name="terraform-test" ] name="terraform-renamed"

Terraform lost reference and wants to create it again:

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # routeros_interface_bridge.test has been deleted
  - resource "routeros_interface_bridge" "test" {
      - id                  = "terraform-test" -> null
      - name                = "terraform-test" -> null
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using
ignore_changes, the following plan may include actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # routeros_interface_bridge.test will be created
  + resource "routeros_interface_bridge" "test" {
      + id                      = (known after apply)
      + name                    = "terraform-test"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

This will create NEW bridge, and probably tries to assign ports to it causing collisions / fails - I didn't test this, but I assume disaster :)

Test: rename inside Terraform

After creating (same way as in previous step) I've changed:

resource "routeros_interface_bridge" "test" {
  name = "terraform-renamed"
}

plan looks like this:

Terraform will perform the following actions:

  # routeros_interface_bridge.test will be updated in-place
  ~ resource "routeros_interface_bridge" "test" {
        id                  = "terraform-test"
      ~ name                = "terraform-test" -> "terraform-renamed"
        # (23 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

This looks "fine" but notice the "id" will stay the same - misleading! Worst part is that, after apply (and successful renaming), running plan again gives the same output: "routeros_interface_bridge.test has been deleted" and wants to create it again with name = "terraform-renamed" and this obviously fails as this bridge exists:

routeros_interface_bridge.test: Creating...
╷
│ Error: PUT 'https://192.168.1.1/rest/interface/bridge' returned response code: 400, message: 'Bad Request', details: 'failure: already have interface with such name'
│
│   with routeros_interface_bridge.test,

So with current approach name field must be marked as write-once, or user must add it to ignore change set to avoid issues.
Alternatively changing Terraform'sid to RouterOS .id will solve it nice and clean, hopefully.

[stawii]

I checked the current PropIds from the resources and found it quite disturbing that some resources use ID and others use Name even though they are similar or related, ie:

• /interface/vlan uses Name, but /interface/bridge/vlan uses ID
• /interface/bridge uses Name, but /interface/bonding uses ID
• /ip/dhcp-server uses Name, but /ip/dhcp-server/lease uses ID

routeros/datasource_interfaces.go                     PropId(Id)
routeros/datasource_ip_addresses.go                   PropId(Id)
routeros/datasource_ip_routes.go                      PropId(Id)
routeros/datasource_ipv6_addresses.go                 PropId(Id)
routeros/mikrotik_serialize_test.go                   PropId(Id)
routeros/mikrotik_serialize_test.go                   PropId(Id)
routeros/resource_bgp_connection.go                   PropId(Id)
routeros/resource_bgp_template.go                     PropId(Id)
routeros/resource_capsman_channel.go                  PropId(Name)
routeros/resource_capsman_configuration.go            PropId(Name)
routeros/resource_capsman_datapath.go                 PropId(Name)
routeros/resource_capsman_manager.go                  PropId(Name)
routeros/resource_capsman_manager.go                  PropId(Id)
routeros/resource_capsman_manager.go                  PropId(Name)
routeros/resource_capsman_provisioning.go             PropId(Id)
routeros/resource_capsman_rates.go                    PropId(Name)
routeros/resource_capsman_security.go                 PropId(Name)
routeros/resource_interface_bonding.go                PropId(Id)
routeros/resource_interface_bridge.go                 PropId(Name)
routeros/resource_interface_bridge_port.go            PropId(Id)
routeros/resource_interface_bridge_settings.go        PropId(Name)
routeros/resource_interface_bridge_vlan.go            PropId(Id)
routeros/resource_interface_gre.go                    PropId(Name)
routeros/resource_interface_list.go                   PropId(Name)
routeros/resource_interface_list_member.go            PropId(Id)
routeros/resource_interface_ovpn_server.go            PropId(Id)
routeros/resource_interface_pppoe_client.go           PropId(Id)
routeros/resource_interface_veth.go                   PropId(Id)
routeros/resource_interface_vlan.go                   PropId(Name)
routeros/resource_interface_vrrp.go                   PropId(Name)
routeros/resource_interface_wireguard.go              PropId(Name)
routeros/resource_interface_wireguard_peer.go         PropId(Id)
routeros/resource_ip_address.go                       PropId(Id)
routeros/resource_ip_cloud.go                         PropId(Id)
routeros/resource_ip_dhcp_client.go                   PropId(Id)
routeros/resource_ip_dhcp_server.go                   PropId(Name)
routeros/resource_ip_dhcp_server_lease.go             PropId(Id)
routeros/resource_ip_dhcp_server_network.go           PropId(Id)
routeros/resource_ip_dns.go                           PropId(Name)
routeros/resource_ip_dns_record.go                    PropId(Id)
routeros/resource_ip_firewall_addr_list.go            PropId(Id)
routeros/resource_ip_firewall_filter.go               PropId(Id)
routeros/resource_ip_firewall_mangle.go               PropId(Id)
routeros/resource_ip_firewall_nat.go                  PropId(Id)
routeros/resource_ip_pool.go                          PropId(Name)
routeros/resource_ip_route.go                         PropId(Id)
routeros/resource_ip_service.go                       PropId(Name)
routeros/resource_ipv6_address.go                     PropId(Id)
routeros/resource_ipv6_firewall_addr_list.go          PropId(Id)
routeros/resource_ipv6_firewall_filter.go             PropId(Id)
routeros/resource_ipv6_route.go                       PropId(Id)
routeros/resource_ovpn_server.go                      PropId(Id)
routeros/resource_ppp_profile.go                      PropId(Id)
routeros/resource_ppp_secret.go                       PropId(Id)
routeros/resource_routing_ospf_area.go                PropId(Id)
routeros/resource_routing_ospf_instance.go            PropId(Id)
routeros/resource_routing_ospf_interface_template.go  PropId(Id)
routeros/resource_routing_table.go                    PropId(Id)
routeros/resource_snmp.go                             PropId(Id)
routeros/resource_snmp_community.go                   PropId(Id)
routeros/resource_system_certificate.go               PropId(Id)
routeros/resource_system_identity.go                  PropId(Name)
routeros/resource_system_scheduler.go                 PropId(Name)
routeros/resource_system_user.go                      PropId(Id)

[vaerh]

Hi!
I apologize that there is no movement on this problem. While the changes are simple (change of name to identifier), I need to write a function to update the statefile. That's exactly what we have problems with.
I can't promise that I'll do it anytime soon. But I'll try!

[vaerh]

Solved.

The release is available on GitHub release