[BUG] Incorrect awareness attribute count captured in decisionExplanationString in AwarenessAllocationDecider

[anshu1106]

Describe the bug
In some scenarios, when allocation decision is No by AwarenessAllocationDecider. No Allocation explanation reason message is showing incorrect number of awareness attributes when forced zone awareness is enabled.
For ex - A domain with 3 availability zones and forced zone awareness set as “cluster.routing.allocation.awareness.force.zone.values” = “a,b,c” gives shard not allocated reasons as:

there are too many copies of the shard allocated to nodes with attribute [zone], there are [3] total configured shard copies for this shard id and **[4] total attribute values**, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1]

Though count of awareness attribute is 3 but it is captured as 4.

To Reproduce
Steps to reproduce the behavior:

1. Create an unbalanced domain with 3 availability zones and node count ( 2,1,0). ie 2 data nodes in zone a, 1 data node in b and no data node in zone c. This distribution can be seen at times during zonal failures.

2. Set cluster.routing.allocation.awareness.force.zone.values = a,b,c

3. Create an index with 1 shard and 2 replica ie 3 shard copies. 2 shard copied get assigned to nodes in zone a and b but 1 shard copy stay unassigned due to forced zone awareness.

4. Add a data node in zone c and try to assign unassigned shard copy to a node in zone a.

5. Shard allocation failed with reason :
   there are too many copies of the shard allocated to nodes with attribute [zone], there are [3] total configured shard copies for this shard id and [4] total attribute values, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1] by AwarenessAllocationDecider

6. Wrote a unit test that failed due to this bug

public void testAllocationExplainForUnassignedShardsWithUnbalancedZones() {
    Settings settings = Settings.builder()
        .put("cluster.routing.allocation.node_concurrent_recoveries", 10)
        .put(ThrottlingAllocationDecider.CLUSTER_ROUTING_ALLOCATION_NODE_INITIAL_REPLICAS_RECOVERIES_SETTING.getKey(), 10)
        .put(ClusterRebalanceAllocationDecider.CLUSTER_ROUTING_ALLOCATION_ALLOW_REBALANCE_SETTING.getKey(), "always")
        .put("cluster.routing.allocation.awareness.attributes", "zone")
        .put("cluster.routing.allocation.awareness.force.zone.values", "a,b,c")
        .build();
      AllocationService strategy = createAllocationService(settings);

        logger.info("Building initial routing table for 'testAllocationExplainForUnassignedShardsWithUnbalancedZones'");
        Metadata metadata = Metadata.builder()
            .put(IndexMetadata.builder("test").settings(settings(Version.CURRENT)).numberOfShards(1).numberOfReplicas(2))
            .build();

        RoutingTable initialRoutingTable = RoutingTable.builder().addAsNew(metadata.index("test")).build();

        ClusterState clusterState = ClusterState.builder(ClusterName.CLUSTER_NAME_SETTING.getDefault(Settings.EMPTY))
            .metadata(metadata)
            .routingTable(initialRoutingTable)
            .build();

        logger.info("--> adding 3 nodes in different zones and do rerouting");
        clusterState = ClusterState.builder(clusterState)
            .nodes(
                DiscoveryNodes.builder()
                    .add(newNode("A-0", singletonMap("zone", "a")))
                    .add(newNode("A-1", singletonMap("zone", "a")))
                    .add(newNode("B-0", singletonMap("zone", "b")))
            )
            .build();
        clusterState = strategy.reroute(clusterState, "reroute");
        assertThat(clusterState.getRoutingNodes().shardsWithState(STARTED).size(), equalTo(0));
        assertThat(clusterState.getRoutingNodes().shardsWithState(INITIALIZING).size(), equalTo(1));

        logger.info("--> start the shard (primary)");
        clusterState = startInitializingShardsAndReroute(strategy, clusterState);
        assertThat(clusterState.getRoutingNodes().shardsWithState(STARTED).size(), equalTo(1));
        assertThat(clusterState.getRoutingNodes().shardsWithState(INITIALIZING).size(), equalTo(1));
        // One Shard is unassigned due to forced zone awareness
        assertThat(clusterState.getRoutingNodes().shardsWithState(UNASSIGNED).size(), equalTo(1));

        List<ShardRouting> unassignedShards = clusterState.getRoutingTable().shardsWithState(UNASSIGNED);

        ClusterSettings EMPTY_CLUSTER_SETTINGS = new ClusterSettings(
            Settings.EMPTY,
            ClusterSettings.BUILT_IN_CLUSTER_SETTINGS
        );
        // Add a new node in zone c
        clusterState = ClusterState.builder(clusterState)
            .nodes(DiscoveryNodes.builder(clusterState.nodes()).add(newNode("C-0", singletonMap("zone", "c"))))
            .build();

        final AwarenessAllocationDecider decider = new AwarenessAllocationDecider(settings, EMPTY_CLUSTER_SETTINGS);

        final RoutingAllocation allocation = new RoutingAllocation(
            new AllocationDeciders(Collections.singleton(decider)),
            clusterState.getRoutingNodes(),
            clusterState,
            null,
            null,
            0L
        );
        allocation.debugDecision(true);

        Decision decision = null;
        RoutingNodes nodes = clusterState.getRoutingNodes();

        for (RoutingNode node : nodes) {
            // Try to allocate unassigned shared to A-0, fails because of forced zone awareness
            if (node.nodeId().equals("A-0")) {
                decision = decider.canAllocate(unassignedShards.get(0), node, allocation);
                assertEquals(Decision.Type.NO, decision.type());
                assertEquals(decision.getExplanation(), "there are too many copies of the shard allocated to nodes with attribute" +
                    " [zone], there are [3] total configured shard copies for this shard id and [3]" +
                    " total attribute values, expected the allocated shard count per attribute [2] to" +
                    " be less than or equal to the upper bound of the required number of shards per attribute [1]");
            }

        }
    }

Expected behavior
Decision explanation should capute correct number of attributes/zones = 3

It should be :-

there are too many copies of the shard allocated to nodes with attribute [zone], there are [3] total configured shard copies for this shard id and [3] total attribute values, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1]

Screenshots
If applicable, add screenshots to help explain your problem.