Prioritized Replica Recovery is reversed by date

[pickypg]

Prioritized allocation enables the recovery in the order of index.priority > index.creation_date > index.name (reversed). However, I've found that when allowing it to work based on index.creation_date (the default mechanism), it does it in the reverse of the expected order relative to replicas.

It's easy enough to reproduce with enough daily indices by manually deleting the replicas from one node, throttling the heck out of recovery, and speeding up monitoring:

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.node_concurrent_recoveries" : 1,
    "indices.recovery.concurrent_streams" : 1,
    "indices.recovery.concurrent_small_file_streams" : 1,
    "indices.recovery.max_bytes_per_sec" : "1mb",
    "marvel.agent.interval" : "500ms"
  }
}

As I was watching it, I decided to take some screenshots:

1. 
2. 

This also appears to not be honoring the index.priority either, as I tried to use it as a workaround and it did not impact the recovery order at all, which makes me assume that this is not even coming into play during replica recovery.

[clintongormley]

@s1monw could you take a look at this please?

[s1monw]

I don't understand what you are testing here. I can't see the priorities you are giving, I don't see if the replicas where allocated before and if not there will be no ordering as far as I can tell. I don't see if primaries got allocated first and I wonder what you expected to see sorry it's unclear.

[nik9000]

I don't understand what you are testing here. I can't see the priorities you are giving, I don't see if the replicas where allocated before and if not there will be no ordering as far as I can tell. I don't see if primaries got allocated first and I wonder what you expected to see sorry it's unclear.

I'm not familiar with the screenshot source but it looks like the indexes are recovering oldest to newest rather than newest to oldest. But I'm likely reading that wrong.

[pickypg]

@s1monw

I don't understand what you are testing here.

Replica recovery order with 2 nodes.

1. I throttled recovery as shown.
2. I took the second node offline.
3. I deleted all of its .marvel-* indices from the offline node.
4. I restarted the offline node and watched recovery.

I can't see the priorities you are giving.

I only set index.priority after seeing the images above. I picked arbitrary indices in the middle of the group and gave higher values for them individually (e.g., .marvel-2015.08.22 I gave the priority of 200). All of the creation dates are going to be roughly around midnight of the date of the index (no weirdness or cheating on creation of the indices).

I don't see if primaries got allocated first

They did. Synced flushed replica shards (not shown) also got recovered before these replicas were recovered.

I wonder what you expected to see sorry it's unclear.

I expected to see what @nik9000 suggested: the newest to oldest recovery of the replicas. Basically, .marvel-2015.08.28's replica should be recovered before .marvel-2015.08.27's replica, which should be recovered before .marvel-2015.08.26's replica (and so on).

It seems like the replica's do not consider priority in their recovery order and the oldest indices are being recovered.

[s1monw]

I deleted all of its .marvel-* indices from the offline node.

if you don't let the gateway allocator fetch any replicas to recover it won't respect priorities and will leave the rest to the shard balancer. The balancer will do it's own sorting at this point. This has never been implemented