NatsKV watch by key error if keys count greater than 650

[Bykiev]

Observed behavior

When watching the keys and the number of keys is greater than 650 the error is thrown: NATS.Client.JetStream.NatsJSApiException" in System.Private.CoreLib.dll

Expected behavior

No error should be thrown

Server and client version

NATS server 2.10.20
NATS .NET v2.4.0

Host environment

No response

Steps to reproduce

var config = new NatsKVConfig(bucket)
{
    Storage = NatsKVStorageType.Memory
};

var store = await _context.CreateStoreAsync(config, cancellationToken);

await foreach (var m in store.WatchAsync<ParameterData>(keys, cancellationToken: cancellationToken))

[mtmk]

thanks for the report @Bykiev what is the message from NatsJSApiException? I can't reproduce on my local machine:

using NATS.Client.Core;
using NATS.Client.JetStream;
using NATS.Client.KeyValueStore;

var nc = new NatsConnection();
var js = new NatsJSContext(nc);
var kv = new NatsKVContext(js);

var store = await kv.CreateStoreAsync("b1");

var keys = new List<string>();
for (var i = 0; i < 1_000; i++)
{
    keys.Add($"key{i}");
}

await foreach (var m in store.WatchAsync<int>(keys))
{
    Console.WriteLine($"Received {m.Key} = {m.Value}");
}

Running against server with no config:

$ nats-server -js

[Bykiev]

Unfortunately I don't see any messages, seems the keys count can be specific to PC. Can you please try to test with more keys?

[mtmk]

got you. When I increase to 10_000 I'm seeing the exception:

NatsJSApiNoResponseException: No API response received from the server

also server is giving warnings:

[37092] 2024/09/25 14:23:31.000705 [WRN] Internal subscription on "$JS.API.CONSUMER.CREATE.KV_b1.KZBDOGWKEC3kkgyBgx5RUh" took too long: 13.5648353s

You might be able to get around it by increasing the timeout on the client side:

var nc = new NatsConnection(new NatsOpts { RequestTimeout = TimeSpan.FromSeconds(30) });

But I would imagine this feature isn't supposed to have 100's of keys instead you should structure your keys and use wildcards e.g.:

await foreach (var m in store.WatchAsync<int>(["orders.new.>", "shipments.new.>"]))

cc @ripienaar

[ripienaar]

Indeed, you should definitely look to restructure your subjects/keys. The bottleneck here is on the server and for now this just won't work. We're aware we should improve the handling of consumers with my subjects on the server - however in no way is 650 subjects in a watch the expected usage :)

[Bykiev]

Thank you guys for your research, we will try to rethink our use case!