java.lang.IllegalStateException: missing file for budgets : /var/run/zilla/budgets127

[vordimous]

Zilla crashes with the below error

Steps to reproduce the behavior:

1. Run the Zilla quickstart
2. wait 3-5 min
3. The below error happens and the server will restart
4. This repeats every 3-5 min

 org.agrona.concurrent.AgentTerminationException: java.lang.IllegalStateException: missing file for budgets : /var/run/zilla/budgets127
     at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.DispatchAgent.doWork(DispatchAgent.java:707)
     at org.agrona.core/org.agrona.concurrent.AgentRunner.doDutyCycle(AgentRunner.java:291)
     at org.agrona.core/org.agrona.concurrent.AgentRunner.run(AgentRunner.java:164)
     at java.base/java.lang.Thread.run(Thread.java:1623)
 Caused by: java.lang.IllegalStateException: missing file for budgets : /var/run/zilla/budgets127
     at org.agrona.core/org.agrona.IoUtil.checkFileExists(IoUtil.java:507)
     at org.agrona.core/org.agrona.IoUtil.mapExistingFile(IoUtil.java:396)
     at org.agrona.core/org.agrona.IoUtil.mapExistingFile(IoUtil.java:361)
     at [email protected]/io.aklivity.zilla.runtime.engine.internal.layouts.BudgetsLayout$Builder.build(BudgetsLayout.java:144)
     at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.DispatchAgent.newBudgetDebitor(DispatchAgent.java:1620)
     at org.agrona.core/org.agrona.collections.Int2ObjectHashMap.computeIfAbsent(Int2ObjectHashMap.java:268)
     at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.DispatchAgent.supplyDebitor(DispatchAgent.java:536)
     at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.budget.KafkaMergedBudgetAccountant.supplyDebitor(KafkaMergedBudgetAccountant.java:49)
     at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaCacheClientFetchFactory$KafkaCacheClientFetchStream.onClientReplyWindow(KafkaCacheClientFetchFactory.java:1645)
     at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaCacheClientFetchFactory$KafkaCacheClientFetchStream.onClientMessage(KafkaCacheClientFetchFactory.java:981)
     at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaCacheClientFetchFactory.lambda$newStream$2(KafkaCacheClientFetchFactory.java:248)
     at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.DispatchAgent.handleReadReply(DispatchAgent.java:1276)
     at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.DispatchAgent.handleRead(DispatchAgent.java:1045)
     at [email protected]/io.aklivity.zilla.runtime.engine.internal.concurent.ManyToOneRingBuffer.read(ManyToOneRingBuffer.java:181)
     at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.DispatchAgent.doWork(DispatchAgent.java:701)
     ... 3 more
     Suppressed: java.lang.Exception: [engine/data#0]        [0x00000000000001fe] streams=[consumeAt=0x00057e78 (0x0000000000057e78), produceAt=0x00058118 (0x0000000000058118)]
             at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.DispatchAgent.doWork(DispatchAgent.java:705)
             ... 3 more

[jfallows]

We'll need to check with command-log or zilla dump to be certain, but this looks like perhaps an invalid value being passed for budgetId in reply window from one of the bindings in the quickstart, sent towards kafka cache client.

[attilakreiner]

These two commits will fix this issue:

1. 4d25e89 (already merged to develop)
2. Fix mergedReplyBudgetId #579 (PR in review)

We expect this issue to be fixed in the next release where both these commits will be included.