Bug executing in Front End Transactions AGAIN

[AdrianKBL]

Description

Good evening,

Again (Yes I've reported this 2 times already), we get each 300 - 500 NFTs minted executing transactions in Front End different errors. Same as the others times when I reported.

x1 Mint Transactions with 10 NFTs
x1 Hbar Transfer Transaction
Loop

I'm getting this 2 different errors. It's need to be fixed ASAP, all our customers are having problems with it and they can't even mint their collections.

Steps to reproduce

x1 Mint Transactions with 10 NFTs
x1 Hbar Transfer Transaction
Loop

Additional context

No response

Hedera network

mainnet

Version

2.20.0

Operating system

macOS

[ochikov]

Hello @AdriMsM91,
As you can see, you are getting errors from executing transactions from Node 3 and Node 4. They are down:

Also, I don't know your exact configuration, but if you explicitly defined to use those nodes, you cannot get through them, because they are not functioning. Did you use the setNodeAccountIds functionality?

Also, you can update to the latest version, because the latest version support looping through different proxies of a down node to try to proceed the transaction through another proxy.

[AdrianKBL]

Hello @ochikov

We're not setting any nodes in any transaction. The own SDK retries to another node If It fails, also as you can see there are some other problems as I reported before also.

Transaction must been frozen before calculating the hash will be stable, try calling freeze.

[ochikov]

@AdriMsM91
I've performed ~5000 transactions last few days and with the newest version, I cannot reproduce the bug that the SDK cannot switch to another Node. Please update to the latest version. We've fixed also the freezing issue.

[petreze]

@AdriMsM91
Also on the Transaction must been frozen before calculating the hash will be stable, try calling freeze. topic. First and foremost - the error message can come from several places from the SDK, so we cannot guess what triggers this on your side. Secondly, the message is self-explanatory - the transaction has to be frozen in order to achieve what you are trying to accomplish.

On the other issue - I will also try to reproduce it
x1 Mint Transactions with 10 NFTs and then x1 Hbar Transfer Transaction repeatedly in a loop
I will write you if there is any progress

[AdrianKBL]

@ochikov I will update once our big mint happens tomorrow. Thanks.

@petreze Thanks, appreciated.

[ochikov]

@AdriMsM91 do we have any updates from the updating to the newest version?

[AdrianKBL]

We will be updating tomorrow. Today we have our big mint and we don't want to risky anything.

[AdrianKBL]

Good night @petreze @ochikov

We upgraded today, so we will let you know If any of our customers reports again. Btw I saw in different Discords from wallets, people having the same Error than I'm reporting, as you can see, today in HashPack this has been reported also.

So I guess obviously there is a problem with this, since If It fails ramdomly is nothing to be about freezing it or not, is more, we do the .execute() directly and It works 90% of the time, 10% fails.

[ochikov]

@AdriMsM91 We were able to reproduce the error when we did not have any healthy nodes left. We are going to investigate it. Thank you for the report.

[AdrianKBL]

Yeah @ochikov also with the last version 2.25.0 , today in production we have the same problem.

[petreze]

@AdriMsM91
Just to clarify, how exactly are you setting up the client for mainnet?

[ochikov]

@AdriMsM91 In addition to @petreze I can add that we were able to reproduce it ONLY when there weren't healthy nodes left. For example with the following setup through web:

 const client = Client.forNetwork({
      "https://node01-00-grpc.swirlds.com:443": "0.0.4",
    });

After that when the Node become overwhelmed, we are getting 503 error and then there is no other healthy node to work with. Please see the images bellow:

I remember that your setup is using your own proxies. We need more info about the setup, because this is the only way we can reproduce the error.

[AdrianKBL]

Hello @petreze and @ochikov

Since long time ago, we don't use our setup for proxies, basically since Swirlds Labs deployed every proxy for every node, we are not using that anymore.

This is how I'm creating the client for Mainnet. Also talking in the wallet-ecosystem-channel this weekend, looks like it is happening to many people.

let client = Client.forNetwork(network);

[petreze]

@AdriMsM91
Good but the important thing is the network object. What exactly are you passing to forNetwork()?

[AdrianKBL]

@petreze

Client.forNetwork('mainnet');

[petreze]

Hey @AdriMsM91
We think that we discovered the issue and made a fix for it which is available in the newest 2.26.0-beta.2 version of the SDK. Could you please try and test with it?

[AdrianKBL]

Hello @petreze @ochikov

Actually is impossible to test anything in Mainnet, we have upgraded to the newest version and we just keep getting different errors each time. Each 300-400 NFTs, now I'm getting this error. It's really frustrating tbh.

max attempts of 10 was reached for request with last error being: HttpError: failed with error code: 503

[petreze]

Newest official v2.25.0 or the one I mentioned - v2.26.0-beta.2?

[AdrianKBL]

@petreze

v2.26.0-beta.2

[SimiHunjan]

@AdriMsM91 do you encounter the same issue on testnet or is this on mainnet only?

[AdrianKBL]

@SimiHunjan Both

[SimiHunjan]

Thanks going to check in with the devops team as well. Appreciate your patience.

[nickpoorman]

@AdriMsM91 can you try the following?

x1 Mint Transactions with 10 NFTs

Instead do:

x1 Mint Transactions with 1 NFT

I suspect, during high network usage, you have a greater chance of hitting a throttle due to the 10x amount of work required when minting 10x NFTs in a single transaction.

Although, this doesn't necessarily explain the 503 error code you are receiving.

[AdrianKBL]

Hello Nick,

I just finished the test 1 by 1 in Mainnet instead of batches of 10 and I got the same error in my 37 NFT minted.

grpc-status:14
<html><body><h1>503 Service Unavailable</h1>%0ANo server is available to handle this request.%0A</body></html>%0A

[AdrianKBL]

Hello, good morning.

Any advances on this? We have all our customers opening tickets daily about this problem. They can't even mint more than 200-300 NFTs without getting this error.

Thanks.

[SimiHunjan]

@AdriMsM91 we are still investigating the issue. It is not reproducible on testnet at the moment. In the meantime, is the app able to retry when it receives the 503 unavailable errors? To confirm, you are now only receiving 503 errors?

[ochikov]

Hey @AdriMsM91 I can reproduce the error, but when this error occurs, we are switching to another proxy so there aren't failing transactions because of this error on my side. We are working to investigate why we are receiving this error. It's only reproducible on mainnet.

[AdrianKBL]

I would need to do an incredible work around to be able to retry. Right now we're only getting 503 errors since last updated.

[ochikov]

It's not connected with the update and the new version I think. On test net is not reproducible. I think that the proxies has high load. Why you need to implement work around? The SDK is retrying itself. Also you can set the max retries to different number from the default value (10).

[AdrianKBL]

With the SDK retrying by himself (10 times) isn't enough and the error appears anyways as we saw already. With a work around, I was meaning by generating a new Client, saving the Metadatas, giving the user the Metadatas left once It failed...etc etc.

[ochikov]

I am pretty sure that we are hitting some throttles of the proxies. Will continue to investigate.

[SimiHunjan]

@AdriMsM91 have you tried to remove node3 and node4 from the client node list? Something like:

const client = Client.forNetwork({
    "https://node02.swirldslabs.com:443": new AccountId(5),
    // remove "https://node03.swirldslabs.com:443": new AccountId(6),
    // remove "https://node04.swirldslabs.com:443": new AccountId(7),
    "https://node05.swirldslabs.com:443": new AccountId(8),
    "https://node06.swirldslabs.com:443": new AccountId(9),
    "https://node07.swirldslabs.com:443": new AccountId(10),
    "https://node08.swirldslabs.com:443": new AccountId(11),
    "https://node09.swirldslabs.com:443": new AccountId(12),
    "https://node10.swirldslabs.com:443": new AccountId(13),
    "https://node11.swirldslabs.com:443": new AccountId(14),
    "https://node12.swirldslabs.com:443": new AccountId(15),
    "https://node13.swirldslabs.com:443": new AccountId(16),
    "https://node14.swirldslabs.com:443": new AccountId(17),
    "https://node16.swirldslabs.com:443": new AccountId(19),
    "https://node17.swirldslabs.com:443": new AccountId(20),
    "https://node18.swirldslabs.com:443": new AccountId(21),
    "https://node19.swirldslabs.com:443": new AccountId(22),
    "https://node20.swirldslabs.com:443": new AccountId(23),
    "https://node21.swirldslabs.com:443": new AccountId(24),
    "https://node22.swirldslabs.com:443": new AccountId(25),
    "https://node23.swirldslabs.com:443": new AccountId(26),
    "https://node24.swirldslabs.com:443": new AccountId(27),
    "https://node25.swirldslabs.com:443": new AccountId(28),
    "https://node26.swirldslabs.com:443": new AccountId(29),
    });

Both those nodes are down and are not receiving requests https://status.hedera.com/. In the image you shared above it looks like it is also highlighting those two nodes.

[SimiHunjan]

hey @AdriMsM91 ! I tried to mint an NFT collection using the app for testnet, but seems I need to buy a pass or own a specific NFT.

When users sign the transaction to mint NFTs do they sign a single transaction that goes to one node that is randomly selected by the SDK or do they sign multiple transactions with different node IDs set randomly by the SDK and then the app will try submitting transactions to each of those nodes until one is successful?

[petreze]

Hello @AdriMsM91
We just released another beta version - 2.26.0-beta.3. Here we fix an issue where an unhealthy node is added again in the healthy node list without decrementing the unhealthy nodes count. This happens after the backoff time of an unhealthy node expires
Could you please test with the following version and give a feedback?

[AdrianKBL]

@petreze Thanks, seems like it's fixed! Minted 2,000 NFTs in Mainnet without any problem.

[petreze]

@AdriMsM91
So glad to hear that !