Revisit Image Worker

[vikiival]

Currently the worker is not in the best condition and throws a lot of errors.
I would revisit it's architecture, pros, cons what can we do better

cc @preschian

[preschian]

Hi there. I would love to improve it. But could you please provide a list of the errors? How can I improve it if I don't know the errors? 😅

[vikiival]

if I don't know the errors?

1. I suffer CORS a lot for beta envs

Unless I do not open it in prod image is never loaded

https://canary.kodadot.xyz/stmn/collection/67

2. Every new content is ultra slow

• use nft-storage, rmrk for

3. make hints for image

tell the worker to fetch from this-gateway this ipfs /hint/?gateway=nft-storage
or /nft-storage/ipfs/

[preschian]

1. I suffer CORS a lot for beta envs

I thought it should be fixed with this kodadot/nft-gallery#6448 (comment)

2. Every new content is ultra slow

Yes, because we use different gateways between uploading and serving the content. Even if we hit a dedicated gateway directly (without image-workers) we face the same problem. lets say we upload to nftstorage, but in the client, we hit ipfs content to dedicated gateway filebase. Every new content will face the same issue with this. because different gateway takes time to resolve that ipfs content

How to improve that

1. use a single gateway for uploading and serving the ipfs content. afaik, nftstorage doesn't have a dedicated gateway. so, use filebase or infura when uploading to the ipfs
2. this one also possible if we want immediately available after uploading with direct-upload: Image worker is super slow #39 (comment)

fyi, I already give the same response in here #53 (comment)

3. make hints for image

noted, looks feasible

[preschian]

oh, I also already give the response here #39 (comment)

now we have 3 same issues

• Revisit Image Worker #132
• Rework image gateway #53
• Image worker is super slow #39

[niklasp]

Having issues with the image not showing up on a new collection for ~12h https://kodadot.xyz/stmn/collection/70

I am first creating the collection via

and then

const ipfsIdentifier = `ipfs://ipfs/${config.collectionConfig.metadataCid}`;
    apiKusamaAssetHub.tx.nfts.setCollectionMetadata(
      config.collectionConfig.id,
      ipfsIdentifier
    )

[vikiival]

@niklasp which gateway do you use to publish the ipfs cid?

[niklasp]

Pinata via their sdk

[preschian]

oohh, I think we have 2 cases here

1. the content is slow after minting through kodadot.xyz platform. This is the one I talk about here Revisit Image Worker #132 (comment)
2. the content is slow after minting outside kodadot.xyz platform. In this case, need to figure out how to improve this. Need one endpoint, something like /upload to upload IPFS content to our CFI/R2 and then our indexer hit that endpoint with send the ipfs data 🤔 maybe use ipfs sdk directly instead of checking the ipfs content from the gateway

[vikiival]

So fast draft

We should split this service into two separated workers

1.Gateway

User will access it via KodaDot and should return response as fast as possible.
This will work like this:

1. check cache if present return
2. check image store if present return redirect (to cf-image, or r2 link) + set cache
3. if none above fetch head on n-gateways (return redirect to fastest) + set short cache

2. Storage

two ways to access it:

• GET ipfs/* try to fetch content from either cfi/r2 if not present return 404 for gateway and (probably perform async retrieval)
• GET :gw/ipfs/* tell worker to retrieve content via this gateway.

[jsidorenko]

https://kodadot.xyz/stmn/collection/91
the images for NFTs are not loading returning the 500 error.
I've checked on random image from that collection https://ipfs.io/ipfs/QmaGn7c9h1rJyKxuSQTscXYWQnHx4AyjiK6pVFzfhnHM2M and its filesize is 15 Mb (could be the reason)

[preschian]

kodadot.xyz/stmn/gallery/91
the images for NFTs are not loading returning the 500 error.

Hi, thanks for reporting

That page return null from our indexer

NFT with id 91 not found. Fallback to RPC Node

I've checked on random image from that collection ipfs.io/ipfs/QmaGn7c9h1rJyKxuSQTscXYWQnHx4AyjiK6pVFzfhnHM2M and its filesize is 15 Mb (could be the reason)

we already handle big size image
ipfs (15MB): https://ipfs.io/ipfs/QmaGn7c9h1rJyKxuSQTscXYWQnHx4AyjiK6pVFzfhnHM2M
our workers (30kB): https://image.w.kodadot.xyz/ipfs/QmaGn7c9h1rJyKxuSQTscXYWQnHx4AyjiK6pVFzfhnHM2M

[jsidorenko]

Sorry, my bad, updated the link, should be https://kodadot.xyz/stmn/collection/91

[preschian]

Sorry, my bad, updated the link, should be kodadot.xyz/stmn/collection/91

yes, correct that one for collection pages 🙏

[preschian]

1.Gateway

User will access it via KodaDot and should return response as fast as possible. This will work like this:

1. check cache if present return
2. check image store if present return redirect (to cf-image, or r2 link) + set cache
3. if none above fetch head on n-gateways (return redirect to fastest) + set short cache

2. Storage

two ways to access it:

• GET ipfs/* try to fetch content from either cfi/r2 if not present return 404 for gateway and (probably perform async retrieval)

anyway @vikiival for this section is what we already have on image worker. see my comments here #53 (comment)

GET :gw/ipfs/* tell worker to retrieve content via this gateway.

for this one, what is the difference between hitting image worker first instead hit directly to ipfs gateway?

• image-w.kodadot.xyz/cf-ipfs/ipfs/cid
• cloudflare-ipfs.com/ipfs/cid

[vikiival]

for this one, what is the difference between hitting image worker first instead hit directly to ipfs gateway?

Because squid can for example make hints which gw to use.
The difference is this way it will now from which gw to take content and save.

[vikiival]

for this section is what we already have on image worker. see my comments here

1. R2 returns content not public url as you mentioned before.

2. Retrieval is awaited.

[preschian]

Because squid can for example make hints which gw to use. The difference is this way it will now from which gw to take content and save.

hhmm, I still need to understand. Can you elaborate more on what you expect with the endpoint and the problem with the public gateway? We already can do that with a public gateway, is it?

• R2 returns content not public url as you mentioned before.
• Retrieval is awaited.

oohh. the context of what I mean with public URLs is from our UI side. so, our UI can hit between image-w.kodadot.xyz/content?original=true or hit r2 URL directly stockpile.kodadot.xyz/content to get original content

on our image-worker side, we already return r2 content for non-CFI content. you can see one of the example sections here

workers/image/src/index.ts

Lines 106 to 117 in ba891cc

	// else, render r2 object and cache it
	const headers = new Headers();
	object.writeHttpMetadata(headers);
	headers.set('Access-Control-Allow-Origin', '*');
	headers.set('etag', object.httpEtag);
	response = new Response(object.body, {
	headers,
	});
	response.headers.append('Cache-Control', `s-maxage=${CACHE_MONTH}`);
	c.executionCtx.waitUntil(cache.put(cacheKey, response.clone()));

[jsidorenko]

Because squid can for example make hints which gw to use.

@vikiival any ideas on how they detect this?

[vikiival]

Can you elaborate more on what you expect with the endpoint and the problem with the public gateway? We already can do that with a public gateway, is it?

Yes,

We have a indexer, that is indexing nfts,
From Indexer I fetch metadata, so I know urls for image and animation_url
I would like to tell the gateway "Hey, I have new urls that should be stored on cfi/r2 beacause user will need that

Because squid can for example make hints which gw to use.

Why gw? because with dummy approach we know who minted majority of nfts and which gw they use so image worker does not blindly use infura if I/We know that image can be retrieved ultra fast via nft-storage

[preschian]

Why gw? because with dummy approach we know who minted majority of nfts and which gw they use so image worker does not blindly use infura if I/We know that image can be retrieved ultra fast via nft-storage

did you mean like this? something like hitting multiple endpoints from the indexer side?

before	after
- image-w.kodadot.xyz/ipfs/cid - nftstorage.link/ipfs/cid	- image-w.kodadot.xyz/ipfs/cid - image-w.kodadot.xyz/nftstorage/ipfs/cid

I still don't understand the difference in the "after" column. can you explain what the problem is in the "before" column?

because we can still face rate limits problem even if we wrap nft-storage to our image workers, right?