Do I need a GPU on my Daemon/Storage Miner? #7629

hcm609 · 2021-11-15T23:39:20Z

hcm609
Nov 15, 2021

I have read many examples that a GPU is used on the machine that hosts the Market/Storage (lotus-miner) and main Lotus Daemon.
Was anyone able to avoid having a GPU on the Daemon/Market/Storage Miner if the machine he uses is quite powerful (no GPU though) and having no issues on the WindowPoSt?

Answered by stuberman

Dec 2, 2021

./lotus-bench sealing --sector-size=512MiB

results (v28) SectorSize:(536870912), SectorNumber:(1)
seal: addPiece: 1.438371558s (356 MiB/s)
seal: preCommit phase 1: 27.241093166s (18.8 MiB/s)
seal: preCommit phase 2: 5.56242225s (92.05 MiB/s)
seal: commit phase 1: 21.720827ms (23.02 GiB/s)
seal: commit phase 2: 7.530819516s (67.99 MiB/s)
seal: verify: 2.857278ms
unseal: 30.998674773s (16.52 MiB/s)

generate candidates: 4.842439ms (103.3 GiB/s)
compute winning post proof (cold): 847.707073ms
compute winning post proof (hot): 755.703106ms
verify winning post proof (cold): 20.286334ms
verify winning post proof (hot): 1.15645ms

compute window post proof (cold): 410.542677ms
compute window pos…

View full answer

stuberman · 2021-11-15T23:51:20Z

stuberman
Nov 15, 2021

You absolutely need a good GPU (Nvidia >8GB VRAM) on Lotus-Worker (C2 and PC2)
Lotus-Miner in order to win block rewards and perform PoST also needs a good GPU
Daemon and Market do not need.

5 replies

hcm609 Nov 16, 2021
Author

Thank you Stuberman.

For a lotus-miner (without GPU) I did a lotus-bench and here are the results:
Can you share me one of your lotus-Miner to understand the difference?
#lotus-bench sealing --sector-size=512MiB
results (v28) SectorSize:(536870912), SectorNumber:(1)
seal: addPiece: 1.736670812s (294.8 MiB/s)
seal: preCommit phase 1: 40.631669438s (12.6 MiB/s)
seal: preCommit phase 2: 13.784939529s (37.14 MiB/s)
seal: commit phase 1: 28.500149ms (17.54 GiB/s)
seal: commit phase 2: 37.186278325s (13.77 MiB/s)
seal: verify: 2.238998ms
unseal: 48.132575626s (10.64 MiB/s)

generate candidates: 355.437µ?s (1.374 TiB/s)
compute winning post proof (cold): 3.314640921s
compute winning post proof (hot): 3.254919879s
verify winning post proof (cold): 24.198857ms
verify winning post proof (hot): 2.154261ms

compute window post proof (cold): 1.256051343s
compute window post proof (hot): 1.246975362s
verify window post proof (cold): 12.730943ms
verify window post proof (hot): 1.630329ms

If we take the time above for the compute window post proof (hot): 1.246975362s and since this sector is 0.5GiB then the time this server would perform will be 64x1.246975362s=79.36s which is much lower than 30 s that we need to reply to the chain?

f8-ptrk Nov 17, 2021

you have 70 epochs, 35 minutes to compute a windowed post. but you cannot just multiply the times for a 512MiB sector to get a approximate time for a 32GiB sector.

it is possible without a GPU, sure - i wouldn't rely on it working out - especially if you do anything else on that server - and sealing counts as anyhting else. you'd basically need to stop everyhting every day for the wdPost/s to be sure they will be calculated in < 35 minutes (you actually have a few minutes less to have time to get the message on chain)

recommendation from me too: get a gpu

hcm609 Nov 18, 2021
Author

I was planning to have one server for the lotus daemon and the market and the storage only.
I run a test on this server and got the following results:
results (v28) SectorSize:(34359738368), SectorNumber:(1)

generate candidates: 14.932787ms (2.093 TiB/s)
compute winning post proof (cold): 9.788316319s
compute winning post proof (hot): 10.779173967s
verify winning post proof (cold): 257.439261ms
verify winning post proof (hot): 19.896124ms

compute window post proof (cold): 16m27.301253126s
compute window post proof (hot): 14m44.97797004s
verify window post proof (cold): 2.061472925s
verify window post proof (hot): 7.847024ms

Are the 35min-time needed the same as for the compute window post proof (cold) or (hot)?
Also the winning Post is the (cold) or (hot)?
I am asking the above just to see if I am very marginal.

LexLuthr Dec 2, 2021
Collaborator

@hcm609 Do you still need help with this or were you able to get the details you were looking for?

hcm609 Dec 2, 2021
Author

I am still waiting for an answer. The above are my stats. Does anyone with GPU have better results that the above?

stuberman · 2021-12-02T22:33:38Z

stuberman
Dec 2, 2021

./lotus-bench sealing --sector-size=512MiB

results (v28) SectorSize:(536870912), SectorNumber:(1)
seal: addPiece: 1.438371558s (356 MiB/s)
seal: preCommit phase 1: 27.241093166s (18.8 MiB/s)
seal: preCommit phase 2: 5.56242225s (92.05 MiB/s)
seal: commit phase 1: 21.720827ms (23.02 GiB/s)
seal: commit phase 2: 7.530819516s (67.99 MiB/s)
seal: verify: 2.857278ms
unseal: 30.998674773s (16.52 MiB/s)

generate candidates: 4.842439ms (103.3 GiB/s)
compute winning post proof (cold): 847.707073ms
compute winning post proof (hot): 755.703106ms
verify winning post proof (cold): 20.286334ms
verify winning post proof (hot): 1.15645ms

compute window post proof (cold): 410.542677ms
compute window post proof (hot): 403.66147ms
verify window post proof (cold): 8.143837ms
verify window post proof (hot): 1.411802ms

4 replies

stuberman Dec 2, 2021

This is using 2080 Ti

f8-ptrk Dec 2, 2021

no it doesn't

f8-ptrk Dec 2, 2021

512MiB sectors ignore GPUs

f8-ptrk Dec 2, 2021

at least in lotus-miner/worker.

can you run that again with BELLMAN_NO_GPU=1 to be absolutely sure?

stuberman · 2021-12-02T22:37:41Z

stuberman
Dec 2, 2021

Funny - but logs shows it is using GPU

2021-12-02T22:32:45.806Z INFO lotus-bench lotus-bench/main.go:348 computing winning post snark (hot)
2021-12-02T22:32:45.806 INFO filcrypto::proofs::api > generate_winning_post: start
2021-12-02T22:32:45.806 INFO filecoin_proofs::api::winning_post > generate_winning_post:start
2021-12-02T22:32:45.806 INFO filecoin_proofs::caches > trying parameters memory cache for: WINNING_POST[536870912]
2021-12-02T22:32:45.806 INFO filecoin_proofs::caches > found params in memory cache for WINNING_POST[536870912]
2021-12-02T22:32:45.806 INFO storage_proofs_core::compound_proof > vanilla_proofs:start
2021-12-02T22:32:45.830 INFO storage_proofs_core::compound_proof > vanilla_proofs:finish
2021-12-02T22:32:45.832 INFO storage_proofs_core::compound_proof > snark_proof:start
2021-12-02T22:32:45.832 INFO bellperson::groth16::prover > Bellperson 0.18.0 is being used!
2021-12-02T22:32:45.904 INFO bellperson::groth16::prover > synthesis time: 72.088703ms
2021-12-02T22:32:45.904 INFO bellperson::groth16::prover > starting proof timer
2021-12-02T22:32:45.911 INFO bellperson::gpu::locks > GPU is available for FFT!
2021-12-02T22:32:45.911 INFO bellperson::gpu::program > Using kernel on CUDA.
2021-12-02T22:32:45.914 INFO bellperson::gpu::fft > FFT: 1 working device(s) selected.
2021-12-02T22:32:45.914 INFO bellperson::gpu::fft > FFT: Device 0: GeForce RTX 2080 Ti
2021-12-02T22:32:45.914 INFO bellperson::domain > GPU FFT kernel instantiated!
2021-12-02T22:32:45.993 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2021-12-02T22:32:45.993 INFO bellperson::gpu::program > Using kernel on CUDA.
2021-12-02T22:32:45.995 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0.01)
2021-12-02T22:32:45.995 INFO bellperson::gpu::multiexp > Multiexp: Device 0: GeForce RTX 2080 Ti (Chunk-size: 7488999)
2021-12-02T22:32:45.995 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2021-12-02T22:32:46.561 INFO bellperson::groth16::prover > prover time: 656.438748ms
2021-12-02T22:32:46.562 INFO storage_proofs_core::compound_proof > snark_proof:finish
2021-12-02T22:32:46.562 INFO filecoin_proofs::api::winning_post > generate_winning_post:finish
2021-12-02T22:32:46.562 INFO filcrypto::proofs::api > generate_winning_post: finish
2021-12-02T22:32:46.562 INFO filcrypto::proofs::api > verify_winning_post: start
2021-12-02T22:32:46.562 INFO filecoin_proofs::api::winning_post > verify_winning_post:start
2021-12-02T22:32:46.562 INFO filecoin_proofs::caches > trying parameters memory cache for: WINNING_POST[536870912]-verifying-key
2021-12-02T22:32:46.562 INFO filecoin_proofs::caches > no params in memory cache for WINNING_POST[536870912]-verifying-key
2021-12-02T22:32:46.562 INFO storage_proofs_core::parameter_cache > parameter set identifier for cache: FallbackPoSt::PublicParams{sector_size: 536870912, challenge_count: 1, sector_count: 66}
2021-12-02T22:32:46.562 INFO storage_proofs_core::parameter_cache > ensuring that all ancestor directories for: "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-7d739b8cf60f1b0709eeebee7730e297683552e4b69cab6984ec0285663c5781.vk" exist
2021-12-02T22:32:46.562 INFO storage_proofs_core::parameter_cache > checking cache_path: "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-7d739b8cf60f1b0709eeebee7730e297683552e4b69cab6984ec0285663c5781.vk" for verifying key
2021-12-02T22:32:46.569 INFO storage_proofs_core::parameter_cache > read verifying key from cache "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-7d739b8cf60f1b0709eeebee7730e297683552e4b69cab6984ec0285663c5781.vk"
2021-12-02T22:32:46.582 INFO filecoin_proofs::api::winning_post > verify_winning_post:finish
2021-12-02T22:32:46.582 INFO filcrypto::proofs::api > verify_winning_post: finish
2021-12-02T22:32:46.582 INFO filcrypto::proofs::api > verify_winning_post: start
2021-12-02T22:32:46.582 INFO filecoin_proofs::api::winning_post > verify_winning_post:start
2021-12-02T22:32:46.582 INFO filecoin_proofs::caches > trying parameters memory cache for: WINNING_POST[536870912]-verifying-key
2021-12-02T22:32:46.582 INFO filecoin_proofs::caches > found params in memory cache for WINNING_POST[536870912]-verifying-key
2021-12-02T22:32:46.583 INFO filecoin_proofs::api::winning_post > verify_winning_post:finish
2021-12-02T22:32:46.583 INFO filcrypto::proofs::api > verify_winning_post: finish
2021-12-02T22:32:46.583Z INFO lotus-bench lotus-bench/main.go:388 computing window post snark (cold)
2021-12-02T22:32:46.583 INFO filcrypto::proofs::api > generate_window_post: start
2021-12-02T22:32:46.583 INFO filecoin_proofs::api::window_post > generate_window_post:start
2021-12-02T22:32:46.583 INFO filecoin_proofs::caches > trying parameters memory cache for: Window_POST[536870912]
2021-12-02T22:32:46.583 INFO filecoin_proofs::caches > no params in memory cache for Window_POST[536870912]
2021-12-02T22:32:46.583 INFO storage_proofs_core::parameter_cache > parameter set identifier for cache: FallbackPoSt::PublicParams{sector_size: 536870912, challenge_count: 10, sector_count: 2}
2021-12-02T22:32:46.583 INFO storage_proofs_core::parameter_cache > ensuring that all ancestor directories for: "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0cfb4f178bbb71cf2ecfcd42accce558b27199ab4fb59cb78f2483fe21ef36d9.params" exist
2021-12-02T22:32:46.583 INFO storage_proofs_core::parameter_cache > checking cache_path: "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0cfb4f178bbb71cf2ecfcd42accce558b27199ab4fb59cb78f2483fe21ef36d9.params" for parameters
2021-12-02T22:32:46.590 INFO storage_proofs_core::parameter_cache > read parameters from cache "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0cfb4f178bbb71cf2ecfcd42accce558b27199ab4fb59cb78f2483fe21ef36d9.params"
2021-12-02T22:32:46.590 INFO storage_proofs_core::compound_proof > vanilla_proofs:start
2021-12-02T22:32:46.595 INFO storage_proofs_core::compound_proof > vanilla_proofs:finish
2021-12-02T22:32:46.596 INFO storage_proofs_core::compound_proof > snark_proof:start
2021-12-02T22:32:46.596 INFO bellperson::groth16::prover > Bellperson 0.18.0 is being used!
2021-12-02T22:32:46.693 INFO bellperson::groth16::prover > synthesis time: 96.936878ms
2021-12-02T22:32:46.693 INFO bellperson::groth16::prover > starting proof timer
2021-12-02T22:32:46.697 INFO bellperson::gpu::locks > GPU is available for FFT!
2021-12-02T22:32:46.697 INFO bellperson::gpu::program > Using kernel on CUDA.
2021-12-02T22:32:46.700 INFO bellperson::gpu::fft > FFT: 1 working device(s) selected.
2021-12-02T22:32:46.700 INFO bellperson::gpu::fft > FFT: Device 0: GeForce RTX 2080 Ti
2021-12-02T22:32:46.700 INFO bellperson::domain > GPU FFT kernel instantiated!
2021-12-02T22:32:46.729 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2021-12-02T22:32:46.729 INFO bellperson::gpu::program > Using kernel on CUDA.
2021-12-02T22:32:46.734 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0.01)
2021-12-02T22:32:46.734 INFO bellperson::gpu::multiexp > Multiexp: Device 0: GeForce RTX 2080 Ti (Chunk-size: 7488999)
2021-12-02T22:32:46.734 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2021-12-02T22:32:46.994 INFO bellperson::groth16::prover > prover time: 300.107505ms
2021-12-02T22:32:46.994 INFO storage_proofs_core::compound_proof > snark_proof:finish
2021-12-02T22:32:46.994 INFO filecoin_proofs::api::window_post > generate_window_post:finish
2021-12-02T22:32:46.994 INFO filcrypto::proofs::api > generate_window_post: finish
2021-12-02T22:32:46.994Z INFO lotus-bench lotus-bench/main.go:396 computing window post snark (hot)
2021-12-02T22:32:46.994 INFO filcrypto::proofs::api > generate_window_post: start
2021-12-02T22:32:46.994 INFO filecoin_proofs::api::window_post > generate_window_post:start
2021-12-02T22:32:46.994 INFO filecoin_proofs::caches > trying parameters memory cache for: Window_POST[536870912]
2021-12-02T22:32:46.994 INFO filecoin_proofs::caches > found params in memory cache for Window_POST[536870912]
2021-12-02T22:32:46.994 INFO storage_proofs_core::compound_proof > vanilla_proofs:start
2021-12-02T22:32:46.999 INFO storage_proofs_core::compound_proof > vanilla_proofs:finish
2021-12-02T22:32:47.000 INFO storage_proofs_core::compound_proof > snark_proof:start
2021-12-02T22:32:47.001 INFO bellperson::groth16::prover > Bellperson 0.18.0 is being used!
2021-12-02T22:32:47.096 INFO bellperson::groth16::prover > synthesis time: 95.618692ms
2021-12-02T22:32:47.096 INFO bellperson::groth16::prover > starting proof timer
2021-12-02T22:32:47.100 INFO bellperson::gpu::locks > GPU is available for FFT!
2021-12-02T22:32:47.100 INFO bellperson::gpu::program > Using kernel on CUDA.
2021-12-02T22:32:47.103 INFO bellperson::gpu::fft > FFT: 1 working device(s) selected.
2021-12-02T22:32:47.103 INFO bellperson::gpu::fft > FFT: Device 0: GeForce RTX 2080 Ti
2021-12-02T22:32:47.103 INFO bellperson::domain > GPU FFT kernel instantiated!
2021-12-02T22:32:47.134 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2021-12-02T22:32:47.134 INFO bellperson::gpu::program > Using kernel on CUDA.
2021-12-02T22:32:47.138 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0.01)
2021-12-02T22:32:47.138 INFO bellperson::gpu::multiexp > Multiexp: Device 0: GeForce RTX 2080 Ti (Chunk-size: 7488999)
2021-12-02T22:32:47.138 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2021-12-02T22:32:47.397 INFO bellperson::groth16::prover > prover time: 300.999377ms
2021-12-02T22:32:47.397 INFO storage_proofs_core::compound_proof > snark_proof:finish
2021-12-02T22:32:47.397 INFO filecoin_proofs::api::window_post > generate_window_post:finish
2021-12-02T22:32:47.397 INFO filcrypto::proofs::api > generate_window_post: finish

6 replies

f8-ptrk Dec 2, 2021

i try to get nemo to say something about this that is definitive

cryptonemo Dec 6, 2021
Collaborator

What is the question, specifically? It's certainly possible to run with CPU only, but it's not recommended, as using GPU is faster.

stuberman Dec 6, 2021

I think @f8-ptrk question was whether 512MiB sectors use GPU. in the ./lotus-bench sealing --sector-size=512MiB test

stuberman Dec 6, 2021

My logs show that it does use the GPU (see logs above)

cryptonemo Dec 6, 2021
Collaborator

Yes, any/all supported sector sizes can use GPU.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do I need a GPU on my Daemon/Storage Miner? #7629

{{title}}

Replies: 3 comments 15 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Do I need a GPU on my Daemon/Storage Miner? #7629

Replies: 3 comments · 15 replies

hcm609 Nov 16, 2021 Author

hcm609 Nov 18, 2021 Author

LexLuthr Dec 2, 2021 Collaborator

hcm609 Dec 2, 2021 Author

cryptonemo Dec 6, 2021 Collaborator

cryptonemo Dec 6, 2021 Collaborator

Replies: 3 comments 15 replies

hcm609 Nov 16, 2021
Author

hcm609 Nov 18, 2021
Author

LexLuthr Dec 2, 2021
Collaborator

hcm609 Dec 2, 2021
Author

cryptonemo Dec 6, 2021
Collaborator

cryptonemo Dec 6, 2021
Collaborator