Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(novelfire): add retry logic when loading chapters on large novels #1449

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

JohnRamberger
Copy link
Contributor

Issue

#1420

Description

  • After investigation found that NovelFire is ratelimiting after loading ~30 chapter pages (~3000 chapters).
  • To fix: Added retry logic when loading chapters in chunks.

Performance Testing

Nine Star Hegemon Body Arts (5638 chapters across 57 pages)

[chunkSize=2] Parsed 5638 chapters in 22.85 seconds
[chunkSize=3] Parsed 5638 chapters in 20.34 seconds
[chunkSize=4] Parsed 5638 chapters in 19.00 seconds
->[chunkSize=5] Parsed 5638 chapters in 16.48 seconds
[chunkSize=6] Parsed 5638 chapters in 17.01 seconds
[chunkSize=7] Parsed 5638 chapters in 17.91 seconds
[chunkSize=10] Parsed 5638 chapters in 27.41 seconds

My Vampire System (2545 chapters across 26 pages)

[chunkSize=2] Parsed 2545 chapters in 5.78 seconds
[chunkSize=3] Parsed 2545 chapters in 4.60 seconds
[chunkSize=4] Parsed 2545 chapters in 3.44 seconds
->[chunkSize=5] Parsed 2545 chapters in 2.97 seconds
[chunkSize=6] Parsed 2545 chapters in 2.78 seconds
[chunkSize=7] Parsed 2545 chapters in 2.62 seconds
[chunkSize=10] Parsed 2545 chapters in 2.48 seconds

@nyagami
Copy link
Member

nyagami commented Jan 23, 2025

Why dont implement PagePlugin instead. It sends only one request at once

@JohnRamberger
Copy link
Contributor Author

Ah, didn't know this was a thing until now, I didn't see any documentation. Will use and probably update the docs. Ty!

@Palloxin
Copy link
Contributor

Palloxin commented Jan 23, 2025

Cime in: if it is like lightnovelpub plugin paging, please don't. It's not mature, some uncomfortable dynamics with it.

For example u can't export all pages in one epun, u cant "Jump to" chapters in other pages, u cant browse chaps on the other page while on reading mode, you cant download all in ome (u have to visit each page and download all), and so on.

I can tell you: I avoid paged plugins pretty much, it's probably a common sentiment.

@JohnRamberger
Copy link
Contributor Author

I tend to agree with you, my own sentiment towards paged plugins is negative. Wondering about paginating the plugin vs increasing load times for novels with large chapter counts through retries. I'm leaning towards the second option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants