Exception in thread "main" java.lang.UnsupportedOperationException: JsonObject

[molitoris]

Hey @Meghdut-Mandal

A very cool tool!

I encounter a similar problem. It downloads the first two chapters. Afterwards, I receive a similar exception like @fabian-schmitt in #1.

The connection is estabilished and the first two chapters are loaded. Then it breaks down :(

packtpubscrapper-scrapper-1  | [pool-1-thread-1] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:4}] to db:27017
packtpubscrapper-scrapper-1  | Done loading  ch02lvl1sec02
packtpubscrapper-scrapper-1  | [...]
packtpubscrapper-scrapper-1  | Done loading  ch03lvl1sec11
packtpubscrapper-scrapper-1  | [...]
packtpubscrapper-scrapper-1  | Cached ch03lvl1sec12
packtpubscrapper-scrapper-1  | Cached ch03lvl1sec13
packtpubscrapper-scrapper-1  | Cached ch03lvl1sec14
packtpubscrapper-scrapper-1  | Cached ch03lvl1sec15
packtpubscrapper-scrapper-1  | Cached ch03lvl1sec16
packtpubscrapper-scrapper-1  | Cached ch03lvl1sec17
packtpubscrapper-scrapper-1  | Cached ch03lvl1sec18
packtpubscrapper-scrapper-1  | Cached ch03lvl1sec19
packtpubscrapper-scrapper-1  | Exception in thread "main" java.lang.UnsupportedOperationException: JsonObject
packtpubscrapper-scrapper-1  |  at com.google.gson.JsonElement.getAsString(JsonElement.java:179)
packtpubscrapper-scrapper-1  |  at Scrapper$loadBookPageContent$1.invokeSuspend(Scrapper.kt:108)
packtpubscrapper-scrapper-1  |  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:138)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:112)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:14)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:62)
packtpubscrapper-scrapper-1  |  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:138)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:112)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:14)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:62)
packtpubscrapper-scrapper-1  |  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:138)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:112)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:14)
packtpubscrapper-scrapper-1  |  at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:62)
packtpubscrapper-scrapper-1  |  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
packtpubscrapper-scrapper-1  |  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
packtpubscrapper-scrapper-1  |  at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284)
packtpubscrapper-scrapper-1  |  at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
packtpubscrapper-scrapper-1  |  at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
packtpubscrapper-scrapper-1  |  at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
packtpubscrapper-scrapper-1  |  at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
packtpubscrapper-scrapper-1  |  at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
packtpubscrapper-scrapper-1  |  at Scrapper.loadBookPageContent(Scrapper.kt:85)
packtpubscrapper-scrapper-1  |  at Scrapper.loadContent(Scrapper.kt:43)
packtpubscrapper-scrapper-1  |  at Scrapper.loadChapterSequential(Scrapper.kt:179)
packtpubscrapper-scrapper-1  |  at Scrapper.access$loadChapterSequential(Scrapper.kt:24)
packtpubscrapper-scrapper-1  |  at Scrapper$main$1$2$1.invokeSuspend(Scrapper.kt:157)
packtpubscrapper-scrapper-1  |  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
packtpubscrapper-scrapper-1  |  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
packtpubscrapper-scrapper-1  |  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
packtpubscrapper-scrapper-1  |  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
packtpubscrapper-scrapper-1  |  at java.base/java.lang.Thread.run(Unknown Source)

[Meghdut-Mandal]

The API might have changed. Will check once and let you know.

[molitoris]

I appreciate your help :)

[gaLDWoRE]

Experiencing same issue.

[oyale]

I am also experiencing the same problem.

I have noticed that caching works, so if you retry (Ctrl + c and docker-compose up) enough times, you will end up getting the book.

packtpubscrapper-scrapper-1  | Cached chXXXXXXXXX
packtpubscrapper-scrapper-1  | Requests made  0 with models.BookInfo@XXXXXXX
packtpubscrapper-scrapper-1  | Now converting to Epub
packtpubscrapper-scrapper-1  | Done

[luvpreetsingh]

same issue

EDIT: @oyale you are right!!! I retried like 30-40 times and it worked but I got another error while converting to epub

[bensh]

Seems to be a different issue for me.

packtpubscrapper-master-scrapper-1  |   at io.ktor.client.call.HttpClientCall.body(HttpClientCall.kt:92)
packtpubscrapper-master-scrapper-1  |   at Scrapper.loadBookInfo(Scrapper.kt:191)
packtpubscrapper-master-scrapper-1  |   at Scrapper.access$loadBookInfo(Scrapper.kt:24)
packtpubscrapper-master-scrapper-1  |   at Scrapper$loadBookInfo$1.invokeSuspend(Scrapper.kt)
packtpubscrapper-master-scrapper-1  |   at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
packtpubscrapper-master-scrapper-1  |   at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
packtpubscrapper-master-scrapper-1  |   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
packtpubscrapper-master-scrapper-1  |   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
packtpubscrapper-master-scrapper-1  |   at java.base/java.lang.Thread.run(Unknown Source)