Scraper is unreliable: Some pages are not found at times.

[juhamust]

Description

The number of pages and records the scraper finds and processes varies greatly.

Steps to reproduce

Reproduce the issue by using the provided repository or see the screenshot
https://github.com/juhamust/docusaurus-typesense-search

Screenshot from subsequent runs:

Expected Behavior

The number of records should remain consistent.

Actual Behavior

Metadata

Running both server and scraper within Docker containers running in MacOS. The target website is quite a vanilla Docusaurus website. The Docusaurus website is built and served using Docusaurus.

Typesense Scraper Version: typesense/docsearch-scraper:0.11.0
Typesense Version: typesense/typesense:27.1