Knowledge tool appears to rescrape webpages that have already been scraped in the same session

[sheng-liang]

I asked knowledge to scrape and ingest intel.com.

At some point it scraped 547 pages and ingested 30 pages. Then it somehow got into rescraping URLs that already have been scraped earlier, only to find out the timestamp has not changed. See the attached figure. This should not happen. It should not rescrape any URL until after we scraped the whole site and want to do a sync.

[cjellick]

This was in main and likely due to the server restarting because main otto upgrades on every commit. We can hopefully improve scraping to "pick up where it left off" more gracefully.

[cjellick]

In the event the process does get restarted, perhaps we can have some checkpointing mechanism.

[StrongMonkey]

The fix is merged, but @sangee2004 I am not sure if you have a good way to kill this. Maybe restarting server in the middle of scraping?

[sangee2004]

@StrongMonkey I was able to reproduce this issue by restarting otto server when website sync was in progress which was tracked in #1154 . I will test this use case .

[sangee2004]

@StrongMonkey When I tried the following steps, I see the sync getting blocked after server restart for a long time ( 7 minute delay)

1. Create an agent with knowledge file from coral.org
2. When sync is in progress , add knowledge file from one of the synced folders

When sync and ingestion is still in progress , redeploy otto srerver.

One otto server is restarted , noticed that syncing of files is stalled for a very long time like 7 minutes in my case , in this state .

Agent - https://test.acornlabs.com/admin/agents/a1bsd6m

After about 7 minutes , I see the sync proceed further

Reopening this issue to see why we take 7 minutes for syncing to start progress in this case.

[sangee2004]

This issue is addressed when testing with the latest build.
There is no big delay in sync process to start happening when server was restarted when sync/ingestion was in progress.