Quick fixes that need to be done

[blackforestboi]

So there are few things that need to be done in order to have a more stable version:

• Revert to old search implementation (@arpitgogia)

Sorry @arpitgogia that we have to undo your work here, but we experienced PouchDB to be way to slow. Be aware that there are also changes I made

• The download process oftentimes stops at pages that have Virus warnings or other kind of interruptions. We need to make sure that there is a time out that skips a page in case it takes more than, say 1 minute to process. (@Chaitya62)

• Fix the problem that we also experience with the WebMemex, that with web apps, only the first visited page is really indexed.

Maybe we can take parts of your implementation @gastonche ?

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[gastonche]

I think my implementation can be used, with regards to point 3.

[blackforestboi]

@gastonche great!

Do you have time to make a PR for that?

[gastonche]

I am not yet done with the issue i was working on but sure i can make time for this. I just need directions on how to go about it here
@oliversauter

[blackforestboi]

Ok, work on WebMemex is more important.

This is the file where the capturing process starts: https://github.com/WorldBrain/Research-Engine/blob/master/src/js/content.js

[gastonche]

Okay I'll set it up and quickly make a PR since i will be faster and have implemented it already.

[Chaitya62]

@oliversauter In which js file exactly are you handling the downloads?

[blackforestboi]

@Chaitya62 Here is the file and the function that starts it all: https://github.com/WorldBrain/Research-Engine/blob/master/src/js-browserify/download_urls.js#L169

[Chaitya62]

@oliversauter Hey I have figured out a different approach wanted to ask you before I implement it
checkout this I think there must be an api which will return true or false or something similar What if I just make a call to that before starting the download ?

[blackforestboi]

Thanks @Chaitya62 for doing this research. This will be helpful for the WebMemex implementation of the download process.

But we need a more general approach for this, because there are many cases where the download is interrupted or stops.
We just need to make sure the download process completely finishes.

[Chaitya62]

@oliversauter Okay I ll find someother approach can you suggest any method so that I can reproduce the error? and also the master branch has the latest code right ?

[blackforestboi]

Yes, the master branch has the latest code.

I could not reproduce the error so far, but heard from a lot of users that it happened.
In the case of a virus warning(screenshot), the download process halts and does not resume, if the message is put away.
We just need to make sure, that in case of a longer pause (more than 1 minute) the download continues with the next url in any case. No matter what caused the interuption.

[blackforestboi]

thanks to @gastonche and his PR #111 we could get rid of the webapp problem
Thanks to @arpitgogia and his PR #107 we have the old search implementation back in the master branch.