[BOUNTY] Fix script responsible for preparing IPFS mirror

[lidel]

BOUNTY: $500 (how to claim?)

Summary

• We unpack ZIM and put it on IPFS as a regular HTML+JS+CSS website.
• Before its published we customize JS to a footer is attached to every page, informing reader that its unofficial wikipedia mirror
  • The build inserts custom footer into snapshots (Insert custom footer into snapshots #17, replacement logo and footer #15).
  • Existing scripts no longer work and need to be updated/redone before we start creating new snapshots (this is blocking Automate snapshot updates #58, Update en.wikipedia-on-ipfs.org #61, Update tr.wikipedia-on-ipfs.org #60)

TODO

Pick up where PR at #67 ended and update execute-changes.sh script to:

• Ensure there are no JS errors when pages are loaded
  • see Update tr.wikipedia-on-ipfs.org #60 (comment)
• Make it possible to navigate to other articles
  • ensure relative paths work
    • on https://ipfs.io/ipns/<cid>/wiki/
    • on https://<cid>.ipfs.dweb.link/wiki/
  • When unpacked with extract_zim, all pages are named ArticleName.html but they link to other article names without .html
    • idea: for every article, create ArticleName/index.html with a redirect page to ArticleName.html (similar to this)
• Custom footer needs to be appended to every page
• Update footer contents
  • add link to article snapshot at original Wikipedia
    • oldid= links can be found in page sources, for example:
      href="https://tr.wikipedia.org/w/index.php?title=<title>&amp;oldid=<timestamp>"
  • add link to the source .zim file
    • for now it can be link at download.kiwix.org, in the future it will be .zim on IPFS
  • remove logos/buttons of centralized services
  • include information on takedown policy / contact (eg. if latest snapshot includes information removed in upstream wikipedia)
• Restore original Main Page
  • every wikipedia has a Main Page under different name
    • there are scripts to find out that name and fetch original for the right snapshot, see work started in bb9f48c
    • what needs to happen is to download original, fix it up to work locally and save it as /wiki/index.html

Acceptance Criteria

• PR with necessary changes is submitted and merged to this repo
• Script works and enables us to produce updated IPFS mirror of the latest Turkish snapshot: wikipedia_tr_all_maxi_2019-12.zim
  • CID of a demo output is provided

[jnthnvctr]

@lidel when this does work - would we be pinning this on our own IPFS node (or cluster)?

[lidel]

@jnthnvctr I believe we would pin to cluster like we did with older snapshot, but want to create a collaborative cluster, so we host a copy, but anyone can follow and contribute – see notes in #68

[RubenKelevra]

@lidel yes, we just need to work out implementation details. I would really like to get a static JavaScript implementation which can parse the Wikipedia markup language going, so we can seperate the data from the webpage and update them individually.

This would reduce the amount of storage required massively.

[parkan]

@kanej can you confirm that #70 is intending to close this issue/collect the bounty?

[kanej]

@parkan yes #70 is intending to close the issue and collect the bounty.

[lidel]

#70 is merged, thank you @kanej 👍 🚀

tr.wikipedia-on-ipfs.org continued in #60 (comment)

Update: @parkan reopened this so you can try doing the DAI setup

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 500.0 DAI (500.0 USD @ $1.0/DAI) attached to it as part of the ipfs fund.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $120,150.53 more funded OSS Work available on the Gitcoin Issue Explorer

[parkan]

@kanej see above! 💸

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

These users each claimed they can complete the work by 3 weeks, 5 days from now.
Please review their action plans below:

1) kanej has started work.

Rework the execute changes script into a node based script.

The script will convert the extracted zim directory into a workable website, appending the Distributed Wikipedia Mirror project notes to the bottom of each page.

Learn more on the Gitcoin Issue Details page.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work for 500.0 DAI (500.0 USD @ $1.0/DAI) has been submitted by:

1. @kanej

@parkan please take a look at the submitted work:

• PR by @kanej

---

• Learn more on the Gitcoin Issue Details page
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $120,577.75 more funded OSS Work available on the Gitcoin Issue Explorer

[gitcoinbot]

⚡️ A *Ipfs* Kudos has been sent to @kanej for this issue from @parkan. ⚡️ Nice work @kanej! Your Kudos has automatically been sent in the ETH address we have on file.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

The funding of 500.0 DAI (500.0 USD @ $1.0/DAI) attached to this issue has been approved & issued to @kanej.

• Learn more on the Gitcoin Issue Details page
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $99,191.86 more funded OSS Work available on the Gitcoin Issue Explorer

[parkan]

paid out! thank you @kanej, was the gitcoin experience smooth for you?

@lidel let's close this for real now 🙂

[kanej]

@parkan thanks. I had no problems, straight forward process, but I started with previous experience of ether and gitcoin.