Add research menu and relevant docs

[jinhojang6]

Handle issue #155

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs-waku-org	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 14, 2024 3:06pm

[fryorcraken]

Did you just copy pasted the files?

It might be better to use git submodules or do some scripting.

[jinhojang6]

@fryorcraken I've experimented with using submodules and scripts for Docusaurus, but I've found these methods to be somewhat complicated to maintain.

Do we have similar cases like this? If not, let me take the submodule approach but I'm uncertain if everyone is familiar with using submodules.

FYI, Below JS script works to fetch the files but "https://api.github.com/repos/" has a rate limit.

const https = require('https');
const fs = require('fs');
const path = require('path');

async function fetchFromGitHub(url, callback) {
    https.get(url, { headers: { 'User-Agent': 'Node.js' } }, (res) => {
        let data = '';

        res.on('data', (chunk) => {
            data += chunk;
        });

        res.on('end', () => {
            callback(null, JSON.parse(data));
        });
    }).on('error', (err) => {
        callback(err, null);
    });
}

async function fetchDirectoryContents(dirUrl, basePath, prefixToRemove) {
    fetchFromGitHub(dirUrl, async (err, files) => {
        if (err) {
            console.error('Error fetching files:', err.message);
            return;
        }

        for (const file of files) {
            const relativePath = file.path.replace(new RegExp(`^${prefixToRemove}`), '');
            const filePath = path.join(basePath, relativePath);

            if (file.type === 'file') {
                await downloadAndSaveFile(file.download_url, filePath);
            } else if (file.type === 'dir') {
                await fetchDirectoryContents(file.url, basePath, prefixToRemove); // Recursive call for subdirectories
            }
        }
    });
}

async function downloadAndSaveFile(url, filePath) {
    const fullFilePath = path.join(__dirname, filePath);

    https.get(url, (res) => {
        const directory = path.dirname(fullFilePath);

        // Ensure the directory exists
        fs.mkdirSync(directory, { recursive: true });

        const fileStream = fs.createWriteStream(fullFilePath);
        res.pipe(fileStream);

        fileStream.on('finish', () => {
            fileStream.close();
            console.log('Downloaded and saved:', filePath);
        });
    }).on('error', (err) => {
        console.error('Error downloading file:', err.message);
    });
}

const repositories = [
    {
        baseUrl: 'https://api.github.com/repos/waku-org/nwaku/contents/docs/benchmarks',
        baseSavePath: '/docs/research/benchmarks/',
        prefixToRemove: 'docs/benchmarks/'
    },
    {
        baseUrl: 'https://api.github.com/repos/waku-org/research/contents/docs',
        baseSavePath: '/docs/research/research-and-studies/',
        prefixToRemove: 'docs/'
    }
];

repositories.forEach(repo => {
    fetchDirectoryContents(repo.baseUrl, repo.baseSavePath, repo.prefixToRemove);
});

[fryorcraken]

I can see 3 ways of doing things:

1. git submodules
2. A build script that pulls latest master at build time
3. copy (manually or script) files locally with some manual action from developer and push them to repo (what is done now).

[jinhojang6]

The script above is intended for use as a build script. (Option 2) I think that incorporating submodules might complicate maintenance. If acceptable, I propose adding the script above to the repository.

[jinhojang6]

Added the script : https://github.com/waku-org/docs.waku.org/pull/157/files#diff-aac0d1b75493663edefcca04603a6378d9199e3a8902a07a225c398e3b7c0a52

and updated package.json and README

[jinhojang6]

@fryorcraken Can you please confim if the current "build script" way is okay? You can run yarn build to fetch fils from the 2 repos.

[fryorcraken]

I'll try locally later. @LordGhostX please also review.

[LordGhostX]

I find this approach appealing and it works very fine. My only comment is: can we make all page headings title capitalized?

"incentivization" -> "Incentivization"

"nwaku Benchmarks" -> "Nwaku Benchmarks"

Also, the docs site uses British English, so "Incentivization" should be written as "Incentivisation", running yarn check:spell should throw errors because of this

[fryorcraken]

"incentivization" -> "Incentivization"

---
title: Incentivization
---

Should be added to the incentivization.md file in the research repo cc @jm-clius

Also, the docs site uses British English, so "Incentivization" should be written as "Incentivisation", running yarn check:spell should throw errors because of this

The issue is that this needs to be fixed upstream in the research repo.

@jm-clius how do you feel about adding a spellcheck in the research repo? it helps with typo but does add an extra step.

We preferably would want to do that directly in the research repo so it does not block the PR in the docs repo.

Same question for @Ivansete-status re nwaku repo.

Current cspell error output:

/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:49:24 - Unknown word (pubsubtopic)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:50:86 - Unknown word (pubsubtopic)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:69:25 - Unknown word (jmeter)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:80:9 - Unknown word (jmeter)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:83:111 - Unknown word (jmeter)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:85:9 - Unknown word (jmeter)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:89:180 - Unknown word (jmeter)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:95:54 - Unknown word (analyzed)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:112:98 - Unknown word (queryc)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:112:247 - Unknown word (wakudev)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:112:260 - Unknown word (statusim)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:158:98 - Unknown word (queryc)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:158:247 - Unknown word (wakudev)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:158:260 - Unknown word (statusim)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:196:47 - Unknown word (chronos)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:198:360 - Unknown word (libpqis)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:213:164 - Unknown word (Conn)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:213:282 - Unknown word (messageindex)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:213:312 - Unknown word (storedat)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:213:326 - Unknown word (pubsubtopic)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/benchmarks/postgres-adoption.md:220:87 - Unknown word (wakudev)
40/42 ./docs/research/research-and-studies/incentivization.md 73.87ms X
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:2:8 - Unknown word (Incentivization)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:7:11 - Unknown word (incentivization)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:7:69 - Unknown word (behavior)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:15:3 - Unknown word (Incentivization)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:16:4 - Unknown word (Incentivization)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:18:17 - Unknown word (incentivization)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:53:100 - Unknown word (misbehavior)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:57:72 - Unknown word (Filecoin)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:57:82 - Unknown word (IPFS)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:83:86 - Unknown word (behavior)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:84:62 - Unknown word (behavior)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:103:14 - Unknown word (incentivization)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:120:54 - Unknown word (txid)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:127:85 - Unknown word (txid)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:128:32 - Unknown word (txid)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:152:24 - Unknown word (txid)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:156:9 - Unknown word (txid)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:163:80 - Unknown word (Hopr)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:172:106 - Unknown word (deanonymize)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:186:17 - Unknown word (honors)
/home/fryorcraken/src/waku-org/docs.waku.org/docs/research/research-and-studies/incentivization.md:227:14 - Unknown word (incentivization)

If cspell can be added to nwaku and research, then the matching cspell config files should be pulled in too:

https://cspell.org/docs/getting-started/#command-lint--spell-checking

Configuration file to use. By default cspell looks for cspell.json in the current directory.

So that once spelling is sorted in nwaku/research repo, it will work here.

[fryorcraken]

@jm-clius if you are able to do the same as waku-org/nwaku#2383 then we could get it shipped!

[jm-clius]

Done in waku-org/research#83
Have fixed the incentivisation doc by adding a title and changing to British spelling. I've also added a cspell list of spell exceptions.

[fryorcraken]

@jinhojang6 @LordGhostX . Hanno and Ivan have added cspell checks. Can you now review and see if we can merge?

[LordGhostX]

@jinhojang6 @LordGhostX . Hanno and Ivan have added cspell checks. Can you now review and see if we can merge?

@fryorcraken we're good to merge now 👍

@jinhojang6 I added fs.rmSync to remove the research directory before downloading the remote files (to handle any file structure changes)