diff --git a/.github/workflows/docs-deploy.yml b/.github/workflows/docs-deploy.yml index 323a0cd..2148d1e 100644 --- a/.github/workflows/docs-deploy.yml +++ b/.github/workflows/docs-deploy.yml @@ -20,10 +20,10 @@ jobs: run: | npm ci cd website && npm install && cd - - - name: Build docs - run: | - npm run docs - cd website && npm run build && cd - + - name: Build documentation + run: cd website && npm run build && cd - + - name: Build API documentation + run: npm run docs:api - name: Zip docs run: | mv docs website/build/docs/api @@ -32,11 +32,11 @@ jobs: - name: Copy docs to Toolforge uses: garygrossgarten/github-action-scp@dba89d858413133b6e35da91a3c989cc8cde3ad4 with: - local: /home/runner/work/mwn/mwn/website/build/docs.zip + local: ${{github.workspace}}/website/build/docs.zip remote: /data/project/mwn/docs.zip host: login.toolforge.org - username: ${{ secrets.TOOLFORGE_USERNAME }} - privateKey: ${{ secrets.TOOLFORGE_PRIVATE_KEY }} + username: ${{secrets.TOOLFORGE_USERNAME}} + privateKey: ${{secrets.TOOLFORGE_PRIVATE_KEY}} rmRemote: true - name: Unzip uses: garygrossgarten/github-action-ssh@2b10f41b5a33808f6d24eafd253296766308b7c4 @@ -48,5 +48,5 @@ jobs: unzip /data/project/mwn/docs.zip -d . rm /data/project/mwn/docs.zip host: login.toolforge.org - username: ${{ secrets.TOOLFORGE_USERNAME }} - privateKey: ${{ secrets.TOOLFORGE_PRIVATE_KEY }} + username: ${{secrets.TOOLFORGE_USERNAME}} + privateKey: ${{secrets.TOOLFORGE_PRIVATE_KEY}} diff --git a/README.md b/README.md index 7c46dd6..54589a6 100644 --- a/README.md +++ b/README.md @@ -6,459 +6,52 @@ [![Coverage Status](https://coveralls.io/repos/github/siddharthvp/mwn/badge.svg?branch=master)](https://coveralls.io/github/siddharthvp/mwn?branch=master) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com) -**Quick links: [Getting Started](https://mwn.toolforge.org/docs/setting-up-mwn) — [GitHub](https://github.com/siddharthvp/mwn) — [NPM](https://www.npmjs.com/package/mwn) — [User Documentation](https://mwn.toolforge.org/) — [API Documentation](https://mwn.toolforge.org/docs/api/classes/mwn.html)** +**Quick links: [Getting Started](https://mwn.toolforge.org/docs/getting-started) — [GitHub](https://github.com/siddharthvp/mwn) — [NPM](https://www.npmjs.com/package/mwn) — [User Documentation](https://mwn.toolforge.org/) — [API Documentation](https://mwn.toolforge.org/docs/api/classes/mwn.html)** **Mwn** is a modern and comprehensive MediaWiki bot framework for Node.js, originally adapted from [mwbot](https://github.com/Fannon/mwbot). -Mwn works with both JavaScript and TypeScript. It is created with a design philosophy of allowing bot developers to easily and quickly write bot code, without having to deal with the MediaWiki API complications and idiosyncrasies such as logins, tokens, maxlag, query continuations and error handling. Making raw API calls is also supported for complete flexibility where needed. The [axios](https://www.npmjs.com/package/axios) library is used for HTTP requests. - -Mwn uses [JSON with formatversion 2](https://www.mediawiki.org/wiki/API:JSON_version_2#Using_the_new_JSON_results_format) by default. Use of the legacy formatversion is not recommended. Note that [Special:ApiSandbox](https://en.wikipedia.org/wiki/Special:ApiSandbox) uses formatversion=1 by default, so if you're testing API calls using ApiSandbox be sure to set the correct formatversion there, otherwise the output will be formatted differently. +Mwn works with both JavaScript and TypeScript. It is created with a design philosophy of allowing bot developers to easily and quickly write bot code, without having to deal with the MediaWiki API complications and idiosyncrasies such as logins, tokens, maxlag, query continuations and error handling. [Making raw API calls](https://mwn.toolforge.org/docs/direct-api-calls) is also supported for complete flexibility where needed. Mwn uses [JSON with formatversion 2](https://www.mediawiki.org/wiki/API:JSON_version_2#Using_the_new_JSON_results_format) by default. The [axios](https://www.npmjs.com/package/axios) library is used for HTTP requests. Versioning: while mwn is in version 0, changes may be made to the public interface with a change in the minor version number. -Complete API documentation is available **[here](https://mwn.toolforge.org/docs/api/classes/mwn.html)** ([alternative link](https://tools-static.wmflabs.org/mwn/docs/classes/mwn.html)). In addition to the MediaWiki Action API, the library also provides methods to talk to the Wikimedia EventStreams API, the ORES API, Pageviews API and WikiWho API. - -Amongst the major highlights are `batchOperation` and `seriesBatchOperation` which allow you run a large number of tasks with control over concurrency and sleep time between tasks. Failing actions are automatically retried. +In addition to the MediaWiki Action API, the library also provides methods to talk to the Wikimedia EventStreams API, the ORES API, Pageviews API and WikiWho API. This library uses mocha for tests and has extensive test coverage covering all commonly used code paths. Testing is automated using a CI workflow on Github Actions. -### Setup - To install, run `npm install mwn`. [![Download stats](https://nodei.co/npm/mwn.png?downloads=true&downloadRank=true)](https://nodei.co/npm/mwn/) -Or obtain the latest development copy: - -```sh -git clone https://github.com/siddharthvp/mwn.git -cd mwn -npm install # install dependencies -npm run build # generate JS files from TS. The type errors raised can be ignored. -``` - -#### Node.js compatibility - -Mwn is written in TypeScript v4. The repository contains JavaScript files compiled to CommonJS module system for ES2018 target, which corresponds to Node 10.x. - -If your bot is hosted on [Toolforge](https://tools.wmflabs.org/), note that the system version of node there is v8.11.1. You can install a more recent version of node to your home directory using a version manager like [nvm](https://github.com/nvm-sh/nvm) or [n](http://npmjs.com/package/n). Like: - -```sh -npm install n # install a node version manager -export N_PREFIX=~ -./node_modules/n/bin/n lts # get the latest LTS version of node -export PATH=~/bin:$PATH -``` +### Documentation +Up-to-date documentation is [**hosted on Toolforge**](https://mwn.toolforge.org/). -Check that your `.profile` or `.bashrc` file includes the line `PATH="$HOME/bin:$PATH"`, so that the path includes your home directory every time you open the shell. - -If you're using mwn for a Toolforge webservice, use the [Kubernetes backend](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Node.js#Kubernetes_Configuration) which provides node v10. Mwn is not supported for the [legacy Grid Engine backend](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Node.js#Grid_Engine_Configuration) since it uses node v8.11.1. The [toolforge-node-app-base](https://github.com/siddharthvp/toolforge-node-app-base) template repository can quickly get you started with a basic web tool boilerplate. - -#### MediaWiki compatibility - -Mwn is written for and tested on the latest version of MediaWiki used on WMF wikis. Support for MW versions going back to 1.34 is planned. - -#### Set up a bot password or OAuth credentials - -Mwn supports authentication via both [BotPasswords](https://www.mediawiki.org/wiki/Manual:Bot_passwords) and [OAuth 1.0a](https://www.mediawiki.org/wiki/OAuth/Owner-only_consumers). Use of OAuth is recommended as it does away the need for separate API requests for logging in, and is also more secure. - -Bot passwords, however, are a bit easier to set up. To generate one, go to the wiki's [Special:BotPasswords](https://en.wikipedia.org/wiki/Special:BotPasswords) page. +API documentation (automatically generated via [typedoc](https://npmjs.com/package/typedoc)) is also available at https://mwn.toolforge.org/docs/api. ### Features -**Handling multiple users and wikis**: Mwn can seamlessly work with multiple bot users signed into the same wiki, and multiple wikis at the same time. You just have to create multiple bot instances – each one representing a wiki + user. Each bot instance uses an isolated cookie jar; all settings are also isolated. - -**Maxlag**: The default [maxlag parameter](https://www.mediawiki.org/wiki/Manual:Maxlag_parameter) used by mwn is 5 seconds. Requests failing due to maxlag will be automatically retried after pausing for a duration specified by `maxlagPause` (default 5 seconds). A maximum of `maxRetries` will take place (default 3). - -**Token handling**: [Tokens](https://www.mediawiki.org/wiki/API:Tokens) are automatically fetched as part of `mwn.init()` or `bot.login()` or `bot.getTokensAndSiteInfo()`. Once retrieved, they are stored in the bot state and can be reused any number of times. If any API request fails due to an expired or missing token, the request is automatically retried after fetching a new token. `bot.getTokens()` can be used to refresh the token cache, though mwn manages this, so you'd never need to explicitly use that. - -**Retries**: Mwn automatically retries failing requests `bot.options.maxRetries` times (default: 3). This is useful in case of connectivity resets and the like. As for errors raised by the API itself, note that MediaWiki generally handles these at the response level rather than the protocol level (they still emit a 200 OK response). Mwn will attempt retries for these errors based on the error code. For instance, if the error is `readonly` or `maxlag` , retry is done after a delay. If it's `assertuserfailed` or `assertbotfailed` (indicates a session loss), mwn will try to log in again and then retry. If it's `badtoken`, retry is done after fetching a fresh edit token. - -**Handling query continuation**: Mwn uses [asynchronous generators](https://javascript.info/async-iterators-generators), ([for await...of](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of) loops) to provide a very intuitive interface around MediaWiki API's [query continuation](https://www.mediawiki.org/wiki/API:Query#Example_4:_Continuing_queries). - -Use `bot.continuedQueryGen` everytime you want to fetch more results than what the API gives you in one go (usually 5000 results). `continuedQueryGen` automatically uses the continue parameters in the response to create and send new requests that retrieve data from where the previous response was cut off. - -The following example generates a list of all active users on the wiki (which may be more than 5000). - -```js -let activeusers = []; -for await (let json of bot.continuedQueryGen({ - action: 'query', - list: 'allusers', - auactiveusers: 1, - aulimit: 'max' -})) { - let users = json.query.allusers.map((user) => user.name); - activeusers = activeusers.concat(users); -} -``` - -Specialised derivatives exist to fulfill common needs: - -- `new bot.page('Page name').historyGen()` - fetch page history -- `new bot.page('Page name').logsGen()` - fetch page logs -- `new bot.category('Page name').membersGen()` - fetch category members -- `new bot.user('User name').contribsGen()` - fetch user contributions -- `new bot.user('User name').logsGen()` - fetch user logs - -Every method with a name that ends in `Gen` is an async generator. - -**Emergency shutoff**: Mwn exploits Node's asynchronous event loop to efficiently implement emergency shutoff. - -```js -bot.enableEmergencyShutoff({ - page: 'User:ExampleBot/shutoff', // The name of the page to check - intervalDuration: 5000, // check shutoff page every 5 seconds - condition: function (pagetext) { - if (pagetext !== 'running') { // function to determine whether the bot should continue to run or not - return false; // Example implementation: if some one changes the text to something - } else return true; // other than "running", let's decide to stop! - }, - onShutoff: function (pagetext) { // function to trigger when shutoff is activated - process.exit(); // let's just exit, though we could also terminate - } // any open connections, close files, etc. -}); -``` - -The rate of shutoff checks is not impacted by your actual editing rate, as it takes place separately in a setInterval() loop. Caution: this implementation has not been stress-tested. - -Shutoff once enabled can be disabled anytime, such as when you stop performing write operations and you're now just doing read operations. - -```js -bot.disableEmergencyShutoff(); -``` - -**Bot exclusion compliance**: Mwn's edit() method can be configured to respect {{nobots}} or equivalent. If the text of the page tests positive for the exclusionRegex you set in the bot options, edit will be aborted. - -```js -bot.setOptions({ - exclusionRegex: /\{\{nobots\}\}/i -}); -``` - -It's also possible to do this on a per-edit basis: - -```js -bot.edit( - 'Page name', - (rev) => { - // edit the page the way you want, in this lame example we're just appending lorem ipsum - return rev.content + 'lorem ipsum'; - }, - /* editConfig */ { - exclusionRegex: /\{\{nobots\}\}/i - } -); -``` - -Exclusion compliance is _not_ enabled by default. - -### Getting started - -**Note: More complete [documentation is on Toolforge](https://mwn.toolforge.org/).** - -Importing mwn: - -In JavaScript: - -```js -const { mwn } = require('mwn'); -``` - -Note: Prior to mwn v0.8.0, import was via `const mwn = require('mwn');` - -In TypeScript: - -```ts -import { mwn } from 'mwn'; -``` - -If you're migrating from mwbot, note that: - -- `edit` in mwbot is different from `edit` in mwn. You want to use `save` instead. -- If you were using the default formatversion=1 output format, set formatversion: 1 in the config options. - -Create a new bot instance: - -```js -const bot = await mwn.init({ - apiUrl: 'https://en.wikipedia.org/w/api.php', - - // Can be skipped if the bot doesn't need to sign in - username: 'YourBotUsername', - password: 'YourBotPassword', - - // Instead of username and password, you can use OAuth 1.0a to authenticate, - // if the wiki has Extension:OAuth enabled - OAuthCredentials: { - consumerToken: '16_DIGIT_ALPHANUMERIC_KEY', - consumerSecret: '20_DIGIT_ALPHANUMERIC_KEY', - accessToken: '16_DIGIT_ALPHANUMERIC_KEY', - accessSecret: '20_DIGIT_ALPHANUMERIC_KEY' - }, - - // Set your user agent (required for WMF wikis, see https://meta.wikimedia.org/wiki/User-Agent_policy): - userAgent: 'myCoolToolName 1.0 ([[link to bot user page or tool documentation]])', - - // Set default parameters to be sent to be included in every API request - defaultParams: { - assert: 'user' // ensure we're logged in - } -}); -``` - -This creates a bot instance, signs in and fetches tokens needed for editing. - -You can also create a bot instance synchronously (without using await): - -```js -const bot = new mwn({ - ...options -}); -``` - -This creates a bot instance which is not signed in. Then to authenticate, use `bot.login()` which returns a promise. If using OAuth, use `bot.initOAuth()` followed by `bot.getTokensAndSiteInfo()`. Note that `bot.initOAuth()` does not involve an API call. Any error in authentication will surface on running bot.getTokensAndSiteInfo(). - -The bot options can also be set using `setOptions` rather than through the constructor: - -```js -bot.setOptions({ - silent: false, // suppress messages (except error messages) - retryPause: 5000, // pause for 5000 milliseconds (5 seconds) on maxlag error. - maxRetries: 3 // attempt to retry a failing requests upto 3 times -}); -``` - -### Direct API calls - -The `request` method is for directly querying the API. See [mw:API](https://www.mediawiki.org/wiki/API:Main_page) for options. You can create and test your queries in [Special:ApiSandbox](https://www.mediawiki.org/wiki/Special:ApiSandbox). Be sure to set formatversion: 2 in the options for format=json! - -Example: get all images used on the article Foo - -```js -bot.request({ - action: 'query', - prop: 'images', - titles: 'Foo' -}).then((data) => { - return data.query.pages[0].images.map((im) => im.title); -}); -``` - -Mwn provides a great number of convenience methods so that you can avoid writing raw API calls, see the sections below. - -### Editing pages - -Edit a page. On edit conflicts, a retry is automatically attempted once. - -```js -bot.edit('Page title', (rev) => { - // rev.content gives the revision text - // rev.timestamp gives the revision timestamp - - let text = rev.content.replace(/foo/g, 'bar'); - - return { - // return parameters needed for [[mw:API:Edit]] - text: text, - summary: 'replacing foo with bar', - minor: true - }; -}); -``` - -Some more functions associated with editing pages: - -```js -// Save a page with the given content without loading it first. Simpler verion of `edit`. Does not offer any edit conflict detection. -await bot.save('Page title', 'Page content', 'Edit summary'); - -// Create a new page. -await bot.create('Page title', 'Page content', 'Edit summary'); - -// Post a new section to a talk page: -await bot.newSection('Page title', 'New section header', 'Section content', additionalOptions); -``` - -### Other basic operations - -Read the contents of a page: - -```js -await bot.read('Page title'); -``` - -Read a page along with metadata: - -```js -await bot.read('Page title', { - rvprop: ['content', 'timestamp', 'user', 'comment'] -}); -``` - -Read multiple pages using a single API call: - -```js -bot.read(['Page 1', 'Page 2', 'Page 3']).then((pages) => { - // pages[0], pages[1], pages[2] -}); -``` - -Delete a page: - -```js -await bot.delete('Page title', 'deletion log summary', additionalOptions); -``` - -Move a page along with its subpages: - -```js -await bot.move('Old page title', 'New page title', 'move summary', { - movesubpages: true, - movetalk: true -}); -``` - -Parse wikitext (see [API:Parse](https://www.mediawiki.org/wiki/API:Parsing_wikitext) for additionalOptions) - -```js -await bot.parseWikitext('Input wikitext', additonalOptions); -``` - -Parse the contents of a given page - -```js -await bot.parseTitle('Page name', additionalOptions); -``` - -Upload a file from your system to the wiki: - -```js -await bot.upload('File title', '/path/to/file', 'comment', customParams); -``` - -Download a file from the wiki: - -```js -await bot.download('File:File name.jpg', 'Downloaded file name.jpg'); // 2nd param defaults to the on-wiki name if unspecified -``` - -Creating a page object opens up further possibilities: - -```js -let page = new bot.page('Main Page'); -``` - -See [list of methods available on page object](https://mwn.toolforge.org/docs/api/interfaces/mwnpage.html). - -[Files](https://mwn.toolforge.org/docs/api/interfaces/mwnfile.html) and [categories](https://mwn.toolforge.org/docs/api/interfaces/mwncategory.html) have their own subclasses that add a few additional methods. - -### Working with titles - -Titles can be represented as objects created using the class constructor on `mwn`, as: (`bot` is the mwn object) - -```js -let title = new bot.title('Wikipedia:Articles for deletion'); - -// The constructor throws if given an illegal page name. To avoid this the static method can be used instead -title = bot.title.newFromText('Wikipedia:Articles for deletion'); // same effect - -// Can also construct titles from namespace number and the page name -title = new bot.title('Aritcles for deletion', 4); - -title.getMainText(); // 'Articles for deletion' -title.getNamespaceId(); // 4 - -title = bot.title.newFromText('catEgorY:living people'); // titles will be normalised! -title.toText(); // 'Category:Living people' -``` - -The API of this class is based on that of [mw.Title](https://doc.wikimedia.org/mediawiki-core/master/js/#!/api/mw.Title) in the on-site JS interface. See [full list of methods](https://mwn.toolforge.org/docs/api/interfaces/mwntitle.html). - -### Working with wikitext - -Mwn can be used for parsing wikitext: - -```js -let wkt = new bot.wikitext('This is some wikitext with [[links]] and {{templates|with=params}}.'); - -wkt.parseTemplates(); // -> [Template {wikitext: '{{templates|with=params}}', parameters: [ Parameter {name: 'with', value: 'params', wikitext: '|with=params'}] ], name: 'Templates' }] - -// This requires the bot object to have the namespace data of the wiki available. -// Either the bot should be logged in, or run bot.getSiteInfo() -wkt.parseLinks(); // populates wkt.links, wkt.files, wkt.categories -wkt.links; // -> [{ wikitext: '[[links]]', target: Title { namespace: 0, title: 'links', fragment: null }, displaytext: 'links'}] -``` - -In addition: - -- `bot.wikitext.parseTable(wikitext)` parses simple tables without fancy markup; will throw on unparsable input - -### Bulk processing methods - -**continuedQuery / continuedQueryGen**: Handles query continuation. See "Handling query continuations" in [Features section above](#user-content-features). - -continuedQuery returns a promised resolved with the array of all individual API response. - -Use of `continuedQueryGen` is recommended since continuedQuery will fetch the results of all the API calls before it begins to do anything with the results. `continuedQueryGen` gets the result of each API call and processes them one at a time. - -**massQuery / massQueryGen**: MediaWiki sets a limit of 500 (50 for non-bots) on the number of pages that can be queried in a single API call. To query more than that, `massQuery` or `massQueryGen` can be used. This splits the page list into batches of 500 and sends individual queries and returns a promise resolved with the array of all individual API call responses. - -Example: get the protection status of a large number of pages: - -```js -bot.massQuery({ - action: 'query', - format: 'json', - prop: 'info', - titles: ['Page1', 'Page2', 'Page1300'], // array of page names - inprop: 'protection' -}) // 2nd parameter is taken as 'titles' by default - .then((jsons) => { - // jsons is the array of individual JSON responses. - }); -``` +- **Handling multiple users and wikis**: Mwn can seamlessly work with multiple bot users signed into the same wiki, and multiple wikis at the same time. You just have to create multiple bot instances – each one representing a wiki + user. Each bot instance uses an isolated cookie jar; all settings are also isolated. -Any errors in the individual API calls will not cause the entire massQuery to fail, but the data at the array index corresponding to that API call will be error object. +- **Token handling**: [Tokens](https://www.mediawiki.org/wiki/API:Tokens) are automatically fetched as part of `mwn.init()` or `bot.login()` or `bot.getTokensAndSiteInfo()`. Once retrieved, they are stored in the bot state and can be reused any number of times. If any API request fails due to an expired or missing token, the request is automatically retried after fetching a new token. `bot.getTokens()` can be used to refresh the token cache, though mwn manages this, so you'd never need to explicitly use that. + +- **Maxlag**: The default [maxlag parameter](https://www.mediawiki.org/wiki/Manual:Maxlag_parameter) used by mwn is 5 seconds. Requests failing due to maxlag will be automatically retried after pausing for a duration specified by `maxlagPause` (default 5 seconds). A maximum of `maxRetries` will take place (default 3). -massQueryGen is the generator equivalent that yields each API response as when they're received. +- **Retries**: Mwn automatically retries failing requests `bot.options.maxRetries` times (default: 3). This is useful in case of connectivity resets and the like. As for errors raised by the API itself, note that MediaWiki generally handles these at the response level rather than the protocol level (they still emit a 200 OK response). Mwn will attempt retries for these errors based on the error code. For instance, if the error is `readonly` or `maxlag` , retry is done after a delay. If it's `assertuserfailed` or `assertbotfailed` (indicates a session loss), mwn will try to log in again and then retry. If it's `badtoken`, retry is done after fetching a fresh edit token. -#### Batch operations +- **Handling query continuation**: Mwn uses [asynchronous generators](https://javascript.info/async-iterators-generators), ([for await...of](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of) loops) to provide a very intuitive interface around MediaWiki API's [query continuation](https://www.mediawiki.org/wiki/API:Query#Example_4:_Continuing_queries). See [**Handling query continuation**](https://mwn.toolforge.org/docs/handling-query-continuation). -Perform asynchronous tasks (involving API usage) over a number of pages (or other arbitrary items). `batchOperation` uses a default concurrency of 5. Customise this according to how expensive the API operation is. Higher concurrency limits could lead to more frequent API errors. +- [**Parsing wikitext**](https://mwn.toolforge.org/docs/working-with-wikitext): Mwn provides methods for common wikitext parsing needs (templates, links, and simple tables). -- `batchOperation(pageList, workerFunction, concurrency, maxRetries)`: The `workerFunction` must return a promise. +- [**Titles**](https://mwn.toolforge.org/docs/working-with-titles): Work with page titles with the very same API as the in-browser [mw.Title](https://doc.wikimedia.org/mediawiki-core/master/js/#!/api/mw.Title) that userscript/gadget developers are familiar with. -```js -bot.batchOperation( - pageList, - (page, idx) => { - // do something with each page - // the index of the page in pageList is available as the 2nd argument - // return a promise in the end - }, - /* concurrency */ 5, - /* retries */ 2 -); -``` +- [**Emergency shutoff**](https://mwn.toolforge.org/docs/emergency-shutoff) -- `bot.seriesBatchOperation(pageList, workerFunction, sleepDuration, retries)` can be used for serial operations, with a sleep duration between each task (default 5 seconds). +- [**Bot exclusion compliance**](https://mwn.toolforge.org/docs/exclusion-compliance) -```js -bot.seriesBatchOperation( - pageList, - (page, idx) => { - // do something with each page - // the index of the page in pageList is available as the 2nd argument - // return a promise in the end - }, - 5000, - 2 -); // set the sleep duration in milliseconds as the third parameter, max number of retries for each action is set as the 4th parameter -``` +- [**Batch operations**](https://mwn.toolforge.org/docs/bulk-processing#batch-operations): Perform a large number of tasks (like page edits) with control over the concurrency (default 5). Failing actions can be set to automatically retry. -Note that `seriesBatchOperation` with delay=0 is same as `batchOperation` with concurrency=1. +### Contributing +Patches are very much welcome. See https://mwn.toolforge.org/docs/developing for instructions. -## Licensing +### Licensing -Mwn is released under [GNU Lesser General Public License](https://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License) (LGPL) v3.0, since it borrows quite a bit of code from MediaWiki core (GPL v2). LGPL is a more permissive variant of the more popular GNU GPL. Unlike GPL, LPGL _allows_ the work to be used as a library in software not released under GPL-compatible licenses, and even in proprietary software. However, any derivatives of this library should also be released under LGPL or another GPL-compatible license. +Mwn is released under [GNU Lesser General Public License](https://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License) (LGPL) v3.0, since it borrows quite a bit of code from MediaWiki core (GPL v2). LGPL is a more permissive variant of GNU GPL. Unlike GPL, it enables this library to be used in software not released under GPL-compatible licenses, and even in proprietary software. However, any derivatives of this library should be released under an GPL-compatible license (like LGPL). diff --git a/package.json b/package.json index 6d6df6c..af69ac6 100644 --- a/package.json +++ b/package.json @@ -17,7 +17,8 @@ "test": "nyc --reporter=lcov --reporter=text mocha tests/", "coveralls": "nyc report --reporter=text-lcov | coveralls", "test:ts": "ts-mocha -p tsconfig.json tests/ts/*", - "docs": "node scripts/generate-docs.js" + "docs": "cd website && npm start", + "docs:api": "node scripts/generate-docs.js" }, "engines": { "node": ">=10" diff --git a/tests/test_wiki.js b/tests/test_wiki.js index 3a3dffb..0955a8c 100644 --- a/tests/test_wiki.js +++ b/tests/test_wiki.js @@ -7,11 +7,6 @@ let bot = new mwn({ ...loginCredentials.account1, userAgent: 'mwn (https://github.com/siddharthvp/mwn)', }); -let bot2 = new mwn({ - silent: true, - userAgent: 'mwn (https://github.com/siddharthvp/mwn)', - ...loginCredentials.account2, -}); async function setup() { // Switching to BotPassword authentication due to OAuth being unreliable in CI due to @@ -28,4 +23,4 @@ async function setup() { async function teardown() {} // Export everything -module.exports = { mwn, bot, bot2, log, expect, assert, sinon, setup, teardown }; +module.exports = { mwn, bot, log, expect, assert, sinon, setup, teardown }; diff --git a/tsconfig.json b/tsconfig.json index 4d30a97..71ff1cc 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -11,5 +11,6 @@ "include": ["./src/**/*"], "ts-node": { "logError": true - } + }, + "compileOnSave": true } diff --git a/website/docs/1-setting-up-mwn.md b/website/docs/1-getting-started.md similarity index 72% rename from website/docs/1-setting-up-mwn.md rename to website/docs/1-getting-started.md index d2bc23a..38f995f 100644 --- a/website/docs/1-setting-up-mwn.md +++ b/website/docs/1-getting-started.md @@ -2,7 +2,7 @@ sidebar_position: 1 --- -# Setting up Mwn +# Getting started Installation: `npm install mwn` @@ -25,10 +25,16 @@ import { mwn } from 'mwn'; If you're migrating from mwbot, note that: -- `edit` in mwbot is different from `edit` in mwn. You want to use `save` instead. -- If you were using the default formatversion=1 output format, set formatversion: 1 in the config options. +- `edit` in mwbot is different from `edit` in mwn. You want to use `save` instead. +- If you were using the default formatversion=1 output format, set formatversion: 1 in the config options. -Create a new bot instance: +#### Set up a bot password or OAuth credentials + +Mwn supports authentication via both [BotPasswords](https://www.mediawiki.org/wiki/Manual:Bot_passwords) and [OAuth 1.0a](https://www.mediawiki.org/wiki/OAuth/Owner-only_consumers). Use of OAuth is recommended as it does away the need for separate API requests for logging in, and is also more secure. + +Bot passwords, however, are a bit easier to set up. To generate one, go to the wiki's [Special:BotPasswords](https://en.wikipedia.org/wiki/Special:BotPasswords) page. + +#### Initializing bot object ```js const bot = await mwn.init({ diff --git a/website/docs/14-developing.md b/website/docs/14-developing.md new file mode 100644 index 0000000..999f26e --- /dev/null +++ b/website/docs/14-developing.md @@ -0,0 +1,45 @@ +# Developing + +Obtain the latest development copy: + +```sh +git clone https://github.com/siddharthvp/mwn.git +cd mwn +npm install # install dependencies +``` + +Build: run `npm run build`. This generates the JS files in `build` from the TypeScript sources in `src`. The type errors raised can be ignored. You may want to configure your IDE to do this automatically on every code change. + +### Tests + +Mwn has a comprehensive test suite that uses [mocha](https://www.npmjs.com/package/mocha) runner and [chai](https://www.npmjs.com/package/chai) for expectations. Tests should be added for all bug fixes and feature changes. There are three types of tests: +1. Tests that don't require a wiki: `npm run test:nowiki` +2. Tests that are based on [testwiki](https://test.wikipedia.org/): `npm run test:testwiki` + +You need to setup botpassword login credentials for testwiki to run these tests. Create a `.auth.js` file in repository root, with this structure: + +```json +{ + "username": "", + "password": "" +} +``` +No edits (or any other type of write action) will be done on testwiki. The credentials are only for reading content. + +3. Tests that require a local wiki: `npm run test:localwiki` + +For these, you need to spin up a local MediaWiki installation: + - Ensure that you have docker and docker-compose installed. If you're using Windows, you need to use WSL 2 and follow the steps at https://docs.docker.com/desktop/windows/wsl/ to link Docker Desktop to WSL. + - Run `npm run setuplocalwiki` + +If all goes well, you should now have a wiki up and running at [http://localhost:8080](http://localhost:8080). Sign in with username `Wikiuser` and password `wikipassword` (See [the setup.sh script](https://github.com/siddharthvp/mwn/blob/master/tests/docker/setup.sh) for other setup details.) + +**Troubleshooting**: If your local MW install fails with an error that a db table doesn't exist, try changing the sleep duration in https://github.com/siddharthvp/mwn/blob/master/tests/docker/main.sh#L2 to a higher value. The MariaDB database doesn't start accepting connections until it has initialised (time needed for that could be more on certain systems). + +All tests except the ones that depend on testwiki will also be run automatically via GitHub Actions when you create a PR, so no worries if you're having trouble with local MW installation. + +### Documentation + +When adding new features, please also update the corresponding documentation in `website/docs`. Documentation can be previewed by running `npm run docs` which starts a server at [http://localhost:3000](http://localhost:3000) that updates the rendered docs live as they are edited. + +The docs are hosted on https://mwn.toolforge.org and generated via [Docusaurus](https://docusaurus.io/). Build and deployment take place via GitHub Actions whenever they're updated in the master branch. diff --git a/website/docs/5-direct-api-calls.md b/website/docs/5-direct-api-calls.md index 71cdc03..0744003 100644 --- a/website/docs/5-direct-api-calls.md +++ b/website/docs/5-direct-api-calls.md @@ -1,6 +1,8 @@ # Direct API calls -The `request` method is for directly querying the API. See [mw:API](https://www.mediawiki.org/wiki/API:Main_page) for options. You can create and test your queries in [Special:ApiSandbox](https://www.mediawiki.org/wiki/Special:ApiSandbox). Be sure to set formatversion: 2 in the options for format=json! +The `request` method is for directly querying the API. See [mw:API](https://www.mediawiki.org/wiki/API:Main_page) for options. + +You can create and test your queries in [Special:ApiSandbox](https://www.mediawiki.org/wiki/Special:ApiSandbox). However, note that ApiSandbox uses json formatversion=1 by default whereas Mwn defaults to formatversion=2. So you should set formatversion: 2 in the format=json options, otherwise the output can be formatted differently for certain API endpoints. Use of the legacy formatversion is not recommended. Example: get all images used on the article Foo @@ -14,7 +16,7 @@ bot.request({ }); ``` -Mwn provides a great number of convenience methods so that you can avoid writing raw API calls, see the sections below. +Mwn provides a great number of convenience methods so that you can avoid writing raw API calls, see the later sections. #### Raw web requests diff --git a/website/src/pages/index.js b/website/src/pages/index.js index adea55d..541717f 100644 --- a/website/src/pages/index.js +++ b/website/src/pages/index.js @@ -16,7 +16,7 @@ function HomepageHeader() {
+ to="/docs/getting-started"> Mwn Tutorial - 5min ⏱️