Fix remaining dead links and add check_link to CI (#322)

What does this PR do?

I forget to check link like this: <a href="/core/1/...">

This PR fix remaining dead links and updates the script

I also added the script to travis to check for dead links before merging

.travis.yml

@@ -21,12 +21,20 @@ env:
     - secure: "K1Hf0b0hgRRkIbudmJZCz96ZeObzRJsC4hUZ7N1aolAKkqRcb1HeKkWwD6Ql/kp2shnQ+QQhPVDlvJSDORJ1D+vZlDiYn8fYSjgcXLC1yfkzXkbezLQGZdyTGjpRyyv2vRt1JHtvLIXLrAg4INWyR2mQR/vW8lMdJkVcXnGu+CIdvncnQfYD6axVqWmqCgElEtCOZ3RUxxm2CHSqy7Caf9hF217lxAHORW6QEtfu86WvRoiIX08W9BQpNKvbdafiu0myVTnZUkn+w0ZhTGaS+nTd7AmIbPw8wQsRbm7ex9BTR4PpZMQOE/Aqtm5T7kEbpMK+oXRKIv2Y40YKTDaC27NQl12fZ0ORR6Gttm9DqXXACW8G5F87FkYJUSC8vo8UJFJSJhAKtzsqJRY5rIAr/tSFGwtxhhE4XV2eFC25zaNJsRXhzTyDB+OF9O0EUS2YgEZVfr3cxmUIDimKOsKP918zbjvBtLrZ1KJEs/1yR05J423CZ64T0RzLJ7HATzLt7Wo/nlzij73UglntxZlXWkhmZUGkT/UeNvTE+aPbfam+BWI2Y5v3qrxvtZ9IyOhSIVAsxi+/siRjv256pJnFLgpoGX8hH6rRP0TC3ri+8Sv/xV9ZxiUgUHhOhkUB57Uv1zHEOLR23vMRU1o8y4oA16bckc2SJ3SWrFXKT4qujgI="
 
 stages:
+  - name: Build
+  - name: Lint
   - name: Tests
   - name: Deployment
     if: type != pull_request AND branch =~ ^master|3-dev$
 
 jobs:
   include:
+    - stage: Lint
+      name: Lint and check for dead links
+      language: ruby
+      script:
+      - gem install typhoeus
+      - ruby check_link.rb -p src/
 
     # -------------------------------------------------------------------------
     # Snippets tests

check_link.rb

@@ -1,11 +1,29 @@
-require 'byebug'
 require 'json'
 require 'uri'
 require 'typhoeus'
 require 'optparse'
+require 'set'
 
 class LinkChecker
-  INTERNAL_LINK_REGEXP = /\[[\.\w\s\-]+\]\(([\w\/\-\#]*)\)/
+  INTERNAL_LINK_REGEXPS = [
+    /\[[:\.\w\s\-]+\]\((\/[\w\/\-\#]*)\)/,
+    /<a href="(\/[\w\/\-\#]*)">/
+  ]
+
+  IGNORED_EXTERNAL_LINKS = [
+    'http://kuzzle:7512',
+    'http://localhost',
+    'http://<',
+    'http://elasticsearch',
+    'http:head',
+    'http:options',
+    'http:post',
+    'http:put',
+    'http:get',
+    'http:delete',
+    'http:patch',
+    'http://...'
+  ]
 
   attr_reader :internal, :external
 
@@ -16,8 +34,8 @@ def initialize(options)
 
     @hydra = Typhoeus::Hydra.new(max_concurrency: 200)
 
-    @internal = []
-    @external = {}
+    @internal = Set.new
+    @external = Set.new
   end
 
   def run
@@ -37,48 +55,54 @@ def run
 
   def report_stdout
     puts "Found #{@internal.count} uniq internal dead links:\n"
-    puts @internal
+    puts @internal.to_a
     puts
 
     puts "Found #{@external.count} uniq external dead links:\n"
+    puts @internal.to_a
+    puts
   end
 
   def report_json
-    File.write(@json_file, JSON.pretty_generate({ external: @external, internal: @internal }))
+    File.write(@json_file, JSON.pretty_generate({ external: @external.to_a, internal: @internal.to_a }))
+  end
+
+  def exit_code
+    return 1 if @internal.count > 0 || @external.count > 0
+    return 0
   end
 
   private
 
   def scan_internal_links(file_path, content)
-    match = content.scan(INTERNAL_LINK_REGEXP)
-    match.each do |(relative_path)|
-      # Remove anchor
-      relative_path.gsub!(/#[\w-]+/, '')
-
-      if relative_path.end_with?('.png')
-        full_path = "src/#{relative_path}"
-      else
-        full_path = "src/#{relative_path}/index.md"
-      end
+    INTERNAL_LINK_REGEXPS.each do |regexp|
+      match = content.scan(regexp)
+      match.each do |(relative_path)|
+        # Remove anchor
+        relative_path.gsub!(/#[\w-]+/, '')
+
+        if relative_path.end_with?('/')
+          full_path = "src/#{relative_path}/index.md"
+        else
+          full_path = "src/#{relative_path}"
+        end
 
-      # Remove double //
-      full_path.gsub!(/\/\//, '/')
+        # Remove double //
+        full_path.gsub!(/\/\//, '/')
 
-      next if File.exists?(full_path)
+        next if File.exists?(full_path)
 
-      @internal ||= []
-      @internal << full_path
+        @internal << full_path
+      end
     end
-
-    @internal.uniq!
   end
 
   def scan_external_links(file_path, content)
     external_links = URI.extract(content, ['http', 'https'])
-    external_links.keep_if do |external_link|
-      !external_link.start_with?('http://kuzzle') &&
-        !external_link.start_with?('http://localhost') &&
-        !external_link.start_with?('http://<')
+
+    external_links.delete_if do |external_link|
+      external_link.start_with?(*IGNORED_EXTERNAL_LINKS) ||
+      external_link == 'http://'
     end.each do |external_link|
       # Remove markdown parenthesis and other garbage
       external_link.gsub!(/[\)][\.:,]*/, '')
@@ -87,8 +111,7 @@ def scan_external_links(file_path, content)
 
       request.on_complete do |response|
         if response.code != 200
-          @external[external_link] ||= []
-          @external[external_link] << file_path.gsub(/\/\//, '/')
+          @external << external_link
         end
       end
 
@@ -128,3 +151,5 @@ def each_dir(start, &block)
 
 link_checker.report_stdout
 link_checker.report_json
+
+exit link_checker.exit_code

src/core/1/api/essentials/volatile-data/index.md

@@ -77,7 +77,7 @@ There is one special case, where volatile data are stored by Kuzzle for a later
 Volatile data passed to a new subscription query are used two times by Kuzzle:
 
 - if the new subscription triggers [user notifications](/core/1/api/essentials/notifications#user-notification-default), its volatile data are included into those
-- if that subscription is cancelled, whether because of a call to [realtime:unsubscribe](/core/1/api/api-reference/controller-realtime/unsubscribe/), or after the user disconnects: the volatile data provided **at the time of the subscription** are once again copied into user notifications triggered by that event
+- if that subscription is cancelled, whether because of a call to [realtime:unsubscribe](/core/1/api/controllers/realtime/unsubscribe/), or after the user disconnects: the volatile data provided **at the time of the subscription** are once again copied into user notifications triggered by that event
 
 This allows other real-time subscribers to get context information about a user joining or leaving the same subscription as them.
 

src/core/1/guides/cookbooks/elasticsearch/installation/index.md

@@ -10,7 +10,7 @@ order: 100
 
 We want you to work with Elasticsearch while you are reading this cookbook,
 to do so you will need [cURL](https://curl.haxx.se/download.html), a terminal (Linux, Mac, Cygwin...)
-and, optionally, [docker](https://www.docker.com/products/docker) to speed up the installation.
+and, optionally, [docker](https://docs.docker.com/install/) to speed up the installation.
 
 Alternatively you can simply trust the results we provide in the cookbook and skip the installation chapter.
 

src/core/1/guides/essentials/configuration/index.md

@@ -11,7 +11,7 @@ The Kuzzle **configuration** is stored in a [kuzzlerc file](https://github.com/k
 
 Kuzzle uses [rc](https://github.com/dominictarr/rc) to **override** its default configuration by either:
 
-- loading parameters from a `.kuzzlerc` file ([sample file](https://github.com/kuzzleio/kuzzle/blob/master/.kuzzlerc.sample));
+- loading parameters from a `.kuzzlerc` file ([sample file](https://github.com/kuzzleio/kuzzle/blob/master/.kuzzlerc.sample)) ;
 - loading parameters from environment variables with a `kuzzle_` prefix.
 
 ### Example 1: configuring Kuzzle using a custom `.kuzzlerc` file

src/core/1/guides/essentials/data-validation/index.md

@@ -16,7 +16,7 @@ With Kuzzle, instead of programming the validation logic yourself, you can pick
 For a detailed look at data validation, please refer to our [Data Validation Reference](/core/1/guides/cookbooks/datavalidation).
 
 ::: info
-You can bypass data validation by using [bulk:write](/core/1/api/api-reference/bulk-controller/write) or [bulk:mWrite](/core/1/api/api-reference/bulk-controller/m-write) actions.
+You can bypass data validation by using [bulk:write](/core/1/api/controllers/bulk/write) or [bulk:mWrite](/core/1/api/controllers/bulk/m-write) actions.
 :::
 
 ---

src/core/1/guides/essentials/installing-kuzzle/index.md

@@ -93,7 +93,8 @@ Recover the public IP or the hostname provided by AWS before you proceed.
 Check that Kuzzle is up and running with following HTTP request:
 
 ```sh
-$ curl 'http://yourInstanceIpOrHostname:7512?pretty'
+# Replace kuzzle with your instance hostname or IP
+$ curl 'http://kuzzle:7512?pretty'
 {
   "requestId": "9b07a095-7143-49a5-9079-a34e937fdf3e",
   "status": 200,

src/core/1/guides/kuzzle-depth/request-life-cycle/index.md

@@ -36,7 +36,7 @@ The following diagram shows how a request flows between the client application,
 
 ![read_scenario_http_details](./Synchronous_Request_HTTP_Protocol_Sequence.png)
 
-- The HTTP client will request a document by making an HTTP GET request. For instance, to retrieve a document with `_id` equal to `739c26bc-7a09-469a-803d-623c4045b0cb` in the `users` collection, the client will perform the following request: `GET http://kuzzlebackend:7512/myindex/users/739c26bc-7a09-469a-803d-623c4045b0cb`.
+- The HTTP client will request a document by making an HTTP GET request. For instance, to retrieve a document with `_id` equal to `739c26bc-7a09-469a-803d-623c4045b0cb` in the `users` collection, the client will perform the following request: `GET http://kuzzle:7512/myindex/users/739c26bc-7a09-469a-803d-623c4045b0cb`.
 
 - The _Entry Point_ receives the message and passes it on to the _HTTP Router_.
 

src/core/1/plugins/essentials/getting-started/index.md

@@ -40,7 +40,7 @@ The main Plugin class is defined in the `index.js`. You can start edit it adding
 We need to provide the `configuration` and the `context` to plugins. In that purpose, plugins must have an `init` function which will have them as parameters : this `init` function is the very first one to be called by Kuzzle and is mandatory to start a plugin. You can now write your own functions and your own routes as described inside the `index.js`. You can also write unit tests : see `steps.js`.
 
 <div class="alert alert-info">
-You can find more information about the <code>init</code> function <a href="/core/1/plugins/plugin-layout/init-function/"> here</a>.
+You can find more information about the <code>init</code> function <a href="/core/1/plugins/guides/manual-setup/init-function/"> here</a>.
 </div>
 <div class="alert alert-success">
 You have now everything you need to start writing your own Kuzzle plugin.

src/core/1/plugins/guides/pipes/index.md

@@ -14,7 +14,7 @@ Pipes can:
 - Decide to abort a task. If a pipe throws an error, Kuzzle interrupts the task, and forwards a standardized version of the thrown error to the originating user
 - Change the received information. Kuzzle will use the updated information upon resuming the task
 
-<div class="alert alert-warning">If a pipe takes too long to respond, Kuzzle will eventually abort the entire task with a <a href="/core/1/plugins/errors/gatewaytimeouterror">GatewayTimeout</a> error. The timeout value can be changed in the <a href="/core/1/guides/essentials/configuration/">configuration files.</a></div>
+<div class="alert alert-warning">If a pipe takes too long to respond, Kuzzle will eventually abort the entire task with a <a href="/core/1/plugins/plugin-context/errors/gatewaytimeouterror">GatewayTimeout</a> error. The timeout value can be changed in the <a href="/core/1/guides/essentials/configuration/">configuration files.</a></div>
 
 ---
 

src/core/1/protocols/native-protocols/mqtt/index.md

@@ -88,7 +88,7 @@ client.on('message', (topic, raw) => {
 
 ### Using Kuzzle subscriptions
 
-Kuzzle allows to [subscribe](https://docs.kuzzle.io/core/1/api/controllers/realtime/subscribe/) to messages and events using advanced filters.
+Kuzzle allows to [subscribe](/core/1/api/controllers/realtime/subscribe/) to messages and events using advanced filters.
 
 Each time a subscription is sent, a dedicated MQTT topic is created, named after the `channel` property issued by Kuzzle.
 

src/sdk/cpp/1/controllers/collection/search-specifications/index.md

@@ -54,7 +54,7 @@ An empty body matches all documents in the queried collection.
 
 ## Return
 
-Returns a [kuzzleio::SearchResult](/sdk/cpp/1/search-result).
+Returns a [kuzzleio::SearchResult](/sdk/cpp/1/core-classes/search-result).
 
 ## Exceptions
 

src/sdk/cpp/1/controllers/document/search/index.md

@@ -65,7 +65,7 @@ An empty body matches all documents in the queried collection.
 
 ## Return
 
-Returns a [kuzzleio::SearchResult](/sdk/cpp/1/search-result) instance.
+Returns a [kuzzleio::SearchResult](/sdk/cpp/1/core-classes/search-result) instance.
 
 ## Exceptions
 

src/sdk/java/1/essentials/getting-started/index.md

@@ -120,7 +120,7 @@ Now, you know how to:
 - Subscribe to notifications
 
 ::: info
-Having trouble? Get in touch with us on [Gitter](https://gitter.im/kuzzleio/kuzzle)! We're happy to help.
+Having trouble? Get in touch with us on [Gitter](https://gitter.im/kuzzleio/kuzzle) ! We're happy to help.
 :::
 
 ## Where do we go from here?

src/sdk/js/6/getting-started/webpack/index.md

@@ -13,7 +13,7 @@ We will walk you through creating scripts that can **store** documents in Kuzzle
 
 ## Running Kuzzle
 
-Before going through this tutorial, you should have a Kuzzle server running. Please refer to the [Running Kuzzle Tutorial](https://docs.kuzzle.io/core/1/guides/getting-started/running-kuzzle/) if you don't have one yet.
+Before going through this tutorial, you should have a Kuzzle server running. Please refer to the [Running Kuzzle Tutorial](/core/1/guides/getting-started/running-kuzzle/) if you don't have one yet.
 
 ## Fun with the SDK