Skip to content

Commit

Permalink
Trivial tense/typos (#146)
Browse files Browse the repository at this point in the history
* Trivial tense/typos

* Update dmoz.org to dmoztools.net since dmoz.org now redirects. (#145) (#147)
  • Loading branch information
Russ Ferriday authored and Madison Bahmer committed Nov 7, 2017
1 parent 62be326 commit 95d5247
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 4 deletions.
4 changes: 2 additions & 2 deletions docs/topics/introduction/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,7 @@ At this point you should have a Scrapy Cluster setup that has been tested and ap

.. note:: You can append ``&`` to the end of the following commands to run them in the background, but we recommend you open different terminal windows to first get a feel of how the cluster operates.

The following commands outline what you would run in a traditional environment. If using a container based solution these commands are ran when you run the container itself.
The following commands outline what you would run in a traditional environment. If using a container based solution these commands are run when you run the container itself.

**Bare Bones:**

Expand Down Expand Up @@ -425,7 +425,7 @@ Which ever setup you chose, every process within should stay running for the rem

.. note:: If you chose to set the Rest service up, this section may also be performed via the :doc:`../rest/index` endpoint. You just need to ensure the JSON identified in the following section is properly fed into the :ref:`feed <feed_endpoint>` rest endpoint.

*The follwing commands can be ran from the command line, whether that is on the machine itself or inside the Kafka Monitor container depends on the setup chosen above.*
*The following commands can be run from the command line, whether that is on the machine itself or inside the Kafka Monitor container depends on the setup chosen above.*

1) We now need to feed the cluster a crawl request. This is done via the same Kafka Monitor python script, but with different command line arguements.

Expand Down
4 changes: 2 additions & 2 deletions docs/topics/kafka-monitor/design.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ Soon enough those same applications wanted the ability to retrieve information a

The Kafka Monitor reads from the desired inbound Kafka topic, and applies the currently loaded Plugin's JSON APIs to the received message. The first Plugin to have a valid `JSON Schema <http://json-schema.org/latest/json-schema-core.html>`_ for the received JSON object is then allowed to do its own processing and manipulation of the object.

In Scrapy Cluster's use case, the default Plugins write their requests into Redis keys, but the functionality does not stop there. The Kafka Monitor settings can alter which plugins are loaded, or add new plugins to extend functionality. These modules allow the Kafka Monitor core to have a small footprint but allow extension or different plugins to be ran.
In Scrapy Cluster's use case, the default Plugins write their requests into Redis keys, but the functionality does not stop there. The Kafka Monitor settings can alter which plugins are loaded, or add new plugins to extend functionality. These modules allow the Kafka Monitor core to have a small footprint but allow extension or different plugins to be run.

The Kafka Monitor can be ran as a single process, or part of the same Kafka consumer group spread across multiple machines and processes. This allows distributed and fault tolerant throughput to ensure the crawl requests to the cluster are always read.
The Kafka Monitor can be run as a single process, or part of the same Kafka consumer group spread across multiple machines and processes. This allows distributed and fault tolerant throughput to ensure the crawl requests to the cluster are always read.

From our own internal debugging and ensuring other applications were working properly, a utility program called Kafka Dump was also created in order to be able to interact and monitor the kafka messages coming through. This is a small dump utility with no external dependencies, to allow users to get an insight into what is being passed through the Kafka topics within the cluster.

Expand Down

0 comments on commit 95d5247

Please sign in to comment.