Merge pull request #160 from saradickinson/2.2.0_release

Update docs for 2.2 release

docs/src/Hedgehog_Installation_Guide_(from_Source).xml

@@ -1,2 +1,2 @@
 <ac:structured-macro ac:macro-id="3e356863-675c-4c21-87c5-e0d78285a0f4" ac:name="warning" ac:schema-version="1"><ac:rich-text-body>
-<p><span>Installation from source is no longer officially supported in 2.1.1. Please see the 'Hedgehog Installation Guide (from Packages)' for how to install 2.1.1.</span></p></ac:rich-text-body></ac:structured-macro>
+<p><span>Installation from source is no longer officially supported . Please see the 'Hedgehog Installation Guide (from Packages)' for how to install 2.2.0.</span></p></ac:rich-text-body></ac:structured-macro>

docs/src/Hedgehog_Upgrade_Guide_(to_Packges).xml

@@ -8,70 +8,25 @@
 <p><span style="line-height: 1.5625;">If desired, back up the existing installation (the directory structure in the User Guide is a useful reference) and database.&nbsp;</span></p>
 <h2><span style="line-height: 1.5625;">1. Pre-installation</span></h2>
 <h3>1.2 Cron jobs</h3>
-<p>Stop refile and grok cron job running:</p><ac:structured-macro ac:macro-id="08a576ef-79c7-41f7-aa6b-ac1e1d695865" ac:name="code" ac:schema-version="1"><ac:plain-text-body><![CDATA[sudo -u hedgegho crontab -e
+<p>Stop refile and grok cron job running:</p><ac:structured-macro ac:macro-id="08a576ef-79c7-41f7-aa6b-ac1e1d695865" ac:name="code" ac:schema-version="1"><ac:plain-text-body><![CDATA[sudo -u hedgeghog crontab -e
 # Import XML data every 15 mins
-00,15,30,45 * * * * <prefix>/bin/refile_and_grok.sh -c >> /var/log/hedgehog/refile_and_grok_xml_to_db.sh.log 2>&1]]></ac:plain-text-body></ac:structured-macro>
+# 00,15,30,45 * * * * <prefix>/bin/refile_and_grok.sh -c >> /var/log/hedgehog/refile_and_grok_xml_to_db.sh.log 2>&1]]></ac:plain-text-body></ac:structured-macro>
 <p>Comment out any other jobs that are about to run. If the dsc-extractor process is running wait for it to finish before continuing.</p>
 <h3>1.3 Disable web front end</h3>
 <p>It is recommended to disable the web front end during upgrade e.g by enabling a holding page.</p>
-<p>&nbsp;</p><ac:structured-macro ac:macro-id="f05d9ecc-d0b2-41af-bfe5-8ffb598a7395" ac:name="info" ac:schema-version="1"><ac:rich-text-body>
-<p><strong>Source install vs Package install:</strong></p>
-<ul>
-<li><span>If 2.1.0 was installed from source, perform steps 2 and 4.&nbsp;</span></li>
-<li>If 2.1.0 was installed from packages, perform steps 3 and 4.</li></ul></ac:rich-text-body></ac:structured-macro>
-<h2>2. Installation of packages</h2>
-<h3>2.1 Install packages</h3>
-<ul>
-<li>Refer to section 2.1 of the 'Hedgehog Installation Guide' to install the required packages.&nbsp;</li></ul>
-<h3>2.3 Cater for different directory structure</h3>
-<p>Installing from a package places some files in a different directory structure compared to installing from code.</p>
-<ul>
-<li><strong>Data</strong>&nbsp;- if it is desired to retain existing data (XML files, cached plots and RSSAC reports) then the existing&nbsp;<em>&lt;prefix&gt;/var/hedgehog</em>&nbsp;directory should be copied or linked (e.g. symbolically or via re-mounting) to the new location shown below
-<ul>
-<li>If this is not required then simply run the&nbsp;<em>/usr/bin/hedgehogctl database_update_nodes</em>&nbsp;script again to create the directory structure for upload of XML data in the new location AFTER completing step 2.1<br />&nbsp;</li></ul></li></ul>
-<ul>
-<li><strong>Tools and scripts - </strong>those&nbsp;installed via source under&nbsp;<em>&lt;prefix&gt;</em>&nbsp;in&nbsp;<em>bin</em>&nbsp;and&nbsp;<em>libexec/hedgehog</em>&nbsp;are installed by the packages under&nbsp;<em>/usr/</em>&nbsp;as shown below. Update any cron jobs or scripts accordingly.&nbsp;</li></ul>
-<p>&nbsp;</p>
-<table>
-<tbody>
-<tr>
-<th>Source location</th>
-<th>Package location</th>
-<th colspan="1">Contents</th></tr>
-<tr>
-<td>&lt;prefix&gt;/etc/hedgehog/</td>
-<td>/etc/hedgehog/</td>
-<td colspan="1">Hedgehog configuration files (yaml and nodes)</td></tr>
-<tr>
-<td colspan="1">&lt;prefix&gt;/etc/hedgehog/*.sample</td>
-<td colspan="1">/usr/share/doc/hedgehog/examples/*.sample</td>
-<td colspan="1">Sample Hedgehog configuration files</td></tr>
-<tr>
-<td>&lt;prefix&gt;/var/hedgehog/</td>
-<td>/var/lib/hedgehog/</td>
-<td colspan="1">
-<p>'data' directory containing XML data</p>
-<p>'www' directory containing cached plots and rssac reports</p></td></tr>
-<tr>
-<td colspan="1">&lt;prefix&gt;/bin/hedgehogctl</td>
-<td colspan="1">/usr/bin/hedgehogctl</td>
-<td colspan="1">Wrapper script for Hedgehog tools</td></tr>
-<tr>
-<td colspan="1">&lt;prefix&gt;/bin/refile_and_grok</td>
-<td colspan="1">/usr/bin/refile_and_grok</td>
-<td colspan="1">refile_and_grok script</td></tr>
-<tr>
-<td colspan="1">&lt;prefix&gt;/libexec/hedgehog/</td>
-<td colspan="1">/usr/lib/hedgehog/</td>
-<td colspan="1">Hedgehog tools scripts and DDL updates</td></tr></tbody></table>
-<h2>3. Upgrade of Packages</h2><ac:structured-macro ac:macro-id="4b63d758-f1de-456d-adf6-7a76caeca497" ac:name="code" ac:schema-version="1"><ac:plain-text-body><![CDATA[sudo apt-get update
-sudo apt-get upgrade]]></ac:plain-text-body></ac:structured-macro>
-<h2>4. Post-install</h2>
+<h2>2. Upgrade of Packages</h2><ac:structured-macro ac:macro-id="4b63d758-f1de-456d-adf6-7a76caeca497" ac:name="code" ac:schema-version="1"><ac:plain-text-body><![CDATA[sudo apt-get update
+sudo apt-get upgrade
+# Manually run the script 000013_ddl_dnskey_vs_asn to upgrade the database
+sudo -u hedgehog /usr/lib/hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn
+ 
+# Setup the database partitions for the new dataset
+# Use the current year and month. 
+# Ensure that the cron job for managing database partitions will run again before next month. If necessary run this command again for next month.
+sudo -u hedgehog hedgehogctl database_manage_partitions -m YYYY-MM ]]></ac:plain-text-body></ac:structured-macro>
+<h2>3. Post-install</h2>
 <h3>4.1 Test web front end</h3>
 <p>Re-enable and test the web front end. Make sure to restart apache.</p><ac:structured-macro ac:macro-id="add1086e-e973-4e29-b437-dfbd969e203a" ac:name="code" ac:schema-version="1"><ac:parameter ac:name="language">bash</ac:parameter><ac:plain-text-body><![CDATA[sudo service apache2 reload]]></ac:plain-text-body></ac:structured-macro>
-<h3>4.2 Re-enable cron jobs&nbsp;</h3><ac:structured-macro ac:macro-id="29fa3a11-7cb7-447f-8242-11ab26f5bd1e" ac:name="warning" ac:schema-version="1"><ac:rich-text-body>
-<p>Remember to use the new script locations when re-enabling the cron jobs when upgrading from source.</p></ac:rich-text-body></ac:structured-macro>
-<p>&nbsp;</p>
+<h3>3.2 Re-enable cron jobs&nbsp;</h3>
 <p>Restart refile and grok cron job running:</p><ac:structured-macro ac:macro-id="f6849b42-4b08-4fe5-81ed-86ee9f4f597c" ac:name="code" ac:schema-version="1"><ac:plain-text-body><![CDATA[sudo -u hedgehog crontab -e
 # Import XML data every 15 mins
 00,15,30,45 * * * * /usr/bin/refile_and_grok -c >> /var/log/hedgehog/refile_and_grok_xml_to_db.log 2>&1]]></ac:plain-text-body></ac:structured-macro>

docs/src/Hedgehog_Upgrade_Guide_(to_Source).xml

@@ -1,2 +1,2 @@
 <ac:structured-macro ac:macro-id="ebbacdf6-f54f-4f4a-b46f-4276888db4a2" ac:name="warning" ac:schema-version="1"><ac:rich-text-body>
-<p>Installation from source is no longer officially supported in 2.1.1. Please see the 'Hedgehog Upgrade Guide (to Packages)' for how to upgrade from a 2.1.0 source install to 2.1.1.</p></ac:rich-text-body></ac:structured-macro>
+<p>Installation from source is no longer officially supported.</p></ac:rich-text-body></ac:structured-macro>

docs/src/Hedgehog_User_Guide.xml

@@ -18,9 +18,9 @@
 <p>Alternatively the DB may be on a remote server and the Data Manager and Web front-end may be on different servers as shown below.</p>
 <p><ac:image ac:height="400"><ri:attachment ri:filename="Hedgehog_component_view_split.jpg" /></ac:image></p>
 <h3>Install from Packages (Ubuntu)</h3>
-<p>In 2.1 packages for Ubuntu are provided for Hedgehog. It is recommended to use this method for new installs and that existing installs migrate to this method in 2.1 (see the Installation and Upgrade Guides for details).</p>
+<p>In 2.2 packages for Ubuntu are provided for Hedgehog. It is recommended to use this method for new installs and that existing installs migrate to this method in 2.1 (see the Installation and Upgrade Guides for details).</p>
 <h3>Install from Source Code</h3>
-<p>2.1 can also be installed directly from source code if desired.</p>
+<p>2.2 can also be installed directly from source code if desired - note: this is not a supported method.</p>
 <h1>Datasets</h1><ac:structured-macro ac:macro-id="1dccef1c-27f8-4b28-84b4-ca7010c69dc6" ac:name="info" ac:schema-version="1"><ac:rich-text-body>
 <p>Hedgehog is currently tested against the output of the original DSC collector. It has not been tested against the recently released DSC 2.0.</p></ac:rich-text-body></ac:structured-macro>
 <p>&nbsp;</p>
@@ -49,7 +49,9 @@ dataset traffic_volume_queries dns Transport:transport IPVersion:dns_ip_version
 dataset traffic_volume_responses dns Transport:transport IPVersion:dns_ip_version replies-only; 
 dataset traffic_sizes_queries dns Transport:transport MsgLen:msglen queries-only; 
 dataset traffic_sizes_responses dns Transport:transport MsgLen:msglen replies-only; ]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:macro-id="4eaba3e3-a548-4198-89ac-4dfc069be91f" ac:name="code" ac:schema-version="1"><ac:plain-text-body><![CDATA[# New in 2.1
-dataset server_addr dns All:null ServerAddr:server queries-only;]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:macro-id="26a887fa-d2f8-4988-9415-ab76c5e22739" ac:name="note" ac:schema-version="1"><ac:rich-text-body>
+dataset server_addr dns All:null ServerAddr:server queries-only;
+# New in 2.2.0
+dataset qtype_vs_client_subnet dns Qtype:qtype ClientSubnet:cip4_net queries-only max-cells=200;]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:macro-id="26a887fa-d2f8-4988-9415-ab76c5e22739" ac:name="note" ac:schema-version="1"><ac:rich-text-body>
 <p>Data collection for the server_addr dataset is not yet supported in the official release of DSC, but is available using the latest code from the DSC github repo. Also, note this dataset will contain the source addresses of all responses (QR = 1) seen on the collection node even if they do not originate from the local_address specified in the dsc.conf (i.e. if another server sends a response to the collection node for some reason).</p></ac:rich-text-body></ac:structured-macro>
 <p>&nbsp;</p>
 <h1>Plots</h1>
@@ -415,7 +417,20 @@ dataset server_addr dns All:null ServerAddr:server queries-only;]]></ac:plain-te
 <td colspan="1">Busiest client locations by country&nbsp;</td></tr>
 <tr>
 <td colspan="1">&nbsp;</td>
-<td colspan="1">Busiest client locations by city</td></tr></tbody></table><ac:structured-macro ac:macro-id="f7a8bc07-2c7e-482e-850b-ad49fc4eae75" ac:name="note" ac:schema-version="1"><ac:rich-text-body>
+<td colspan="1">Busiest client locations by city</td></tr></tbody></table>
+<p>The following plots are new in 2.2</p>
+<table>
+<tbody>
+<tr>
+<th>Hedgehog plot category</th>
+<th>Hedgehog plot display name</th></tr>
+<tr>
+<td><strong>Client Subnet Statistics</strong></td>
+<td>&nbsp;</td></tr>
+<tr>
+<td>&nbsp;</td>
+<td>
+<p>DNSKEY Queries by ASN</p></td></tr></tbody></table><ac:structured-macro ac:macro-id="f7a8bc07-2c7e-482e-850b-ad49fc4eae75" ac:name="note" ac:schema-version="1"><ac:rich-text-body>
 <ul>
 <li><strong>RSSAC: Zone size and Zone propagation time graphs.</strong> In order for data to be available for these graphs, an instance of the <em>rssacd</em><span>&nbsp;demon&nbsp;must be run to collect the data (it is not provided by the DSC collector).</span></li>
 <li><strong>GEO: Busiest client location by country.</strong> Both client locations and collection node locations can be displayed on this graph. For node locations to be displayed the nodes must have a city specified via the nodes.csv file.</li></ul></ac:rich-text-body></ac:structured-macro>