Merge pull request #136 from ESGF/devel

Devel

docs/source/conf.py

@@ -24,7 +24,7 @@
 author = u'LLNL ESGF Installer Working Team'
 
 # The short X.Y version
-version = u'4.0.4'
+version = u'4.0.5'
 # The full version, including alpha/beta/rc tags
 release = u''
 

docs/source/config/config.rst

@@ -32,16 +32,16 @@ Regardless of Node type (data-only, index/idp, all, etc.) all three groups, ``da
 In the case of data-only this means another site's index/idp host(s) must be assigned to their respective group.
 
 
-See the sample inventory file, `sample.hosts <https://github.com/ESGF/esgf-ansible/blob/4.0.4/sample.hosts>`_ for more information regarding inventory files.
+See the sample inventory file, `sample.hosts <https://github.com/ESGF/esgf-ansible/blob/4.0.5/sample.hosts>`_ for more information regarding inventory files.
 
 Host Variable Files
 -------------------
 
 The host variable file provides an interface that allows users to configure common options for their deployments.
 It is automatically detected for each managed machine and must be named ``host_vars/[hostname].yml`` where ``[hostname]`` is the hostname of each managed machine.
 Options regarding certificates and keys for various services require the most attention as certain requirements have been placed on these variables.
-See the host variables directory, `host_vars <https://github.com/ESGF/esgf-ansible/blob/4.0.4/host_vars>`_, and find the sample file
+See the host variables directory, `host_vars <https://github.com/ESGF/esgf-ansible/blob/4.0.5/host_vars>`_, and find the sample file
 that best describes the desired use case.
-Also see the sample host variable file, `host_vars/myhost.my.org.yml <https://github.com/ESGF/esgf-ansible/blob/4.0.4/host_vars/myhost.my.org.yml>`_ for a comprehensive overview.
+Also see the sample host variable file, `host_vars/myhost.my.org.yml <https://github.com/ESGF/esgf-ansible/blob/4.0.5/host_vars/myhost.my.org.yml>`_ for a comprehensive overview.
 Advanced users may want to make configuration choices beyond what is provided in the host variable file
 , see the `Contributing Guide <https://github.com/ESGF/esgf-ansible#advice-and-contributing>`_ for more information on this.

docs/source/developers/developers.rst

@@ -0,0 +1,53 @@
+Developers Guide
+================
+
+Adding WSGI services to an ESGF node
+------------------------------------
+
+It's easy to add new services developed with Flask, Django, etc. and proxy using the esgf-httpd configuration and mod_wsgi express.
+
+This guide assumes you have set up a project using flask at the example location ``/opt/esgf/flaskdemo/demo`` and you have the application entry point accessible in the ``/opt/esgf/flaskdemo/demo.wsgi``.  Your demo app must world-readable and recommended to be owned by the ``apache`` user and group.  These instructions assume to be run by a root user, as many server configs disallow a shell run under the apache user, but a regular user can be used to run the service for testing purposes.  
+
+- Create and activate a conda environment to run your webapp. 
+
+.. code:: console
+
+    source /usr/local/conda/bin/activate
+    conda create -n flaskdemo
+    conda activate flaskdemo
+
+.. note:: 
+    if your app was developed in Python 2.7 you'll need to create the environment with the following instead:
+
+.. code:: console
+
+    conda create -n flaskdemo 'python<3.0'
+
+- Install modules needed to run your app.  Our demo uses flask, but you could use django (and expect additional required packages.)  Note that diffculty has been encountered with mod_wsgi version 4.6.7, so we recommend an earlier version:
+
+.. code:: console
+
+   pip install flask 'mod_wsgi<4.6'
+
+- Run the `mod_wsgi-express` command to create a httpd service instance for your webapp and start the instance. 
+
+.. code:: console
+
+    cd /opt/esgf/flaskdemo/demo
+    mod_wsgi-express setup-server --server-root /etc/wsgi-demo --user apache --group apache --host localhost --port 8087 --mount-point /demo demo.wsgi
+    /etc/wsgi-demo/apachectl start
+
+- You should be able to access the demo now under ``http://localhost:8087/demo`` using ``curl`` or ``wget``. 
+
+- For external access on 443 for https, add the following directives to `/etc/httpd/conf/httpd.ssl.conf`: 
+
+.. code:: console
+
+    ProxyPass /demo http://localhost:8087/demo
+    ProxyPassReverse /demo http://localhost:8087/demo
+
+- Restart httpd
+
+- If you want the site available also on 80 for old insecure http, you can add the same directives to ``/etc/httpd/conf/esgf-httpd.conf``.  In addition you need to add a rule to exempt ``/demo`` from the automatic redirection of http traffic to https as done for several of the ESGF Tomcat webapps that are proxied in that section.
+
+

docs/source/faq/comptable.rst

@@ -0,0 +1,45 @@
+Comparison of ESGF v2 vs v4
+===========================
+
+In the following table, if no context given, items in the table in the v2 column are the command line flag to the `esg-node` script.  Items in the v4 column are a playbook .yml file
+
++--------------------------+------------------------+------------------------------+
+|  Action or Location      | ESGF v2 (Bash scripts) | ESGF v4 (Ansible)            |
++==========================+========================+==============================+
+|   CoG location           | NA (mod_wsgi in httpd) | /etc/cog-wsgi-8889           |
++--------------------------+------------------------+------------------------------+
+|   SLCS                   | NA (mod_wsgi in httpd) | /etc/slcs-wsgi-8888          |
++--------------------------+------------------------+------------------------------+
+| Tomcat control           | esg-node function      | catalina.sh                  |
++--------------------------+------------------------+------------------------------+
+| Solr index location      | /esg/solr-index        | /usr/local/solr-home         | 
++--------------------------+------------------------+------------------------------+
+| Bootstrap                | - wget esg-bootstrap   | - (No bootstrap)             |
+|                          | - ./esg-bootstrap      | - git clone esgf-ansible     |
++--------------------------+------------------------+------------------------------+
+| Configuration            | esg-autoinstall.conf   | inventory file               |
+| (auto-installation)      |                        | host variables files         |
++--------------------------+------------------------+------------------------------+
+| Install latest           | --install (--upgrade)  | install.yml                  |   
+| ESGF version             |                        |                              |
++--------------------------+------------------------+------------------------------+
+| CSR                      | --generate-esgf-csrs   | local_certs.yml              |
++--------------------------+------------------------+------------------------------+
+| Certs / CA               | --update-temp-ca       | local_certs.yml              |
++--------------------------+------------------------+------------------------------+
+| Stop                     | --stop                 | stop.yml                     |
++--------------------------+------------------------+------------------------------+
+| Start                    | --start                | start.yml                    |
++--------------------------+------------------------+------------------------------+
+| Restart                  | --restart              | stop.yml + start.yml         |
++--------------------------+------------------------+------------------------------+
+| Status                   | --status               | status.yml                   |     
++--------------------------+------------------------+------------------------------+
+| Certificate Installation | - --install-local-certs| local_certs.yml              |
++--------------------------+------------------------+------------------------------+
+|    (Web)                 | - --install-keypair    | web_certs.yml                |
++--------------------------+------------------------+------------------------------+
+| LetsEncrypt request      | NA                     | install.yml (tryletsencript) |
++--------------------------+------------------------+------------------------------+
+| Shard Replicas           | --add-replica-shard    | shards.yml                   |
++--------------------------+------------------------+------------------------------+

docs/source/faq/faq.rst

@@ -28,4 +28,9 @@ Some tasks can take several minutes to complete. Make sure the verbose flag is s
 This second case could be things such as bad SSH authentication, a temporary network issue, an issue with the machine being deployed to, 
 or a variable improperly specified in the host variables file. 
 In this second case, attempt to determine and resolve the error, then start the deployment again. 
-If it is confidently determined be the first case, please `submit an issue <https://github.com/ESGF/esgf-ansible/issues/new/choose>`_ to this repository.
+
+If it is confidently determined be the first case, please `submit an issue <https://github.com/ESGF/esgf-ansible/issues/new/choose>`_ to this repository.
+
+**Q:  I'm familiar with funcitonality in ESGF v2 using the bash script ``esg-node``, eg. starting, stopping the node, requesting certificates. How can I do that now with ESGF Ansible?
+
+**Answer:**  Please consult the following table that outlines the equivalent functions from ESGF v2 and v4: `Comparison of ESGF Versions <comptable.html>`_ 

docs/source/index.rst

@@ -18,6 +18,8 @@ Welcome to ESGF-Ansible's documentation!
    verify/verify.rst
    whatsnew/whatsnew.rst
    faq/faq.rst
+   faq/comptable.rst
+   developers/developers.rst
 
 Indices and tables
 ==================

docs/source/setup/setup.rst

@@ -3,9 +3,9 @@ Setup
 
 These steps describe how to setup the control machine. SSH access to the managed machine(s) is the responsibility of the user.
 
-1. Clone the esgf-ansible repo at the current version, for example version 4.0.4 ::
+1. Clone the esgf-ansible repo at the current version, for example version 4.0.5 ::
 
-    git clone --branch 4.0.4 https://github.com/ESGF/esgf-ansible.git && cd esgf-ansible
+    git clone --branch 4.0.5 https://github.com/ESGF/esgf-ansible.git && cd esgf-ansible
 
 2. (Optional, but strongly recommended) Create a Python environment, using a tool like ``virtualenv`` or ``conda``, and activate the environment.
 

docs/source/usage/usage.rst

@@ -11,7 +11,7 @@ Inventory Files
 This file specifies the managed machines.
 It is specified at the command line via ``-i [inventory file name]``.
 
-See the sample inventory file, `sample.hosts <https://github.com/ESGF/esgf-ansible/blob/4.0.4/sample.hosts>`_ for more information.
+See the sample inventory file, `sample.hosts <https://github.com/ESGF/esgf-ansible/blob/4.0.5/sample.hosts>`_ for more information.
 
 Also, see `Ansible's Inventory Info <https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html>`_.
 
@@ -20,9 +20,9 @@ Host Variable Files
 These files specify options for each managed machine.
 It is automatically detected for each managed machine and must be named ``host_vars/[hostname].yml`` where ``[hostname]`` is the hostname of each managed machine.
 
-See the sample host variable file, `host_vars/myhost.my.org.yml <https://github.com/ESGF/esgf-ansible/blob/4.0.4/host_vars/myhost.my.org.yml>`_ for more information.
+See the sample host variable file, `host_vars/myhost.my.org.yml <https://github.com/ESGF/esgf-ansible/blob/4.0.5/host_vars/myhost.my.org.yml>`_ for more information.
 
-Also, see the other sample files in `host_vars <https://github.com/ESGF/esgf-ansible/blob/4.0.4/host_vars>`_, and find the sample file
+Also, see the other sample files in `host_vars <https://github.com/ESGF/esgf-ansible/blob/4.0.5/host_vars>`_, and find the sample file
 that best describes the desired use case.
 
 Examples
@@ -76,13 +76,13 @@ A test deployment to all managed test hosts, with SSH via a non-root user, *joe*
 Deployment Control
 ******************
 These examples show various ways of controlling the deployment process. 
-Deployments are done in the order of includes in `install.yml <https://github.com/ESGF/esgf-ansible/blob/4.0.4/install.yml>`_.
+Deployments are done in the order of includes in `install.yml <https://github.com/ESGF/esgf-ansible/blob/4.0.5/install.yml>`_.
 This order is base, idp, index then data. 
 While repeating steps will not cause any problems, it simply slows things down. 
 Additionally, for a more reliable deployment process it may be desired to do one phase at a time. 
 Or if the deployment got interrupted after completing, for example, the `base` steps, these steps could be skipped when the deployment is started again.
 
-Controlling the deployment is done with tags. The tags available in the `install.yml <https://github.com/ESGF/esgf-ansible/blob/4.0.4/install.yml>`_ play are ``base``, ``idp``, ``index``, ``data`` and ``publisher``.
+Controlling the deployment is done with tags. The tags available in the `install.yml <https://github.com/ESGF/esgf-ansible/blob/4.0.5/install.yml>`_ play are ``base``, ``idp``, ``index``, ``data`` and ``publisher``.
 These can be used with ``--tags`` and ``--skip-tags``,  as well as with ``--limit [hostname]`` to control exactly what is done and where.
 The ``base`` steps will be done everytime unless specified via ``--skip-tags``.
 

docs/source/whatsnew/whatsnew.rst

@@ -6,6 +6,26 @@ The Critical section outlines any items the are highly likely to require changes
 The Info section outlines informative items that may be helpful for users to know, but likely do not require action.
 All items described within a release are relative to the previous release.
 
+4.0.5 (2019-10-29)
+******************
+
+Critical
+--------
+- All Globus tools setup steps will be skipped if the user does not provide Globus account credentials
+
+Info
+----
+- Added `developer documentation page  <../developers/developers.html>`_
+- Added `compatability infomation page  <../faq/comptable.html>`_
+- Tomcat updated to ``8.5.46``
+- ESG-Search updated to ``v4.17.10``
+- CoG update to ``v3.15.3``
+- A number of updates within the ESGF-Publisher Conda environment
+- Added playbook for node status tool ``install_ns.yml``
+- Updated ACME Challenge (for LetsEncrypt certifcates) to v2
+- Added ``roles/httpd/templates/robots.txt`` to direct web crawlers to not crawl ESGF data nodes
+
+
 4.0.4 (2019-07-10)
 ******************
 

group_vars/all.yml

@@ -14,7 +14,7 @@ httpd_version: "{{ '2.4' if (is_7) else '2.2' }}"
 versions:
   installer: 4.0.0-alpha1
   java: jdk1.8.0_192
-  search: v4.17.9
+  search: v4.17.10
   stats_api: v1.0.9
   dashboard: v1.5.25
   dashboard_db: 0.0.4
@@ -62,8 +62,8 @@ conda:
   mods: lib/python2.7/site-packages
 
 tomcat:
-  src: http://archive.apache.org/dist/tomcat/tomcat-8/v8.5.39/bin/apache-tomcat-8.5.39.tar.gz
-  root_dir: apache-tomcat-8.5.39
+  src: http://archive.apache.org/dist/tomcat/tomcat-8/v8.5.46/bin/apache-tomcat-8.5.46.tar.gz
+  root_dir: apache-tomcat-8.5.46
   path: /usr/local/tomcat
   webapps: /usr/local/tomcat/webapps
   ts:

group_vars/data.yml

@@ -3,7 +3,7 @@ thredds:
   content: "{{ esg.content }}/thredds"
   tomcat_user:
     name: "dnode_user"
-    pass: "{{ admin_pass |hash('md5') }}"
+    pass: "{{ admin_pass }}"
     roles: "tdrAdmin,tdsConfig"
 
 thredds_webapp:

group_vars/index.yml

@@ -18,7 +18,11 @@ solr:
 
 cog:
   repo: https://github.com/EarthSystemCoG/COG.git
-  version: v3.14.3
+  version: v3.15.3
   dest: /usr/local/cog/cog_install
   base: /usr/local/cog
   wsgi_dir: /etc/cog-wsgi-8889
+
+node_status:
+  repo: https://github.com/ESGF/esgf-utils
+  dest: /usr/local/esgf-utils

install_ns.yml

@@ -0,0 +1,19 @@
+- hosts: all
+  # strategy: free
+  become: yes
+  connection: paramiko
+  any_errors_fatal: true
+  tasks:
+    # Get the admin password
+    - name: Include the admin passwd role
+      block:
+        - include_role:
+            name: passwd
+      tags: always
+
+    - name: Node Status
+      block:
+        - include_role:
+            name: node_status
+      tags: index
+      when: "'index' in group_names"

roles/base/tasks/main.yml

@@ -31,5 +31,6 @@
     loop_var: base_task
 
 - name: Include Globus Tools
+  when: globus_user is defined and globus_pass is defined
   include_role:
     name: globus_tools

roles/data/tasks/thredds.yml

@@ -40,6 +40,16 @@
     src: thredds/log4j2.xml
     dest: "{{ tomcat.webapps }}/{{ thredds_webapp.name }}/WEB-INF/classes/log4j2.xml"
 
+- name: Digest thredds user credential
+  no_log: true
+  command: "{{ tomcat.path }}/bin/digest.sh -a 'sha-256' -s 20 -i 5 {{ thredds.tomcat_user.pass }}"
+  register: cred_digest
+
+- name: Set credential digest variable
+  no_log: true
+  set_fact:
+    thredds_pass_digest: "{{ cred_digest.stdout.split(':')[-1] }}"
+
 - name: Install {{ thredds.tomcat_user.name }} into tomcat-users.xml
   template:
     src: thredds/tomcat-users.xml.j2

roles/data/templates/thredds/tomcat-users.xml.j2

@@ -3,5 +3,5 @@
   <role rolename="tdsConfig"/>
   <role rolename="manager"/>
   <role rolename="tdrAdmin"/>
-  <user username="{{ thredds.tomcat_user.name }}" password="{{ thredds.tomcat_user.pass }}" roles="{{ thredds.tomcat_user.roles }}"/>
+  <user username="{{ thredds.tomcat_user.name }}" password="{{ thredds_pass_digest }}" roles="{{ thredds.tomcat_user.roles }}"/>
 </tomcat-users>

roles/globus_certs/meta/main.yml

@@ -21,5 +21,5 @@ dependencies:
     }
   - {
       role: 'myproxy_certs',
-      when: "'idp' is in group_names"
+      when: "'idp' in group_names"
     }

roles/httpd/tasks/letsencrypt.yml

@@ -3,7 +3,9 @@
 # Create an ACME Challenge
 - name: Create ACME Challenge
   acme_certificate:
-    acme_directory: https://acme-v01.api.letsencrypt.org/directory
+    acme_version: 2
+    terms_agreed: yes
+    acme_directory: https://acme-v02.api.letsencrypt.org/directory
     account_key_src: /tmp/account_key.pem
     csr: /tmp/httpdhost.csr
     dest: "{{ httpd.hostcert }}"
@@ -34,7 +36,9 @@
 - name: Run ACME Challenge
   when: acme_challenge is changed
   acme_certificate:
-    acme_directory: https://acme-v01.api.letsencrypt.org/directory
+    acme_directory: https://acme-v02.api.letsencrypt.org/directory
+    acme_version: 2
+    terms_agreed: yes
     account_key_src: /tmp/account_key.pem
     csr: /tmp/httpdhost.csr
     dest: "{{ httpd.hostcert }}"
@@ -45,4 +49,4 @@
   when: acme_challenge is changed
   service:
     name: httpd
-    state: stopped
+    state: stopped

roles/httpd/tasks/main.yml

@@ -3,7 +3,19 @@
   package:
     name:
       - "{{ pkgs.apache }}"
-      - "mod_ssl"
+      - mod_ssl
+      - mod_security
+      - mod_security_crs
+
+- name: Install mod_security.conf
+  template:
+    src: mod_security.conf
+    dest: /etc/httpd/conf.d/mod_security.conf
+
+- name: Install robots.txt
+  template:
+    src: robots.txt
+    dest: /var/www/html/robots.txt
 
 # Create config dirs
 - name: Create esgf-proxy configuration directories

roles/httpd/templates/httpd2.2/esgf-httpd.conf.j2

@@ -79,6 +79,11 @@ LoadModule ssl_module modules/mod_ssl.so
 User apache
 Group apache
 ServerAdmin root@localhost
+
+# Include modsecurity rules
+# Modify the below file to change rules from default
+#Include conf.d/mod_security.conf
+
 UseCanonicalName Off
 HostnameLookups Off
 ErrorLog logs/error_log