Collect Earth Online

Crowd-sourced visual interpretation of on-demand satellite imagery

Installation Requirements

Required

• Java Development Kit (version 8, 11, or 17)
• Clojure CLI tools 1.10+
• Postgresql (version 12)
• PostGIS (version 3.2+)
• p7zip
• node 10
• Python 3

Application Configuration

On startup, the CEO application reads a file called config.edn in the top level directory of this repository, which contains various user-configurable system parameters. A sample file called config.example.edn is provided with this repository to get you started. To begin, simply copy it to config.edn. Then edit it according to the following sections.

Please note that you must replace all values surrounded with angle brackets (e.g., “<api-key>”) with actual values for the application to work correctly. Most of the fields should be self-explanatory.

PostgreSQL Database Setup

PostgreSQL needs to be installed on the machine that will be hosting this website. This installation task is system specific and is beyond the scope of this README, so please follow the instructions for your operating system and PostgreSQL version. However, please ensure that the database server’s superuser account is named postgres and that you know its database connection password before proceeding.

PostgreSQL Server

To confirm that PostgreSQL is running as a server on your machine, you’ll need to check its systmed or sysvinit process. To figure out which you’re using on your system, run ps -p 1 -o comm=. Once you know if you’re using systemd or sysvinit, you can run the following command:

# If you're using systemd:
sudo systemctl status postgresql.service
# If you're using sysvinit:
sudo service postgresql status

If it’s not running, you’ll need to start it by running:

# If you're using systemd:
sudo systemctl start postgresql.service
# If you're using sysvinit:
sudo service postgresql start

Finally, to ensure that your PostgreSQL server always restarts when your system reboots, you can run:

# If you're using systemd:
sudo systemctl enable postgresql.service
# If you're using sysvinit:
sudo update-rc.d postgresql enable

CEO Database

Once the PostgreSQL database server is running on your machine, you should navigate to the top level directory (i.e., the directory containing this README) and run the database build command as follows:

clojure -M:build-db build-all --dev-data

This will begin by creating a new database and role called ceo and then add the postgis and pgcrypto extensions to it. Next, the script will populate the database with the schemas, tables, and functions that are necessary for storing and processing ceo’s data. Finally, it will load some default data into these tables that is necessary for the website to function properly. You can optionally load dev data with --dev-data. This will initialize the DB with 3 users, an imagery source, and a project.

If you wish to use a live copy of the CEO database instead of the dev data and you have a .dump file, run the following command:

clojure -M:build-db restore -f /path/to/ceo/database/ceo-db-<year>-<month>-<day>.dump

Performance Settings

If you want to improve the performance of your Postgresql server, one way is to visit the pgtune website and input your system settings. This will generate a set of configuration options that you can add to the postgresql.conf file in your system-specific Postgresql data directory.

If you would like to keep these settings separate for your own reference, you can add them to a file called “performance.conf” that you should place in your Postgresql data directory. You can then import these settings into postgresql.conf by adding this line to the end of that file:

include = 'performance.conf'

Note that the Postgresql server will need to be restarted for these changes to take effect.

Environment Variable Setup

In order for your command line shell to find the programs installed above, you will need to update your environment variables. Under GNU/Linux, *BSD, or MacOS X, you can set these through your shell’s .*rc or .*profile configuration files in your $HOME directory. In a Windows environment, you will need to set either the USER (for the current user only) or SYSTEM (for any user) environment variables under your system settings menu.

Step 1: Add JAVA_HOME

On a Unix machine using bash (e.g., GNU/Linux or MacOS X):

export JAVA_HOME=<jdk-installation-directory>

On a Unix machine using csh or tcsh (e.g., *BSD):

setenv JAVA_HOME <jdk-installation-directory>

On a Windows machine:

JAVA_HOME = C:\Program Files\Java\jdk-X.X.X

Replace X.X.X with the JDK version installed on your computer. In order to run CEO, your JDK version has to be either 11 or 17.

Step 2: Update PATH

On a Unix machine using bash (e.g., GNU/Linux or MacOS X):

export PATH=$PATH:$JAVA_HOME/bin

On a Unix machine using csh or tcsh (e.g., *BSD):

setenv PATH $PATH\:$JAVA_HOME/bin

On a Windows machine, add these entries to the PATH environment variable:

%JAVA_HOME%\bin
C:\Program Files\PostgreSQL\X\lib
C:\Program Files\PostgreSQL\X\bin
C:\Program Files\7-Zip

Replace X with the PostgreSQL version installed on your computer.

Configuration

Copy config.default.edn to config.edn and update any settings that you wish. The base set of defaults should be sufficient to run CEO in a development setting.

Python dependencies

Use pip to install dependencies. The python / clj interop does not support virtual environments.

pip install -r requirements.txt
pip install earthengine-api --upgrade

Development Web Server

Google Earth Engine authentication

You will need a Google Earth Engine account to use the GeoDash. To apply for a GEE account, visit https://signup.earthengine.google.com/#!/. Once you receive an account, you can store local authentication information with:

earthengine authenticate

This command will open a window in your browser containing a key that you should copy and paste back into your terminal. You do not need to update the :gee section of config.edn if you use this method. If the above method is giving you issues, try following https://developers.google.com/earth-engine/guides/python_install#expandable-2.

Modifying hosts file

Add the following line to your hosts file. This local URL should be used for local development in place of localhost due to CORS settings on CEO’s Mapbox account.

127.0.0.1 local.collect.earth

Create accounts

To be able to crate accounts without having to send email for confirmation, set :auto-validate? of :mail to true in config.edn.

There can only be one superuser account with user_uid of 1 adminstrator flag set to true. Activating this flag for other users can cause their queries to sometimes not return the full result, as some of those queries assume only one root superuser to be existing.

Compile and run

To compile and run the web application, navigate to the top level project directory and run:

npm install
npm run vite-dev

Now, in a separate terminal window (because webpack needs to be running in the background), run the following commands:

npm run server-dev

The website will then be available at http://localhost:8080 unless a port is specified. These can also be configured using the :server section in your config.edn file. When using the clojure -M:server start command (which is what npm run server-dev is an alias for) an http port can be specified with -p and an https port can be specified with -P. In dev mode, server-side exceptions will be displayed in the browser and JavaScript source files will be reloaded whenever you refresh the page.

Going forward you will usually only need to run npm run webpack-dev and npm run server-dev (still in separate terminals) to get your dev environment set up. Watch for updates to SQL files or package.json and run npm run build-db-functions or npm install respectively.

Checking for Reflection Warnings

From the top level project directory run:

clojure -M:check-reflection

This will emit warnings when reflection is needed to resolve Java method calls or field accesses. To resolve any that appear, add type hints to your code. Resolving all reflection warnings can improve system performance.

Production Web Server

Sessions

It is very important to change the default :session-key in config.edn. This key is used to encrypt user session data and should be unique to each deployment. The key must be exactly 16 characters long.

Email Server

To set up the email server for system emails, open the “config.edn” file in the root directory of the application. Edit the default EDN object containing server details to the file, replacing the values with your own.

Enabling HTTPS (optional)

To enable HTTPS from within the server, view the Triangulum HTTPS page for further instructions on enabling HTTPS.

Google Earth Engine service account

For production it is recommended that you use a service account with a key file. You can obtain your key file by logging into your service account, navigating to the account menu, and clicking “Create key > JSON”. Then, download that JSON key file and place it in the root directory of CEO. Set the email for your service account and key path in the :gee section of config.edn.

:gee {:ee-account  "[email protected]"
      :ee-key-path "ceo-gee-key.json"}

Launching the Web Server

To compile and run the web application, navigate to the top level project directory and run:

npm install
npm run webpack-prod
clojure -M:build-db functions -d ceo
clojure -M:server start -m [dev|prod] [-p 8080] [-P 8443] [-r]

The website will then be available at http://localhost:8080 unless a port is specified. These can also be configured using the :server section in your config.edn file. An http port can be specified with -p and an https port can be specified with -P. In dev mode, server-side exceptions will be displayed in the browser and Clojure source files will be reloaded whenever you refresh the page. These features are disabled in prod mode. If -m is unspecified, it will default to prod mode.

Running the Web Server as a System Service

View the Triangulum Systemd page for further instructions on enabling the app as a system service.

Maintaining Daily Logs

By default the server will log to standard out. If you would like to have the system log to YYYY-DD-MM.log, use the “-o path” option to specify an output path. You can either specify a path relative to the top level directory of this repository or an absolute path on your filesystem. The logger will keep the 10 most recent logs.

Using the Announcement Banner

On each page load clojure will read the value of announcement.txt. If text is found, the value will be inserted into a HTML element that displays as a red banner at the top of the page. To add a new announcement, edit announcement.txt and add a new message. To remove the announcement, edit announcement.txt and remove all text.

Contact

Authors:

• David S. Saah (SIG)
• Matt Spencer (SIG)
• Gary W. Johnson (SIG)
• Billy Ashmall (NASA)
• Roberto Fontanarosa (FAO)
• Oliver Baldwin Edwards (SIG)
• Githika Tondapu (NASA)
• Stefano Ricci (FAO)
• Alfonso SanchezPausDiaz (FAO)
• Biplov Bhandari (SIG)

License and Distribution

Copyright © 2016-2022 FAO.

Collect Earth Online is distributed by FAO under the terms of the MIT License. See LICENSE in this directory for more information.