Deafrica Scripts

💻 Technologies

• PYTHON 3.9
• Docker
• GDAL
• Boto3
• rio-cogeo

📝 Description

Scripts for the Digital Earth Africa project which provide utilities and solutions for data acquisition and monitoring functions to assist with data validation and quality assurance.

Utilities

Various resources are included with the following structures:

• deafrica/data: data acquisition and processing utilities
• deafrica/monitoring: data monitoring and reporting utilities
• scripts: Miscellaneous utilities and sample notebooks

Data Acquisition

Data acquisition scripts are available for the following resources:

• ALOS PALSAR: Advanced Land Observation Satellite Phased Array type L-band Synthetic Aperture Radar

• CHIRPS: Climate Hazards Group InfraRed Precipitation with Station data

• Copernicus GLS: Copernicus Global Land Service Land Cover 100m

• Copernicus CCI: 300m Land Cover Classification ESA Climate Change Initiative, Copernicus

• GMW: Script generates mangrove extents mapped by Global Mangrove Watch over Africa in cloud optimized rasters at 0.001 0.001 spatial resolution.

Monitoring

The following elements outline available monitoring utilities and their functions:

• Check dead queues: Monitor DEAfrica dead queues, with an option of Slack alert.
• Landsat gap filler : Transfer scenes that are missing in DEAfrica from USGS S3 bucket. A list of missing scenes is generated by the gap report process and used to fill gaps.
• Landsat gap Report: Check differences across satellites Landsat 8, 7, and 5 between USGS bulk file and DEAfrica's inventory bucket. As per result, gap report will generate a JSON file within a list of missing and orphan scenes.
• Sentinel 2 gap filler: Transfer scenes that are missing in DEAfrica from Element-84 inventory bucket. A list of missing scenes is generated by the gap report process and used to fill those gaps.
• Sentinel 2 gap Report: Check differences between Element-84 inventory bucket and DEAfrica's inventory bucket. As per result, gap report will generate a JSON file within a list of missing and orphan scenes.

⚙️ Setup & Run

Clone the main project repository from Github.

$ git clone https://github.com/digitalearthafrica/deafrica-scripts.git

Building locally

$ cd deafrica-scripts
$ docker build -t Deafrica-Scripts

Download Image from the repository

$ docker pull digitalearthafrica/deafrica-tools

Local Development

Local installation and development will require development dependencies requirements.txt.

From the project root directory, use pip to install dependencies as follows:

$ pip install -r requirements.txt
$ pip install -e .

📝 Tests

Running tests will require development dependencies requirements-test.txt.

From the project root directory, use pip to install test dependencies as follows:

$ pip install -r requirements-test.txt
$ pytest .

⚙️ GitHub Actions

Any changes pulled to any branch will be tested and have the formation checked by black.

A Merge will result in the creation of a new test image on Docker and 45 min later replicated on our AWS ECS.

Copyright Deafrica - GitHub Developed by Rodrigo de Azevedo Carvalho - Linkedin - GitHub