This repository contains scripts for downloading, preprocessing, visualization, model creation and analysis for irrigation detection purpose. Main part will be implementing of irrigation detection method from Irrigation Events Detection over Intensively Irrigated Grassland Plots Using Sentinel-1 Data in V1. and from Near Real-Time Irrigation Detection at Plot Scale Using Sentinel-1 Data in V2.
This repository is under construction and functions may not work or be whack explained.
Most functions used inside jupyter_notebooks/... are stored in irrigation_detection.py and tools.py .
Google Earth Engine mainly for processing and downloading Satellite data. You can apply here
International Soil Moisture Network to get in-situ soil moisture time series data. You can apply here
API's used:
earthengine-api | Python and JavaScript client libraries for calling the Google Earth Engine API
keras | Deep learning API running on top of the machine learning platform TensorFlow
01_SSM_1km_V1 Product Source: World
Load daily .nc files in nested directories as xarray. Subset and mask out unrelevant regions. Apply conversation from digital to physical values.
02_RADOLAN_1km Product Source: Germany
Download Radolan SF product from FTP server. Exctract files. Load extracted files to xarray. Subset data. Apply conversation from digital to physical values.
03_NDVI_10Dmax_1km Product Source | Europe
Download data from ftp server after place an free order. Extract archieves. Load extracted files to xarray. Subset data. Apply conversation from digital to physical values. Find irrigation period.
04_LUCAS_TOPSOIL Product Source: Europe
Preprocess LUCAS TOPSOIL 2015 and 2009 Datasets. Merge to Geopandas GeoDataFrame. Subset and Export data.
05_BDFL5 Product Source: Hessen - Product Source | RLP
Preprocess Soil Information dataset. Spatial join with LUCAS TOPSOIL data. Subset and Export data.
06_Evapo_rel_pot Product Source: Germany
Download potentiel and real Evaporation. Transform projection with Gdal. Load extracted files to xarray. Subset data. Apply conversation from digital to physical values
07_Sentinel_1_grd Product Source: World
Use Earth Engine Cloudcomputing. Get Sentinel - 1 GRD VV Mean for geometry while masking high NDVI values. Construct Geopandas GeoDataFrame and add all Information from filename.SWI map expors. Extract data for plot and grid size
Query data with GEE and geemap to get 10m plot time series and 50m grid(10x10km) time series data for different Obversation modes of Sentinel 1. Get NDVI, SWI, VV, Angle. Download as tif.
PYSMM python module to extract point or grid soil moisture information with GEE.
Irrigation detection with data from 07_Sentinel_1_grd notebook
Acces all exported datasets from previous notebooks. Actual: LTS, BDFL5, NDVI10Dmax, RADOLAN, SSM, EVAPO_R, EVAPO_P, ... . Subset data. Preprocess and merge
ISMN Dashboard for interactive visualization of Sentinel 1&2 and soil moisture data. Load data from Irrigation_Detection_V1 notebook. Split into equal satellite groups. Matplotlib and hvplot plotting.
Extract TimeSeries data from ismn stations. Find corresponding Sentinel 1, Sentinel 2 and ERA5 pixels and extract all information to geojson file. Merge data and add in-situ soil moisture values closest to satellite derived data.
Use Keras and Tensorflow to train a MLP regression model with Sentinel 1, Sentinel 2 and ERA 5 data. Visualize and Analyse trained model with test data.
Extract features from pdf file
GEE stuff from first project. Visualizations
Statistical and machine learning Stuff from first project. Seasonal Mean, Standart Deviation, Histogram, Correlation, Covariance, Spearman, Pearson, Cross correlation, Windowed time lagged cross correlation, rolling window, Dynamic Time Warping, Clustering, precipitation analysis, Seasonal Patterns, show measurements on map, univariate description, bivariate description, Preprocessing, geographical fescription
clustering images to get soil data from image
Matrix Profile, Clustering, Biclustering, Clustering Time Series, Stump, statistical stuff
Google Earth Engine | Geopsatial big data computing with many datasets
geojson.io | Tool for draw, view, and sharing geometrical features online
georeferencer | Online Georeferencing Side by Side
TablesGenerator.com | LaTeX and HTML table generator
Overleaf | Online collaborative writing and publishing tool with LaTeX enviroment
transfer.sh | Easy file sharing from the command line
Geoportal Hessen | raumbezogenen Daten der Geodateninfrastruktur Hessen (GDI-HE)
DWD Opendata FTP Server | climate enviromentl data Germany
Earth Engine Data Catalog | Climate, Weather, Imagery and Geophysical datasets
Copernicus Open Access Hub | Sentinel Data Download
OpenEO | API to connect R, Python, JavaScript clients to big Earth observation cloud back-ends
THEIA | Products and Algorithms for Land Surfaces
irpi hydrologie | Satellite and Ground Data, Code
Bodeninformationen | Hessen - HLNUG
Soil Moisture Network ISMN | global in-situ soil moisture database
THEIA IRSTEA SOIL MOISTURE CATALOG | radar signal inversion algorithm uses neural networks
Deep-learning on satellite imagery wiki
IDB Index DataBase | A database for remote sensing indices
STAC Index | Spatio Temperal Asset Catalog
Sentinel Online | Official Sentinel Missions Homepage
Irrigation+ | quantitative, accurate and routinely monitoring of irrigation
geemap | interactive mapping with Google Earth Engine, ipyleaflet and ipywidgets
xarray | N-D labeled arrays and datasets in Python
pandas | data analysis and manipulation tool
Geopandas | geospatial data handling
satpy | reading, manipulating, and writing data from satellite instruments
TensorFlow | open source library to help you develop and train ML models
Keras | High level API for TensorFlow
STUMPY | Modern Time Series Analysis with Matrix Profile
tslearn | Machine Learning Tools for Time Series Data
tsfresh | Automatically Calculates a large number of time series characteristic features
sklearn | Framework for Machine Learning
sktime | Framework for Machine Learning with time series
GeostatsPy | Geostatistical library
PYSMM | Sentinel-1 soil-Moisture Mapping Toolbox
ismn | Reader for the data from the Soil Moisture Database(ISMN)
Cosmic-Ray Sensor PYthon tool (crspy) | Cosmic Ray Sensor data into soil moisture estimates
colorbars | matplotlib library for colorbars
matplotlib | creating static, animated, and interactive visualizations in Python
hvPlot | A high-level plotting API for the PyData ecosystem built on HoloViews
Yellowbrick | Extend Sklearn with Visualizations API
streamlit | Data scripts into sharable web apps
heroku | Build data-driven apps with fully managed data services
binder | Turn a Git repo into a collection of interactive notebooks