Skip to content

Commit

Permalink
Update README.org
Browse files Browse the repository at this point in the history
  • Loading branch information
@RasmusBahbah
RasmusBahbah authored Oct 31, 2023
1 parent 4555103 commit 2bad252
Showing 1 changed file with 43 additions and 214 deletions.
257 changes: 43 additions & 214 deletions README.org
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,80 +3,84 @@

* Table of Contents :toc_2:noexport:
- [[#sentinel-3a-s3a-albedo-euro-data-cube-edc-processing-pipeline][Sentinel-3A (S3A) Albedo Euro Data Cube (EDC) Processing Pipeline]]
- [[#fetch-s3-olci--slstr-products][Fetch S3 OLCI & SLSTR products]]
- [[#process-with-snap][Process with SNAP]]
- [[#divide-area-into-tiles][Divide Area Into Tiles]]
- [[#fetch-s3a-olci--slstr-products-using-sentinelhub-sh][Fetch S3A OLCI & SLSTR products using SentinelHub (SH)]]
- [[#start-multiprocessing][Start Multiprocessing]]
- [[#scda--ice-mask][SCDA & Ice Mask]]
- [[#cams-aerosol][cams Aerosol]]
- [[#pysice][pySICE]]
- [[#mosaic][Mosaic]]
- [[#outputs][Outputs]]
- [[#debugging--testing][Debugging & Testing]]
- [[#development-environment][Development Environment]]
- [[#bash][Bash]]
- [[#development-environment][Development Environment]
- [[#python][Python]]
- [[#parallel][Parallel]]
- [[#grass][GRASS]]
- [[#gpt][GPT]]
- [[#snap][SNAP]]
- [[#misc-notes--code-snippets][Misc Notes & Code Snippets]]
- [[#generate-ice-mask][Generate ice mask]]
- [[#footprint][Footprint]]
- [[#regional-masks][Regional masks]]
- [[#post-processing-analysis-and-visualisation][Post-processing, analysis and visualisation]]

* Sentinel-3A (S3A) Albedo Euro Data Cube (EDC) Processing Pipeline

1. Fetch S3 OLCI & SLSTR products
2. Process OLCI & SLSTR using SNAP GPT
3. Process using pySICE
4. Build daily mosaic
1. Divide Area Into Tiles
2. Fetch S3A OLCI & SLSTR products using SentinelHub (SH)
3. Start Multiprocessing
4. Simple Cloud Detection Algorithm (SCDA) & Masking Non-Ice/Snow areas
5. Get Forecasted Aerosol data from Copernicus Atmospgeric Monotoring Service (cams)
6. Compute outputs with pySICE
7. Merge all tiles into daily mosaics
8. Load all daily mosiacis to one multidimensional NetCDF file
9. Upload to Dataverse and Thredds Server

The steps above are encapsulated in [[./S3_wrapper.sh]].
The steps above are encapsulated in [[./src/pysicehub.py]].

In more detail:

** Fetch S3 OLCI & SLSTR products
** Divide Area Into Tiles

+ Download OLCI EFR and SLSTR RBT products for a specified day with [[./dhusget_wrapper.sh]].
+ Every area is divided into 50km x 50km tiles with a 10 km buffer

** Process with SNAP
** Fetch S3A OLCI & SLSTR products

+ This step is a combination of running GPT on [[./S3.xml]] with [[./S3_proc.sh]].
+ Inputs: OLCI and SLSTR scenes
+ Outputs: A folder OLCI timestamp
+ Download OLCI and SLSTR products for each tile concurrently through SH
+ Donwloads the S3A scene with the lowest Sun Zenith Angle

** Start Multiprocessing

+ Use a specified number of cores (default is 12) to process each tile in parrallel '

** Simple Cloud Detection Algorithm (SCDA) & Masking Non-Ice/Snow areas

+ Using SLSTR, clouds in every tile are detected and masked
+ Using a predifened mask, all areas in the tile that are not snow or ice are masked

** Get Forecasted Aerosol data from Copernicus Atmospgeric Monotoring Service (cams)
+ Download raw forecasted aerosol optical depth at 550 nm and 670 nm for each tile
+ interpolate and regrid to SICE 500m grid
+ compute ångstrøm paramter

** pySICE

+ Run [[./sice.py]] passing in one of the folders generated in the previous step.
+ *WARNING*: This step is slow, and may take > 24 hours.
+ Run [[./pysice.py]] on all tiles

** Mosaic

+ As implemented in [[./dm.sh]] and [[./dm.grass.sh]], for each day
+ Combine all the files from to form a mosaic
+ Mask out clouds, then...
+ When scenes overlap, use minimum SZA
+ Merge all tiles into moasic

** Outputs
| File Name | Description | Units |
|-------------------------------+------------------------------------------------------------------------+-------------|
| grain_diameter | | mm |
| snow_specific_surface_area | | m^{2}kg^{-1}|
| ndsi | normalized difference snow index | |
| ndbi | normalized difference blue ice index | |
| auto_cloud | in development | |
| sza | sun zenith angle | |
| vza | viewing zenith angle | |
| saa | sun azimuth angle | |
| vaa | viewing azimuth angle | |
| AOD_550 | Aerosol Optical Depth (550nm) | m |
| ANG | ångstrøm parameter | |
| albedo_bb_planar_nir | near infrared broadband planar albedo | 700-2400 nm |
| albedo_bb_planar_sw | shortwave broadband planar albedo | 300-2400 nm |
| albedo_bb_planar_vis | visible broadband planar albedo | 300-700 nm |
| albedo_bb_spherical_nir | near infrared broadband spherical albedo | 700-2400 nm |
| albedo_bb_spherical_sw | shortwave broadband spherical albedo | 300-2400 nm |
| albedo_bb_spherical_vis | visible broadband spherical albedo | 300-700 nm |
| Oa01..21_reflectance | top of the atmosphere reflectance (OLCI bands 1 through 21) | |
| r_TOA_01..21_reflectance | top of the atmosphere reflectance (OLCI bands 1 through 21) | |
| rBRR_01..21 | bottom of the atmosphere surface reflectance (OLCI bands 1 through 21) | |
| albedo_spectral_planar_01..21 | spectral planar albedo (OLCI bands 1 through 21) | |
| diagnostic_retrieval | See next table. | |
| isnow | See next table. | |


| Diagnostic Code | Description |
Expand All @@ -90,193 +94,18 @@ In more detail:
| 104 | grain_diameter < 0.1, no retrieval, potential cloud flag |
| -n | impossible to solve polluted snow albedo equation at band n |

* Debugging & Testing

Change the year and day for loops in [[./S3_wrapper.sh]] to one year and one day. E.g.

#+BEGIN_SRC bash :results verbatim
for year in 2017; do # one year
for doy in 227 180; do # two example days
# loop contents
done
done
#+END_SRC

#+RESULTS:

* Development Environment
:PROPERTIES:
:header-args:bash+: :eval no-export
:END:

This work is developed with the following software versions

** Bash
#+BEGIN_SRC bash :results verbatim :exports both
bash --version
#+END_SRC

#+RESULTS:
: GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
: Copyright (C) 2016 Free Software Foundation, Inc.
: License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
:
: This is free software; you are free to change and redistribute it.
: There is NO WARRANTY, to the extent permitted by law.

** Python

Run the following using [[./SICE.yml]] in order to set up an identical Anaconda Python environment.
Run the following using [[./environement.yml]] in order to set up an identical Anaconda Python environment.

#+BEGIN_SRC bash :results verbatim
conda env create -f SICE.yml
conda env create -f environement.yml
#+END_SRC

And then run =conda activate SICE= to activate the SICE environment.

=ResolvePackageNotFound= error can be raised. In that case, run =conda env export --no-builds > environment.yml= instead.


** Parallel

#+BEGIN_SRC bash :results verbatim :exports both
parallel --version
#+END_SRC

#+RESULTS:
#+begin_example
GNU parallel 20161222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: http://www.gnu.org/software/parallel

When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.
#+end_example

** GRASS
#+BEGIN_SRC bash :results verbatim :exports both
grass --version 2>&1
#+END_SRC

#+RESULTS:
#+begin_example
GRASS GIS 7.4.0

Geographic Resources Analysis Support System (GRASS) is Copyright,
1999-2018 by the GRASS Development Team, and licensed under terms of the
GNU General Public License (GPL) version >=2.

This GRASS GIS 7.4.0 release is coordinated and produced by
the GRASS Development Team with contributions from all over the world.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.

#+end_example

** GPT

#+BEGIN_SRC sh :results verbatim :exports both
~/local/snap/bin/gpt --diag
#+END_SRC

#+RESULTS:
#+begin_example
SNAP Release version 7.0
SNAP home: /home/kdm/local/snap/bin//..
SNAP debug: null
SNAP log level: null
Java home: /home/kdm/local/snap/jre
Java version: 1.8.0_202
Processors: 8
Max memory: 18.7 GB
Cache size: 1024.0 MB
Tile parallelism: 8
Tile size: 512 x 512 pixels

To configure your gpt memory usage:
Edit snap/bin/gpt.vmoptions

To configure your gpt cache size and parallelism:
Edit .snap/etc/snap.properties or gpt -c ${cachesize-in-GB}G -q ${parallelism}
#+end_example

** SNAP

SNAP extensions used in this toolchain are listed in [[SNAP_extensions.txt]].

* Misc Notes & Code Snippets
:PROPERTIES:
:header-args:bash+: :eval no
:END:

** Generate ice mask

+ Use the icemask from BedMachine v3

#+BEGIN_SRC bash :results verbatim :eval no
grass72 -c EPSG:3413 ./Gtmp
r.in.gdal input=NetCDF:~/data/Greenland/Morlighem_2017/BedMachineGreenland-2017-09-20.nc:mask output=icemask

g.region raster=icemask
g.region res=500 -ap
g.region zoom=icemask

d.mon start=wx0
d.erase
d.rast icemask

r.mapcalc "mask = if(icemask == 4, null(), icemask)" --o
d.rast mask
g.region zoom=mask

r.out.gdal -c -m input=mask output=mask.tif type=Byte createopt=COMPRESS=DEFLATE --o
exit
trash Gtmp
#+END_SRC
*** Buffered ice mask
#+BEGIN_SRC bash :results verbatim
grass -c mask.tif ./Gtmp
r.in.gdal input=mask.tif output=mask
# 50 cells = 25 km @ 500 m
r.mapcalc "ice = if(mask == 2, 1, null())"
r.grow input=ice output=ice_grow radius=50 new=1

r.null ice_grow null=100
r.clump input=ice_grow output=clumps
r.stats -c clumps sort=asc
for ID in $(r.stats -c clumps sort=asc | head -n7 | cut -d" " -f1); do
r.mapcalc "ice_grow = if(clumps == ${ID}, 1, ice_grow)" --o
done
r.null ice_grow setnull=100

r.out.gdal input=ice_grow output=ice_mask_buffer.tif format=GTiff type=Byte createopt="COMPRESS=DEFLATE"
exit
trash Gtmp
#+END_SRC


** Footprint

Get GL outline by drawing in Google Earth, export KML, then:

#+BEGIN_SRC bash :results verbatim :eval no-export
ogrinfo -al GL_outline.kml | grep LINESTRING | sed s/\ 0//g
#+END_SRC

** Regional masks

Regional masks stored in the [[https://github.com/GEUS-SICE/SICE/tree/master/masks][masks]] folder are clips from the
[[https://www.esa-landcover-cci.org/?q=node/197][ESA global Land Cover products]] (download [[https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-land-cover?tab=form][here]]). Masks were downloaded for 2018 and are available
at 300m and 1km resolutions ({region}_300m.tif and {region}_1km.tif, respectively). A description of the 22 [[https://www.esa-landcover-cci.org/?q=node/197][ESA global Land Cover products]] can be found [[https://www.esa-landcover-cci.org/?q=webfm_send/84][here]].

* Post-processing, analysis and visualisation

Post-processing, analysis and visualisation tools have been implemented in the [[https://github.com/GEUS-SICE/SICE-toolbox][SICE toolbox]].

0 comments on commit 2bad252

Please sign in to comment.