Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geospatial #110

Open
wants to merge 54 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
d632a80
added geospatial samples
sfc-gh-obielov Oct 20, 2022
55e861c
removed redundant files
sfc-gh-obielov Oct 20, 2022
d894027
nit
sfc-gh-obielov Oct 20, 2022
174551a
added st_intersection for geometry
sfc-gh-obielov Nov 4, 2022
22dc152
added makevalid function
sfc-gh-obielov Nov 7, 2022
4f20754
Update ST_TRANSFORM.sql
sfc-gh-obielov Nov 7, 2022
2416e93
updated makevalid
sfc-gh-obielov Nov 10, 2022
8a06501
nit
sfc-gh-obielov Nov 10, 2022
24dd137
added code for aggregated function for st_makeline
sfc-gh-obielov Nov 21, 2022
5ab55b5
updated the code
sfc-gh-obielov Nov 21, 2022
7622197
nit
sfc-gh-obielov Nov 21, 2022
61c281a
Merge branch 'main' into geospatial
sfc-gh-obielov Nov 22, 2022
4b16d66
renamed functions
sfc-gh-obielov Nov 22, 2022
8bddb68
reshuffled columns
sfc-gh-obielov Dec 2, 2022
df0e310
added reader for Shapefiles
sfc-gh-obielov Mar 8, 2023
206e11b
Create PY_MAKE2D.sql
sfc-gh-obielov Mar 23, 2023
0708a16
added four more geo functions
sfc-gh-obielov Mar 23, 2023
b7f48d3
merge
sfc-gh-obielov Mar 23, 2023
e2e9453
nit
sfc-gh-obielov Mar 23, 2023
b1fd408
added code sample for dynamic file access
sfc-gh-obielov Mar 30, 2023
c380309
Update PY_LOAD_GEOFILES.sql
sfc-gh-obielov Mar 30, 2023
252e21f
Update PY_LOAD_GEOFILES.sql
sfc-gh-obielov Mar 31, 2023
399df78
added more functions
sfc-gh-obielov Apr 14, 2023
bf2f0bc
Create PY_CONVEXHULL.sql
sfc-gh-obielov Apr 19, 2023
0b5088c
Create JS_TILEENVELOPE.sql
sfc-gh-obielov Apr 19, 2023
0612414
add handling for Null case
sfc-gh-obielov Aug 4, 2023
03230ba
Merge branch 'geospatial'
sfc-gh-obielov Aug 4, 2023
4dfcefc
nit
sfc-gh-obielov Aug 4, 2023
68469f5
Create PY_LOAD_MULTIPLE_SHAPEFILES.sql
sfc-gh-obielov Aug 28, 2023
adbe867
Create fixing_invalid_shapes.sql
sfc-gh-obielov Sep 27, 2023
1f94433
Update fixing_invalid_shapes.sql
sfc-gh-obielov Sep 27, 2023
cb65de1
Update fixing_invalid_shapes.sql
sfc-gh-obielov Sep 27, 2023
2e4423a
Update fixing_invalid_shapes.sql
sfc-gh-obielov Sep 27, 2023
546d669
Update fixing_invalid_shapes.sql
sfc-gh-obielov Sep 27, 2023
b2f3a15
Update fixing_invalid_shapes.sql
sfc-gh-obielov Sep 27, 2023
01e88b2
Update PY_LOAD_GEOFILES.sql
sfc-gh-obielov Nov 23, 2023
e0c750d
Update PY_LOAD_GEOFILES.sql
sfc-gh-obielov Nov 23, 2023
cbe5496
Merge branch 'main' of https://github.com/Snowflake-Labs/sf-samples
sfc-gh-obielov Nov 23, 2023
461cc73
Merge branch 'main' into geospatial
sfc-gh-obielov Nov 23, 2023
ca10fab
added new function
sfc-gh-obielov Nov 28, 2023
fec7aba
Create JS_POLYGON_TO_LINESTRING.sql
sfc-gh-obielov Dec 4, 2023
61a1523
Update JS_POLYGON_TO_LINESTRING.sql
sfc-gh-obielov Dec 4, 2023
7b284c3
Create basic_h3.py
sfc-gh-obielov Dec 15, 2023
50a7233
H3 streamlit sample
sfc-gh-obielov Feb 22, 2024
5da205b
Update main.py
sfc-gh-obielov Feb 22, 2024
c611145
Update PY_LOAD_GEOFILES.sql
sfc-gh-obielov Mar 13, 2024
f094fd6
updated functions
sfc-gh-obielov Mar 13, 2024
8626332
Create PY_COLLECTION_TO_POLYGONS.sql
sfc-gh-obielov Apr 1, 2024
4d8647e
Update basic_h3.py
sfc-gh-obielov Apr 1, 2024
3e98b41
Update basic_h3.py
sfc-gh-obielov Apr 1, 2024
d1620e7
Update basic_h3.py
sfc-gh-obielov Apr 1, 2024
a2cbab2
Merge branch 'main' into geospatial
sfc-gh-obielov Apr 1, 2024
c83c620
remove redundant files
sfc-gh-obielov Apr 1, 2024
6a8faf5
Update PY_COLLECTION_TO_POLYGONS.sql
sfc-gh-obielov Jul 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions geospatial/streamlit-apps/H3-SiS/basic_h3.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# This code of the Streamlit app visualizes data from the following dataset:
# https://app.snowflake.com/marketplace/listing/GZSVZ8ON6J/dataconsulting-pl-opencellid-open-database-of-cell-towers?search=opencellid
# The prerequisite for this app is to install that free dataset from the Marketplace to the 'opencellid' database

import streamlit as st
import pydeck as pdk
from snowflake.snowpark.context import get_active_session
Expand Down
3 changes: 3 additions & 0 deletions geospatial/streamlit-apps/H3-sample/main.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# This code of the Streamlit app visualizes data from the following dataset:
# https://app.snowflake.com/marketplace/listing/GZSVZ8ON6J/dataconsulting-pl-opencellid-open-database-of-cell-towers

import streamlit as st
import pandas as pd
import pydeck as pdk
Expand Down
40 changes: 40 additions & 0 deletions samples/geospatial/Python UDFs/PY_COLLECTION_TO_POLYGONS.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
CREATE OR REPLACE FUNCTION PY_COLLECTION_TO_POLYGONS(geo geography)
returns geography
language python
runtime_version = 3.8
packages = ('shapely')
handler = 'udf'
AS $$
from shapely import wkt
from shapely.ops import unary_union
from shapely.validation import make_valid
from shapely.geometry import shape, mapping
def udf(geo):
if not geo:
return None
try:
collection = shape(geo)
collection = make_valid(collection) if not collection.is_valid else collection

# Filter out non-Polygon/MultiPolygon geometries and flatten nested GeometryCollections
def flatten_geometries(geometry):
if geometry.geom_type in ["Polygon", "MultiPolygon"]:
return [geometry]
elif geometry.geom_type == "GeometryCollection":
return [sub_geom for geo in geometry.geoms for sub_geom in flatten_geometries(geo)]
else:
return []
geometries_checked = flatten_geometries(collection)
if not geometries_checked:
return None
dest_shape = unary_union(geometries_checked)
valid_shape = make_valid(dest_shape) if not dest_shape.is_valid else dest_shape
return mapping(valid_shape)
except Exception as e:
# Logging the exception 'e' for debugging purposes
return e
$$;

select PY_COLLECTION_TO_POLYGONS(to_geography('GEOMETRYCOLLECTION(POINT(-91.72073364257811 41.77182378456081),
POINT(-96.57995223999022 37.66099365286695),POLYGON((-86.36249542236327 40.02077884854819,-84.54580307006836 36.46298689866559,
-90.70569992065428 36.01300517087813,-86.36249542236327 40.02077884854819)))'))
12 changes: 6 additions & 6 deletions samples/geospatial/Python UDFs/PY_LOAD_GEOFILES.sql
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,7 @@
-- parameters:
-- - Snowflake share of where the file is located
-- - Name of the file, including path
-- - GEOFILE_NAME - The name of the GEO file inside the package zip that needs to
-- loaded
-- - GEOFILE_NAME - The name of the GEO file inside the package zip that needs to be loaded

-- === UDTF Code ===
CREATE OR REPLACE FUNCTION PY_LOAD_GEOFILE(PATH_TO_FILE string, filename string)
Expand All @@ -33,6 +32,7 @@ AS $$
from shapely.geometry import shape
from snowflake.snowpark.files import SnowflakeFile
from fiona.io import ZipMemoryFile
import fiona
class GeoFileReader:
def process(self, PATH_TO_FILE: str, filename: str):
fiona.drvsupport.supported_drivers['libkml'] = 'rw'
Expand All @@ -49,25 +49,25 @@ $$;
create or replace table GEOLAB.GEOGRAPHY.TABLE_NAME as
SELECT properties:Field_1::string as field_1,
properties:Field_2::string as Field_2,
to_geography(wkb, True) as geometry FROM table(PY_LOAD_GEOFILE(build_scoped_file_url(@stage_name, 'archive_name.zip'), 'file_name.shp'));
to_geography(wkb, True) as geometry FROM table(PY_LOAD_GEOFILE(build_scoped_file_url(@stage_name, 'ZIP_FILE_NAME.zip'), 'GEOFILE_NAME.shp'));

-- === Example execution (MapInfo TAB File) ===
create or replace table GEOLAB.GEOGRAPHY.TABLE_NAME as
SELECT properties:Field_1::string as field_1,
properties:Field_2::string as Field_2,
to_geography(wkb, True) as geometry FROM table(PY_LOAD_GEOFILE(build_scoped_file_url(@stage_name, 'archive_name.zip'), 'file_name.tab'));
to_geography(wkb, True) as geometry FROM table(PY_LOAD_GEOFILE(build_scoped_file_url(@stage_name, 'ZIP_FILE_NAME.zip'), 'GEOFILE_NAME.tab'));

-- === Example execution (Google Earth KML File) ===
select
to_geography(wkb, True) as geometry,
properties:Name::string as Name,
properties:altitudeMode::string as altitudeMode
from table(PY_LOAD_GEOFILE(build_scoped_file_url(@tmobile, 'archive_name.zip'), 'file_name.kml'));
from table(PY_LOAD_GEOFILE(build_scoped_file_url(@tmobile, 'ZIP_FILE_NAME.zip'), 'GEOFILE_NAME.kml'));

-- === Example execution (OGC GeoPackage) ===
-- Note: A specific layer in the .gpkg file is opened by specifying layer name in UDTF `with zip.open(filename,layer=layername) as collection:`
SELECT
properties:Field_1::string as field_1,
properties:Field_2::string as Field_2,
to_geography(wkb, True) as geometry
FROM table(PY_LOAD_GEOFILE(build_scoped_file_url(@stage_name, 'archive_name.zip'), 'file_name.gpkg'));
FROM table(PY_LOAD_GEOFILE(build_scoped_file_url(@stage_name, 'ZIP_FILE_NAME.zip'), 'GEOFILE_NAME.gpkg'));
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@
-- required the timeout to be increased to over 10 minutes.

-- === Parameters ===
-- - ZIP_FILE_NAME - The scoped URL to the zipped file on the Snowflake share and is
-- - PATH_TO_FILE - The scoped URL to the zipped file on the Snowflake share and is
-- generated using the build_scoped_file_url function. This function requires two
-- parameters:
-- - Snowflake share of where the file is located
-- - Name of the file, including path
-- - @stage_name - Snowflake stage of where the file is located
-- - ZIP_FILE_NAME - Name of the file, including path

-- === UDTF Code ===
CREATE OR REPLACE FUNCTION SHAPE_MULTI_FILE(PATH_TO_FILE string)
Expand Down
18 changes: 0 additions & 18 deletions samples/geospatial/Python UDFs/ST_INTESECTION.sql

This file was deleted.

17 changes: 0 additions & 17 deletions samples/geospatial/Python UDFs/ST_MAKELINE.sql

This file was deleted.

24 changes: 0 additions & 24 deletions samples/geospatial/Python UDFs/ST_TRANSFORM.sql

This file was deleted.

18 changes: 0 additions & 18 deletions samples/geospatial/Python UDFs/ST_UNION_AGG.sql

This file was deleted.