Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RASTERIZE A SPATVECTOR MAKES R SESSION TO ABORT #1369

Closed
anjelinejeline opened this issue Dec 8, 2023 · 11 comments
Closed

RASTERIZE A SPATVECTOR MAKES R SESSION TO ABORT #1369

anjelinejeline opened this issue Dec 8, 2023 · 11 comments

Comments

@anjelinejeline
Copy link

anjelinejeline commented Dec 8, 2023

Hi I am using terra 1.7-46
I am trying to rasterize a large point dataset (worldwide coverage) but when I run the function rasterize the R session aborts for a fatal error.
I would like to have a raster with cell value equals to the number of points
Could you please fix this bug?

Thank you !

df<-read.csv("df.csv")[,-1]

df<-st_as_sf(df,coords=c("x", "y"), crs="+proj=moll +lon_0=0 +x_0=0 +y_0=0")

data(World)
World<-World |> 
  filter(continent %in% c("Asia","Africa","Europe","Oceania","South America","North America")) |> 
  st_transform("+proj=moll +lon_0=0 +x_0=0 +y_0=0")

# Create a raster template of the World with 1kmx1km resolution

r <- terra::rast(World)
terra::res(r) <- 1000


Vect<-terra::vect(df)

Raster<-terra::rasterize(Vect, r, fun="count", field="ID")
@anjelinejeline
Copy link
Author

anjelinejeline commented Dec 8, 2023

If I change the above code with the following it runs but an empty raster is returned

r <- terra::rast(World,ncols=1000,nrows=1000,
                 crs="+proj=moll +lon_0=0 +x_0=0 +y_0=0")

Vect<-terra::vect(df)

Raster<-terra::rasterize(Vect,r, fun="count",field="ID")

terra::plot(Raster)

@Rapsodia86
Copy link

Is "World" a point dataset?
If so, use "length" instead of "counts"; see: https://rdrr.io/cran/terra/man/rasterize.html

fun - summarizing function for when there are multiple geometries in one cell. For lines and polygons you can only use "min", "max", "mean", "count" and "sum" For points you can use any function that returns a single number; for example mean, length (to get a count), min or max

@rhijmans
Copy link
Member

rhijmans commented Dec 9, 2023

I cannot help with this because I do not have df nor World. And if I make a similar example that works fine.

library(terra)
r <- project(rast(ymin=-60), "+proj=moll +lon_0=0 +x_0=0 +y_0=0")
# res(r) <- 1000
# easier to see the results with a larger resolution
res(r) <- 500000

s <- seq(-500000, 5000000, 1000)
df <- data.frame(x=s, y=s)
v <- vect(df, c("x", "y"))
x <- terra::rasterize(v, r, fun="count")

@anjelinejeline
Copy link
Author

anjelinejeline commented Dec 9, 2023

Hi @rhijmans @Rapsodia86 the World shapefile is from the package tmap.
The df is point dataframe in sf format

library("tmap") # World shapefile
data(World)
World <- World[World$continent %in% c("Asia","Africa","Europe","Oceania","South America","North America"), ]
World <- st_transform(World, "+proj=moll +lon_0=0 +x_0=0 +y_0=0")

r <- terra::rast(World, resolution=1000)

The idea is to have a grid of **1kmx1km **covering the entire World and then count the points falling in each grid...
I also tried with rasterizeGeom
but once again the R session aborts by itself...
Can you help me understand what I am doing wrong, is the way I create the raster of the World fine? and if the terra package can work with such a big grid of the world?

Vect<-terra::vect(df)

Raster<-terra::rasterizeGeom(Vect, r,
                                      fun="count",unit="m",
                                      resolution=c(1000,1000))

image

Thank you
Angela

@rhijmans
Copy link
Member

rhijmans commented Dec 9, 2023

I have already shown above how you can do this (in a simpler way). If that does not work for you, it may be related to "df". But I do not have that, and you do not even show what it looks like. With the information at hand I cannot say what is going on.

@anjelinejeline
Copy link
Author

Hi @rhijmans ,
Please find attached df...
It's quite big but I believe that the terra package should be handle it .. so I don't know why the rasterization does not work
df.csv

@HassanMasoomi
Copy link

HassanMasoomi commented Dec 10, 2023

It works fine for me ("df.csv" is the one you provided above)...

library(sf)
library(terra)
library(tmap)
library(data.table)

# sample raster
data(World)
World <- World[World$continent %in% c("Asia","Africa","Europe","Oceania","South America","North America"),]
World <- sf::st_transform(World, "+proj=moll +lon_0=0 +x_0=0 +y_0=0")
sample_raster <- terra::rast(World, resolution = 1000)

# assuming that x & y provided in df are already in the same crs as sample_raster
df <- fread("df.csv")
df_vect <- vect(df, geom = c("x", "y"), 
                crs = "+proj=moll +lon_0=0 +x_0=0 +y_0=0")
final_raster <- terra::rasterize(df_vect, sample_raster, fun = "count")

# another method (maybe better if your df is huge?!!); result will be on memory!
# 1: find the cell each row in located in
df[, cell := cellFromXY(sample_raster, .SD[, .(x, y)])]
# 2: summarize df by cell
new_df <- df[, .(count = .N), keyby = cell]
# 3: set each cell values in your raster
final_raster2 <- sample_raster
final_raster2[new_df$cell] <- new_df$count

@rhijmans , regarding the last line in the code above, is there any function to set the values of a bunch of cells for a raster (a function with filename argument)?

@anjelinejeline
Copy link
Author

Hi @HassanMasoomi
Thank you !!
I was able to create it using the first method .. the R session did not abort this time around.
If I look at the statistics using global the values seem fine but if I try to plot it the plot is empty .. I believe that it may be due to the fine resolution..
Have you tried to plot it ?
I'd like to visualize the results somehow ..
Any advice on how to do that ?

Thank you

@HassanMasoomi
Copy link

I don't know your goal of using the raster with that high resolution here. But, maybe you want to explore tmap and plot you results as points:

library(sf)
library(terra)
library(tmap)
library(data.table)

# sample raster
data(World)
World <- World[World$continent %in% c("Asia","Africa","Europe","Oceania","South America","North America"),]
World <- sf::st_transform(World, "+proj=moll +lon_0=0 +x_0=0 +y_0=0")
sample_raster <- terra::rast(World, resolution = 1000)

# assuming that x & y provided in df are already in the same crs as sample_raster
df <- fread("df.csv")

# 1: find the cell each row in located in
df[, cell := cellFromXY(sample_raster, .SD[, .(x, y)])]
# 2: summarize df by cell
new_df <- df[, .(count = .N), keyby = cell]

# plot
new_df[, x := xFromCell(sample_raster, cell)]
new_df[, y := yFromCell(sample_raster, cell)]
new_df_vect <- vect(new_df, geom = c("x", "y"),
                    crs = "+proj=moll +lon_0=0 +x_0=0 +y_0=0")
tmap_mode("view")
tm_shape(sf::st_as_sf(new_df_vect)) + tm_dots(size = "count", col = "darkred", 
                                              alpha = 0.75, id = "count")

@anjelinejeline
Copy link
Author

@HassanMasoomi
Thank you but I would like to plot it as a raster grid .. is there no way I can visualise it as a raster ?

@anjelinejeline
Copy link
Author

Hi @rhijmans it now works ... I changed the machine .. I guess that the crash was due to my laptop ..
I have just another question ..

If I want to assign to each cell of my sample raster [created with sample_raster=rast(World, resolution = 1000)]
not the count of points of a spatial vector x but rather the sum of an attribute (myattribute) of all points within that specific raster cell ..should I do like this:

# Sum the variable of interest
  ras=rasterize(x,sample_raster,field="myattribute", fun="sum")

I tried and it seems that it works but I would like to double check with you whether this is fine...

Best wishes
Angela

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Dec 10, 2024
# version 1.7-83

## bug fixes

- `flip(direction="vertical")` failed in some cases
  [#1518](rspatial/terra#1518) by Ed Carnell

- `zonal(as.raster=TRUE)` failed when the zonal raster was categorical
  [1514](rspatial/terra#1514) by Jessi L
  Brown

- `distance<data.frame,data.frame>` and `<matrix,matrix>` ignored the
  unit
  argument. [#1545](rspatial/terra#1545) by
  Wencheng Lau-Medrano

- NetCDF files with month time-step encode from 0-11 made R crash
  [#1544](rspatial/terra#1544) by Martin
  Holdrege

- `split<SpatVector>` only worked well if the split field was of type
  character. [#1530](rspatial/terra#1530) by
  Igor Graczykowski

- `gridDist` (and probably some other methods) emitted a "cannot
  overwrite existing file" error when processing large datasets
  [#1522](rspatial/terra#1522) by Clare
  Pearson

- `terrain` did not accept multiple variables
  [#1561](rspatial/terra#1561) by Michael
  Mahoney

- `rotate` was vulnerable to an integer overflow
  [#1562](rspatial/terra#1562) by Sacha
  Ruzzante

- `getTileExtents` could return overlapping tiles or tiles with gaps
  due to floating point
  imprecision. [#1564](rspatial/terra#1564)
  by Michael Sumner


## enhancements

- `as.list<SpatRasterDataset>` sets the names of the list
  [#1513](rspatial/terra#1513)

- a SpatVectorCollection can now be subset with its names; and if made
  from a list it takes the names from the list.
  [1515](rspatial/terra#1515) by jedgroev

- argument `fill_range` to plot<SpatRaster> and `plot<SpatVector>` to
  use the color of the extreme values of the specified range
  [#1553](rspatial/terra#1553) by Mike
  Koontz

- plet<SpatRaster> can now handle rasters with a "local" (Cartesian)
  CRS. [#1570](rspatial/terra#1570) by
  Augustin Lobo.

## new

- `map-region` returns the coordinates of the axes position of a map
  created with `plot<Spat*>`
  [https://github.com/rspatial/terra/issues/1517](https://github.com/rspatial/terra/issues/1517)
  by Daniel Schuch

- `polys<leaflet>` method
  [#1543](rspatial/terra#1543) by Márcia
  Barbosa

- `plot<SpatVectorCollection>` method
  [#1532](rspatial/terra#1532) by jedgroev

- `add_mtext` to add text around the margins of a
  map. [#1567](rspatial/terra#1567) by
  Daniel Schuch

# version 1.7-78

Released 2023-05-22

## bug fixes

- `writeVector` and `readVector` better handle empty geopackage layers
  [#1426](rspatial/terra#1426) by Andrew
  Gene Brown.

- `writeCDF` only wrote global variables if there was more than one
  [#1443](rspatial/terra#1443) by Daniel
  Schlaepfer

- `rasterize` with "by" returned odd layernames
  [#1435](rspatial/terra#1435) by Philippe
  Massicotte

- `convHull`, `minCircle` and `minRect` with a zero-row SpatVector
  crashed R [#1445](rspatial/terra#1445) by
  Andrew Gene Brown

- `rangeFill` with argument `circular=TRUE` did not work properly
  [#1460](rspatial/terra#1460) by Alice

- `crs(describe = TRUE)` returned an mis-ordered extent
  [#1485](rspatial/terra#1485) by Dimitri
  Falk

- `tapp` with a custom function and an index like "yearmonths" could
  shift time for not considering the time
  zone. [#1483](rspatial/terra#1483) by Finn
  Roberts

- `plot<SpatRaster>` could fail when there were multiple values with
  very small differences
  [#1491](rspatial/terra#1491) by srfall

- `as.data.frame<SpatRaster>` with "xy=TRUE" and "wide=FALSE" could
  fail if coordinates were very similar
  [#1476](rspatial/terra#1476) by Pascal
  Oettli

- `rasterizeGeom` now returns the correct layer name
  [#1472](rspatial/terra#1472) by
  HRodenhizer

- `cellSize` with "mask=TRUE" failed if the output was to be written
  to a temp file
  [#1496](rspatial/terra#1496) by Pascal
  Sauer

- `ext<SpatVectorProxy>` did not return the full extent
  [#1501](rspatial/terra#1501) by
  erkent-carb


## enhancements

- `extract` has new argument "small=TRUE" to allow for strict use of
  "touches=FALSE"
  [#1419](rspatial/terra#1419) by Floris
  Vanderhaeghe.

- `as.list<SpatRaster>` has new argument "geom=NULL"

- `rast<list>` now recognizes (x, y, z) base R "image" structures
  [stackoverflow]
  (https://stackoverflow.com/questions/77949551/rspatial-convert-a-grid-list-to-a-raster-using-terra)
  by Ignacio Marzan.

- `inset` has new arguments "offset" and "add"
  [#1422](rspatial/terra#1422) by Armand-CT

- `expanse<SpatRaster>` has argument `usenames`
  [#1446](rspatial/terra#1446) by Bappa Das

- the default color palette is now `terra::map.pal("viridis")` instead
  of `terrain.colors`. The default can be changes with
  `options(terra.pal=...)`
  [#1474](rspatial/terra#1474) by Derek
  Friend

- `as.list<SpatRasterDataset>` now returns a named
  list. [#1513](rspatial/terra#1513) by Eric
  R. Scott


## new

- `bestMatch<SpatRaster>` method

- argument "pairs=TRUE" to `cells` [https://github.com/rspatial/terra/issues/1487](https://github.com/rspatial/terra/issues/1487) by Floris Vanderhaeghe

- `add_grid` to add a grid to a map


# version 1.7-71

Released 2023-01-31

## bug fixes

- k_means did not work if there were NAs
  [#1314](rspatial/terra#1314) by Jakub
  Nowosad

- `layerCor` with a custom function did not work anymore
  [#1387](rspatial/terra#1387) by Jakub
  Nowosad

- `plet` broke when using "panel=TRUE"
  [#1384](rspatial/terra#1384) by Elise
  Hellwig

- using /vis3/ to open a SpatRaster did not work
  [#1382](rspatial/terra#1382) by Mike
  Koontz

- `plot<SpatRaster>(add=TRUE)` sampled the raster data without
  considering the extent of the
  map. [#1394](rspatial/terra#1394) by
  Márcia Barbosa

- `plot<SpatRaster>(add=TRUE)` now only considers the first layer of a
  multi-layer SpatRaster
  [1395](rspatial/terra#1395) by Márcia
  Barbosa

- `set.cats` failed with a tibble was used instead of a data.frame
  [#1406](rspatial/terra#1406) by Mike
  Koontz

- `polys` argument "alpha" was ignored if a single color was
  used. [#1413](rspatial/terra#1413) by
  Derek Friend

- `query` ignore the "vars" argument if all rows were
  selected. [#1398](rspatial/terra#1398) by
  erkent-carb.

- `spatSample` ignored "replace=TRUE" with random sampling,
  na.rm=TRUE, and a sample size larger than the non NA
  cells. [#1411](rspatial/terra#1411) by
  Babak Naimi

- `spatSample` sometimes returned fewer values than requested and
  available for lonlat
  rasters. [#1396](rspatial/terra#1396) by
  Márcia Barbosa.


## enhancements

- `vect<character>` now has argument "opts" for GDAL open options,
  e.g. to declare a file
  encoding. [#1389](rspatial/terra#1389) by
  Mats Blomqvist

- `plot(plg=list(tic=""))` now allows choosing alternative continuous
  legend tic-mark styles ("in", "out", "through" or "none")

- `makeTiles` has new argument "buffer"
  [#1408](rspatial/terra#1408) by Joy
  Flowers.


## new

- `prcomp<SpatRaster>` method
  [#1361](rspatial/terra#1361 (comment))
  by Jakub Nowosad

- `add_box` to add a box around the map. The box is drawn where the
  axes are, not around the plotting region.

- `getTileExtents` provides the extents of tiles. These may be used in
parallelization. See [#1391](https://github.com/rspa
tial/terra/issues/1391) by Alex Ilich.


# version 1.7-65

Released 2023-12-15

## bug fixes

- `flip` with argument `direction="vertical"` filed in some cases with
   large rasters processed in chunks
   [0b714b0](rspatial/terra@0b714b0)
   by Dulci on [stackoveflow](
   https://stackoverflow.com/questions/77304534/rspatial-terraflip-error-when-flipping-a-multi-layer-spatrast-object)

- SpatRaster now correctly handles `NA & FALSE` and `NA | TRUE`
  [#1316](rspatial/terra#1316) by John Baums

- `set.names` wasn't working properly for SpatRasterDataset or
  SpatRasterCollection
  [#1333](rspatial/terra#1333) by Derek Friend

- `extract` with argument "layer" not NULL shifted the layers
  [#1332](rspatial/terra#1332) by Ewan
  Wakefield

- `terraOptions` did not capture "memmin" on
  -[stackoverflow](https://stackoverflow.com/questions/77552234/controlling-chunk
  -size-in-terra) by dww

- `rasterize` with points and a built-in function could crash if no
  field was used
  [#1369](rspatial/terra#1369) by
  anjelinejeline


## enhancements

- `mosaic` can now use `fun="modal"`

- `rast<matrix> and rast<data.frame>` now have option 'type="xylz"
  [#1318](rspatial/terra#1318) by Agustin
  Lobo

- `extract<SpatRaster,SpatVector>` can now use multiple summarizing
  functions [#1335](rspatial/terra#1335) by
  Derek Friend

- `disagg` and `focal` have more optimistic memory requirement
  estimation [#1334](rspatial/terra#1334) by
  Mikko Kuronen

## new

- `k_means<SpatRaster>` method
  [#1314](rspatial/terra#1314) by Agustin
  Lobo

- `princomp<SpatRaster>` method
  [#1361](rspatial/terra#1361) by Alex Ilich

- `has.time<SpatRaster>` method

- new argument "raw=FALSE" to `rast`, `sds`, and `sprc` to allow
  ignoring scale and offset
  [1354](rspatial/terra#1354) by Insang Song


# version 1.7-55

Released 2023-10-14

## bug fixes

- `mosaic` ignored the filename argument if the SpatRasterCollection
  only had a single SpatRaster
  [#1267](rspatial/terra#1267) by Michael
  Mahoney

- Attempting to use `extract` with a raster file that had been deleted
  crashed R. [#1268](rspatial/terra#1268) by
  Derek Friend

- `split<SpatVector,SpatVector>` did not work well in all
  cases. [#1256](rspatial/terra#1256) by
  Derek Corcoran Barrios

- `intersect` with two SpatVectors crashed R if there was a date/time
variable [#1273]( rspatial/terra#1273) by
Dave Dixon

- "values=FALSE" was ignored by
  `spatSample<SpatRaster>(method="weights")`
  [#1275](rspatial/terra#1275) by François
  Rousseu

- `coltab<-` again works with a list as value
[#1280](rspatial/terra#1280) by Diego
Hernangómez

- `stretch` with histogram equalization was not memory-safe
  [#1305](rspatial/terra#1305) by Evan Hersh

- `plot` now resets the "mar" parameter
  [#1297](rspatial/terra#1297) by Márcia
  Barbosa

- `plotRGB` ignored the "smooth" argument
  [#1307](rspatial/terra#1307) by Timothée
  Giraud


## enhancements

- argument "gdal" in `project` was renamed to "use_gdal"
  [#1269](rspatial/terra#1269) by Stuart
  Brown.

- SpatVector attributes can now be stored as an ordered factor
  [#1277](rspatial/terra#1277) by Ben Notkin

- `plot<SpatVector>` now uses an "interval" legend when breaks are
  supplied [#1303](rspatial/terra#1303) by
  Gonzalo Rizzo

- `crop<SpatRaster>` now keeps more metadata, including variable names
  [#1302](rspatial/terra#1302) by rhgof

- `extract(fun="table")` now returns an easier to use data.frame
[#1294](rspatial/terra#1294) by Fernando
Aramburu.


## new
- `metags<-` and `metags` to set arbitrary SpatRaster/file level
   metadata [#1304](https://github.com/rspatial/terra/issues/ 1304) by
   Francesco Chianucci

# version 1.7-46

Released 2023-09-06

## bug fixes

- `plot<SpatVector>` used the wrong main label in some cases
  [#1210](rspatial/terra#1210) by Márcia
  Barbosa

- `plotRGB` failed with an "ext=" argument
  [#1228](rspatial/terra#1228) by Dave Edge

- `rast<array>` failed badly when the array had less than three
  dimensions. [#1254](rspatial/terra#1254)
  by andreimirt.

- `all.equal` for a SpatRaster with multiple layers
[#1236](rspatial/terra#1236) by Sarah
Endicot t

- `zonal(wide=FALSE)` could give wrong results if the zonal SpatRaster
  had "layer" as
  layername. [#1251](rspatial/terra#1251) by
  Jeff Hanson

- `panel` now support argument "range"
  [#141](rspatial/terra#1241) by Jakub
  Nowosad

- `rasterize` with `by=` returned wrong layernames if the by field was
  not sorted [#1266](rspatial/terra#1266) by
  Sebastian Dunnett

- `mosaic` with multiple layers was not correct
  [#1262](rspatial/terra#1262) by
  Jean-Romain


## enhancements

- `wrap<SpatRaster>` now stores color tables
  [#1215](rspatial/terra#1215) by Patrick
  Brown

- `global` now has a "maxcell" argument
  [#1213](rspatial/terra#1213) by Alex Ilich

- `layerCor` with fun='pearson' now returns output with the layer
  names [#1206](rspatial/terra#1206)

- `vrt` now has argument "set_names"
  [#1244](rspatial/terra#1244) by sam-a-levy

- `vrt` now has argument "return_filename"
  [#1258](rspatial/terra#1258) by Krzysztof
  Dyba

- `project<SpatRaster>` has new argument "by_util" exposing the GDAL
  warp utility [#1222](rspatial/terra#1222) by
  Michael Sumner.


## new
- `compareGeom` for list and SpatRasterCollection
  [#1207](rspatial/terra#1207) by Sarah
  Endicott

- `is.rotated<SpatRaster>` method
  [#1229](rspatial/terra#1229) by Andy Lyons

- `forceCCW<SpatVector>` method to force counter-clockwise orientation
  of polygons [#1249](rspatial/terra#1249)
  by srfall.

- `vrt_tiles` returns the filenames of the tiles in a vrt file
  [#1261](rspatial/terra#1261) by Derek
  Friend

- `extractAlong` to extract raster cell values for a line that are
  ordered along the
  line. [#1257](rspatial/terra#1257) by
  adamkc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants