Error in .jcall("RJavaTools", "Z", "hasField", .jcast(x, "java/lang/Object"), : java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "<parameter1>" is null

[luyuliu]

Sorry for raising the same issue again; I read the same issue posted several months ago, but I am experiencing exactly the same when setting up a new environment here in a new machine.

Using pbf directly downloaded from Geofabrik and an official GTFS files from Columbus. No network file created despite outputting "Finished building network.dat at xxx/network.dat". pbf.mapdb and pbf.mapdb.p files are created though.

I tried both the stable version and the dev version, both giving me the same message. But the test data and code works. No idea how it happened. Attaching environment for reference:

$r5r_package_version
[1] ‘2.0’

$r5_jar_version
[1] "7.1"

$java_version
[1] "21.0.4"

$set_memory
[1] "-Xmx32G"

$session_info
R version 4.4.1 (2024-06-14 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 22631)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8 LC_CTYPE=English_United States.utf8 LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C LC_TIME=English_United States.utf8

time zone: America/Chicago
tzcode source: internal

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] r5r_2.0 rJava_1.0-11

loaded via a namespace (and not attached):
[1] compiler_4.4.1 backports_1.5.0 tools_4.4.1 data.table_1.16.0 checkmate_2.3.2

[luyuliu]

Tried Florida and Miami GTFS. Problem persists.

[luyuliu]

Java version:
openjdk 21.0.4 2024-07-16 LTS
OpenJDK Runtime Environment Microsoft-9889606 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Microsoft-9889606 (build 21.0.4+7-LTS, mixed mode, sharing)

[rafapereirabr]

@luyuliu , thanks for opening the issue. Could you please share the links where I can download the exact OSM .pbf and GTFS feeds ?

[luyuliu]

Thanks for the help!

OSM: https://download.geofabrik.de/north-america/us/ohio-latest.osm.pbf
GTFS: https://www.cota.com/data/cota.gtfs.zip

[rafapereirabr]

Hi @luyuliu . Thanks for sharing those links. I could not replicate your error. I was able to build the network and calculate a travel time matrix by transit without any problem. See code below.

reprex using the data downloaded from the links you shared

options(java.parameters = '-Xmx12G')

library(r5r)
library(mapview)
mapviewOptions(platform = 'leafgl')

data_path <- 'C:/Users/user/Downloads/r5r_test_luyu'

# build network
r5r_core <- r5r::setup_r5(data_path,
                     overwrite = TRUE, 
                     verbose = TRUE)

points <- data.frame(id = c('A', 'B'),
                     lat = c(40.006458, 39.926051), 
                     lon = c(-82.97193, -82.83146))

departure_datetime <- as.POSIXct(
  "15-10-2024 14:00:00",
  format = "%d-%m-%Y %H:%M:%S"
)

t1 <- travel_time_matrix(
  r5r_core,
  origins = points,
  destinations = points,
  mode = 'transit',
  departure_datetime = departure_datetime,
  max_trip_duration = 120
)

t1

# visualize transit network
transit_ntwk <- r5r::transit_network_to_sf(r5r_core)

transit_nodes <- transit_ntwk$stops
transit_edges <- transit_ntwk$routes

mapview(transit_edges) + transit_nodes

[rafapereirabr]

I ran r5r::r5r_sitrep() and the only difference between my output and yours is the Java version.

Your Java version is:

$java_version
[1] "21.0.4"

while I have:

$java_version
[1] "21.0.3"

This is a really minor difference, so I don't think this should be the cause of the problem. So I'm suspecting this might be related with that Java version that is being picked by R. Could you please run the code snippets below and share the output you get?

1. checking Java version with rJava

rJava::.jinit()
ver <- rJava::.jcall("java.lang.System", "S", "getProperty", "java.version")
ver

2. checking Java version with rJavaEnv rjava

rJavaEnv::java_check_version_rjava()

3. checking Java version with rJavaEnv cmd

rJavaEnv::java_check_version_cmd()

4. Windows cmd

On your Windows command line, run:

where java

[luyuliu]

Please find the results below.

rJavaEnv::java_check_version_rjava()
Using current session's JAVA_HOME: C:\Program Files\Microsoft\jdk-21.0.4.7-hotspot
✖ Failed to retrieve Java version.
[1] FALSE

rJavaEnv::java_check_version_cmd()
JAVA_HOME: C:\Program Files\Microsoft\jdk-21.0.4.7-hotspot
Java path: C:\PROGRA1\MICROS4\JDK-21~1.7-H\bin\java.exe
Java version: "OpenJDK 64-Bit Server VM warning: The Windows job object has enabled multiple processor groups (2) but
the UseAllWindowsProcessorGroups flag is off. Some processors might not be used. openjdk version "21.0.4" 2024-07-16
LTS OpenJDK Runtime Environment Microsoft-9889606 (build 21.0.4+7-LTS) OpenJDK 64-Bit Server VM Microsoft-9889606 (build
21.0.4+7-LTS, mixed mode, sharing)"
Error in if (major_java_ver == "1") { : argument is of length zero

where java wouldn't return a location. Don't know why.

[rafapereirabr]

Hmm... The test number 1 is the test that {r5r} runs internally to check the Java version in the user machine. So the result is expected. Good.

The result from {rJavaEnv} is a bit strange because it detects the JAVA_HOME, but at the same time it prints the messge
"Failed to retrieve Java version". Perhaps @e-kotov could gives us some light here?

[luyuliu]

I reinstalled java with the recommended build and the problem persists. I suspect it has something to do with the rJava library like you said.

[luyuliu]

Another minor thing but I want to report it anyway. I am running the instance on a new workstation which has 2 CPU sockets. The R always reports that because the virtualizations setting it cannot fully used all the cores it has. May be irrelevant but want to solve this too later.

[e-kotov]

A few comments on the points discussed above

I am not sure this is a problem with the specific Java version or flavour. I have manually setup this "21.0.4.7-hotspot" version of Java by Microsoft on a Windows server I have access to and compared it to the Amazon Corretto Java activated via rJavaEnv::use_java(21). And in both cases I get the .p files and the .dat file in the end.

where java wouldn't return a location. Don't know why.

You are probably running where java in PowerShell. Windows is ridiculous, as [i]n PowerShell, where is an alias for the Where-Object cmdlet. To explicitly invoke where.exe, include the .exe extension" (source), so you can do where.exe java or run where in cmd.exe instead of the PowerShell.

The Windows job object has enabled multiple processor groups (2) but
the UseAllWindowsProcessorGroups flag is off. Some processors might not be used

I'm not sure if this issue is specific to this particular flavour of Java from Microsoft. With Amazon Corretto Java I don't get this warning, but that does not of course mean that is is not affected. And I also do not have a system with two sockets. My tests show that the code above for Ohio runs exactly at the same speed for the graph preparation part with either flavour of Java. Though as r5r::setup_r5() does not have a threads setting, I imaging it may not be able to use as many cores as during the routing, so it may not be a valid test.

Error in if (major_java_ver == "1") { : argument is of length zero

Thanks for catching this error. I will work on rJavaEnv to handle slightly different output of command line Java. Though at this point rJavaEnv is only supposed to work with Amazon Corretto.

Using current session's JAVA_HOME: C:\Program Files\Microsoft\jdk-21.0.4.7-hotspot
✖ Failed to retrieve Java version.

This must be some edge case with a Java version from Microsoft, where I have not tested the type of output it provides.

Because the manual check via r5r::r5r_sitrep()$java_version succeeds and the r5r::setup_r5() does not fail immediately, I am guessing that despite the error from rJavaEnv, the Java path is identified correctly.

===

From my side, I may suggest the following troubleshooting attempt

@luyuliu

Very important. Make sure you run this code in a clean R session and you have NOT yet done library(rJava) or have not even "touched" (see s-u/rJava#334 for details) {rJava} in any other way, i.e. you don't have any code in an R script that is open in the current IDE with rJava::<some_function> or library(rJava), and you have not typed rJava:: in console. Otherwise rJava namespace will be loaded and will prevent any changes to the environment we are about to make.

library(rJavaEnv)
use_java(21)
# this will silently install Amazon Corretto Java 21 and activate it
# in current R session only. This effect will be reset once you restart R session

options(java.parameters = '-Xmx12G')

library(r5r)

data_path <- '<YOUR_DATA_PATH>'

# build network
r5r_core <- r5r::setup_r5(data_path,
                     overwrite = TRUE, 
                     verbose = TRUE)

[e-kotov]

@luyuliu One other thing I would suggest is to try running on the same machine using the same code as I pasted above, but from a Rocker Docker ( https://rocker-project.org/ ) container, if you have necessary rights to run Docker on your machine.

[luyuliu]

Thank you for the extensive analysis @e-kotov . I reinstall R and try to reinstall rJava. But it returns an error saying the function use_java is unavailable.

library(rJava)
use_java(21)
Error in use_java(21) : could not find function "use_java"
rJavaEnv::java_check_version_rjava()
Using current session's JAVA_HOME: 'C:\Users\lzl0116\AppData\Local\Programs\Eclipse Adoptium\jdk-21.0.4.7-hotspot'
✖ Failed to retrieve Java version.
[1] FALSE

I will try a docker environment later but I hope to solve this in a non-docker environment since there could be negative implications on the performance. And hassles. I want to have docker as the last resort.

[luyuliu]

Oh and IT policies. Our university is strictly enforcing a white list policy, which I do not agree with. I need to consult our IT person before installing anything on this machine.

[e-kotov]

@luyuliu sorry, my bad, I meant "rJavaEnv" not "rJava". Fixed the code above. Please make sure to install latest "rJavaEnv".

[luyuliu]

Okay, the first solution still won't give a positive result. The .dat file is still nowhere to be found.

But, good news is that the second command now can retrieve a valid env. So the suggestion is helpful

rJavaEnv::java_check_version_rjava()
Using current session's JAVA_HOME: C:\Users\lzl0116\AppData\Local/R/cache/R/rJavaEnv/installed/windows/x64/21
With the user-specified JAVA_HOME rJava and other rJava/Java-based packages will use Java version: "21.0.4"
[1] "21"

rJavaEnv::java_check_version_cmd()
JAVA_HOME: C:\Users\lzl0116\AppData\Local/R/cache/R/rJavaEnv/installed/windows/x64/21
Java path: C:\Users\lzl0116\AppData\Local\R\cache\R\rJavaEnv\INSTAL~1\windows\x64\21\bin\java.exe
Java version: "openjdk version "21.0.4" 2024-07-16 LTS OpenJDK Runtime Environment Corretto-21.0.4.7.1 (build
21.0.4+7-LTS) OpenJDK 64-Bit Server VM Corretto-21.0.4.7.1 (build 21.0.4+7-LTS, mixed mode, sharing)"
[1] "21"

Now I think there are some subtle different behaviors, like the size of the p file will be slightly larger. And when rerun the setup_r5 will instantly finish (but still won't output dat file)

[luyuliu]

I guess I may need to do a complete reinstallation for R and some packages, since I know some packages may be stored in the personal folder so the normal uninstallation of r may not remove those. I will try that again.

[rafapereirabr]

@e-kotov , thanks so much for your help. While reading the discussion above, it is clear to me that the fact that @luyuliu is able to run the reproducible example with r5r's sample data is actually a good sign that this issue is not casused by the Java version or by possiblity of R not picking up the right Java version. This must be something else.

I just remembered one case another user had some time ago. She had a similar issue in that r5r::r5r::setup_r5() was not generating the local files of the graph. It took us some time to realize that the problem, at least in her case, was that she (her R session) did not have permission to write new files to the data path.

[luyuliu]

Thanks for the suggestion. I changed the work folder to D drive and the problem persists. Hmmm. I am running an administrator account, so this should not be a problem.

[luyuliu]

Just to test the write privilege: I tried to output a csv file in R studio command line and the csv file was created successfully

points <- read.csv(paste(c(base_data_path , "\od_points.csv"), collapse=""))
write.csv(points, file = "d:\Luyu\academic\2022_history_access\test_run\output_file.csv", row.names = FALSE)

[luyuliu]

I am going to paste my test code here just in case.

`
library(r5r)
base_data_path <- "d:\Luyu\academic\history_access\test_run"
options(java.parameters = "-Xmx8G")

mode <- c("WALK", "TRANSIT")
max_walk_time <- 30 # minutes
max_trip_duration <- 60 # minutes

r5r_core <- setup_r5(data_path = base_data_path)
`

[rafapereirabr]

Ok. thanks for the feedback. It's still a puzzle to me why I cannot replicate the error you get. Could you try it in another computer? perhaps on your personal laptop?

[rafapereirabr]

I am going to paste my test code here just in case.

` library(r5r) base_data_path <- "d:\Luyu\academic\history_access\test_run" options(java.parameters = "-Xmx8G")

mode <- c("WALK", "TRANSIT") max_walk_time <- 30 # minutes max_trip_duration <- 60 # minutes

r5r_core <- setup_r5(data_path = base_data_path) `

Regarding this code here, @luyuliu , two quick suggestions:

1. Run options(java.parameters = "-Xmx8G") before loading the library
2. Add verbose = TRUE to setup_r5(). This will print loads of messages from R5 and might helps find the problem.

r5r_core <- setup_r5(data_path = base_data_path, 
                     verbose = TRUE)

[luyuliu]

Okay, I tested on my personal laptop and it went through. I think I am going to do a total clean reinstall of r and rstudio, since I do not install r but the IT does. I think it may messed up some dependency on the library side.

I also notice that on the workstation side I never received those warning message:

2024-09-22 11:13:57,077 [main] ERROR c.c.r.s.StreetLayer - Restriction 14333081 has no restriction tag, skipping
2024-09-22 11:13:57,078 [main] ERROR c.c.r.s.StreetLayer - Restriction 14354639 has no restriction tag, skipping
2024-09-22 11:13:57,079 [main] ERROR c.c.r.s.StreetLayer - Restriction 14395716 has no restriction tag, skipping
2024-09-22 11:13:57,080 [main] ERROR c.c.r.s.StreetLayer - Restriction 14395761 has no restriction tag, skipping
2024-09-22 11:13:57,080 [main] ERROR c.c.r.s.StreetLayer - Restriction 14409723 has no restriction tag, skipping
2024-09-22 11:13:57,081 [main] ERROR c.c.r.s.StreetLayer - Restriction 14424084 has no restriction tag, skipping
2024-09-22 11:13:57,082 [main] ERROR c.c.r.s.StreetLayer - Restriction 14434003 has no restriction tag, skipping
2024-09-22 11:13:57,084 [main] ERROR c.c.r.s.StreetLayer - Restriction 14449573 has no restriction tag, skipping
2024-09-22 11:13:57,084 [main] ERROR c.c.r.s.StreetLayer - Restriction 14449636 has no restriction tag, skipping
2024-09-22 11:13:57,084 [main] ERROR c.c.r.s.StreetLayer - Restriction 14449637 has no restriction tag, skipping
2024-09-22 11:13:57,084 [main] ERROR c.c.r.s.StreetLayer - Restriction 14449654 has no restriction tag, skipping
2024-09-22 11:13:57,085 [main] ERROR c.c.r.s.StreetLayer - Restriction 14451486 has no restriction tag, skipping
2024-09-22 11:13:57,085 [main] ERROR c.c.r.s.StreetLayer - Restriction 14452744 has no restriction tag, skipping
2024-09-22 11:13:57,085 [main] ERROR c.c.r.s.StreetLayer - Restriction 14452745 has no restriction tag, skipping
2024-09-22 11:13:57,085 [main] ERROR c.c.r.s.StreetLayer - Restriction 14452746 has no restriction tag, skipping
2024-09-22 11:13:57,085 [main] ERROR c.c.r.s.StreetLayer - Restriction 14456979 has no restriction tag, skipping
2024-09-22 11:13:57,085 [main] ERROR c.c.r.s.StreetLayer - Restriction 14456980 has no restriction tag, skipping
2024-09-22 11:13:57,086 [main] ERROR c.c.r.s.StreetLayer - Restriction 14472170 has no restriction tag, skipping
2024-09-22 11:13:57,086 [main] ERROR c.c.r.s.StreetLayer - Restriction 14472171 has no restriction tag, skipping
2024-09-22 11:13:57,088 [main] ERROR c.c.r.s.StreetLayer - Restriction 14516403 has no restriction tag, skipping
2024-09-22 11:13:57,090 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 14578679, no way from from to to via via, skipping
2024-09-22 11:13:57,090 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 14582455, does not have from, to and via, skipping
2024-09-22 11:13:57,090 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 14582456, does not have from, to and via, skipping
2024-09-22 11:13:57,091 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 14582457, does not have from, to and via, skipping
2024-09-22 11:13:57,091 [main] ERROR c.c.r.s.StreetLayer - Restriction 14586681 has no restriction tag, skipping
2024-09-22 11:13:57,091 [main] ERROR c.c.r.s.StreetLayer - Restriction 14586706 has no restriction tag, skipping
2024-09-22 11:13:57,091 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 14597339, does not have from, to and via, skipping
2024-09-22 11:13:57,091 [main] ERROR c.c.r.s.StreetLayer - Turn restriction 14597490 has multiple 'to' members, skipping.
2024-09-22 11:13:57,092 [main] ERROR c.c.r.s.StreetLayer - Restriction 14597536 has no restriction tag, skipping
2024-09-22 11:13:57,095 [main] ERROR c.c.r.s.StreetLayer - Restriction 14686466 has no restriction tag, skipping
2024-09-22 11:13:57,097 [main] ERROR c.c.r.s.StreetLayer - Restriction 14709327 has no restriction tag, skipping
2024-09-22 11:13:57,099 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 14733796, does not have from, to and via, skipping
2024-09-22 11:13:57,099 [main] ERROR c.c.r.s.StreetLayer - Restriction 14737841 has no restriction tag, skipping
2024-09-22 11:13:57,100 [main] ERROR c.c.r.s.StreetLayer - Restriction 14761227 has no restriction tag, skipping
2024-09-22 11:13:57,100 [main] ERROR c.c.r.s.StreetLayer - Restriction 14765263 has no restriction tag, skipping
2024-09-22 11:13:57,102 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 14900112, does not have from, to and via, skipping
2024-09-22 11:13:57,102 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 14945292, does not have from, to and via, skipping
2024-09-22 11:13:57,102 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 14945293, does not have from, to and via, skipping
2024-09-22 11:13:57,104 [main] ERROR c.c.r.s.StreetLayer - Restriction 15062857 has no restriction tag, skipping
2024-09-22 11:13:57,104 [main] ERROR c.c.r.s.StreetLayer - Restriction 15081314 has no restriction tag, skipping
2024-09-22 11:13:57,105 [main] ERROR c.c.r.s.StreetLayer - Restriction 15088698 has no restriction tag, skipping
2024-09-22 11:13:57,105 [main] ERROR c.c.r.s.StreetLayer - Restriction 15097617 has no restriction tag, skipping
2024-09-22 11:13:57,105 [main] ERROR c.c.r.s.StreetLayer - Restriction 15097624 has no restriction tag, skipping
2024-09-22 11:13:57,108 [main] ERROR c.c.r.s.StreetLayer - Restriction 15525543 has no restriction tag, skipping
2024-09-22 11:13:57,108 [main] ERROR c.c.r.s.StreetLayer - Restriction 15525544 has no restriction tag, skipping
2024-09-22 11:13:57,108 [main] ERROR c.c.r.s.StreetLayer - Restriction 15525545 has no restriction tag, skipping
2024-09-22 11:13:57,108 [main] ERROR c.c.r.s.StreetLayer - Restriction 15525546 has no restriction tag, skipping
2024-09-22 11:13:57,109 [main] ERROR c.c.r.s.StreetLayer - Restriction 15534964 has no restriction tag, skipping
2024-09-22 11:13:57,109 [main] ERROR c.c.r.s.StreetLayer - Restriction 15534965 has no restriction tag, skipping
2024-09-22 11:13:57,109 [main] ERROR c.c.r.s.StreetLayer - Restriction 15534966 has no restriction tag, skipping
2024-09-22 11:13:57,109 [main] ERROR c.c.r.s.StreetLayer - Restriction 15534967 has no restriction tag, skipping
2024-09-22 11:13:57,123 [main] ERROR c.c.r.s.StreetLayer - Restriction 16135550 has no restriction tag, skipping
2024-09-22 11:13:57,125 [main] ERROR c.c.r.s.StreetLayer - Restriction 16154341 has no restriction tag, skipping
2024-09-22 11:13:57,132 [main] ERROR c.c.r.s.StreetLayer - Restriction 16391433 has no restriction tag, skipping
2024-09-22 11:13:57,132 [main] ERROR c.c.r.s.StreetLayer - Restriction 16450100 has no restriction tag, skipping
2024-09-22 11:13:57,132 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 16477554, does not have from, to and via, skipping
2024-09-22 11:13:57,133 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 16477555, does not have from, to and via, skipping
2024-09-22 11:13:57,134 [main] ERROR c.c.r.s.StreetLayer - To way exits vertex 2762936 twice, restriction 16661208 is therefore ambiguous, skipping
2024-09-22 11:13:57,136 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 16811517, does not have from, to and via, skipping
2024-09-22 11:13:57,136 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 16811518, does not have from, to and via, skipping
2024-09-22 11:13:57,136 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 16811519, does not have from, to and via, skipping
2024-09-22 11:13:57,136 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 16811520, does not have from, to and via, skipping
2024-09-22 11:13:57,141 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17171129, does not have from, to and via, skipping
2024-09-22 11:13:57,141 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17171130, does not have from, to and via, skipping
2024-09-22 11:13:57,142 [main] ERROR c.c.r.s.StreetLayer - Turn restriction 17171131 has multiple 'from' members, skipping.
2024-09-22 11:13:57,142 [main] ERROR c.c.r.s.StreetLayer - Turn restriction 17171132 has multiple 'from' members, skipping.
2024-09-22 11:13:57,142 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17171133, does not have from, to and via, skipping
2024-09-22 11:13:57,142 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17171134, does not have from, to and via, skipping
2024-09-22 11:13:57,142 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17171135, does not have from, to and via, skipping
2024-09-22 11:13:57,142 [main] ERROR c.c.r.s.StreetLayer - Turn restriction 17171136 has multiple 'to' members, skipping.
2024-09-22 11:13:57,142 [main] ERROR c.c.r.s.StreetLayer - Turn restriction 17171137 has multiple 'to' members, skipping.
2024-09-22 11:13:57,142 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17171138, does not have from, to and via, skipping
2024-09-22 11:13:57,144 [main] ERROR c.c.r.s.StreetLayer - Restriction 17316562 has no restriction tag, skipping
2024-09-22 11:13:57,144 [main] ERROR c.c.r.s.StreetLayer - Restriction 17369264 has no restriction tag, skipping
2024-09-22 11:13:57,144 [main] ERROR c.c.r.s.StreetLayer - Restriction 17369265 has no restriction tag, skipping
2024-09-22 11:13:57,147 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17523444, does not have from, to and via, skipping
2024-09-22 11:13:57,147 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17523445, does not have from, to and via, skipping
2024-09-22 11:13:57,147 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17523446, does not have from, to and via, skipping
2024-09-22 11:13:57,147 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17523447, does not have from, to and via, skipping
2024-09-22 11:13:57,152 [main] ERROR c.c.r.s.StreetLayer - Restriction 17712025 has no restriction tag, skipping
2024-09-22 11:13:57,157 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17854062, does not have from, to and via, skipping
2024-09-22 11:13:57,164 [main] ERROR c.c.r.s.StreetLayer - Restriction 17884469 has no restriction tag, skipping
2024-09-22 11:13:57,164 [main] ERROR c.c.r.s.StreetLayer - Restriction 17884470 has no restriction tag, skipping
2024-09-22 11:13:57,165 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17915128, does not have from, to and via, skipping
2024-09-22 11:13:57,165 [main] ERROR c.c.r.s.StreetLayer - Invalid turn restriction 17915129, does not have from, to and via, skipping
2024-09-22 11:13:57,166 [main] ERROR c.c.r.s.StreetLayer - Restriction 17938559 has no restriction tag, skipping
2024-09-22 11:13:57,167 [main] ERROR c.c.r.s.StreetLayer - Restriction 17951809 has no restriction tag, skipping
2024-09-22 11:13:57,167 [main] ERROR c.c.r.s.StreetLayer - Restriction 17951810 has no restriction tag, skipping

Which indicates that it may never reach the point to actually construct the network. So weird.

[e-kotov]

If there was no write access, p and mapdb would not be created.

@rafapereirabr
So far as I can tell, the p + mapdb files and the network.dat are created in a very different way. mapdb file seems to be written to disk with mapdb serialiser, and this is handled by the r5 jar, the network dat is dumped to disk with {r5r} calling the r5's function with this code:

r5r/java-r5rcore/src/org/ipea/r5r/Network/NetworkBuilder.java

Lines 68 to 70 in 934c4ac

	try {
	KryoNetworkSerializer.write(tn, new File(dataFolder, "network.dat"));
	writeNetworkSettings(dataFolder, networkConfig);

So this could be the point of failure (for whatever reason), which is still weird given that the example from the tutorial works fine.

@luyuliu are you getting the network_settings.json file, or is it also missing?

@luyuliu I would not expect a too significant performance hit (especially if you have a lot of cores) if you run it through Docker.

[e-kotov]

@luyuliu your i get the same log messages as you do using your data files. Here's a full log for reference. Even with all these warnings, I get the network.dat file.
Untitled-1.txt

[rafapereirabr]

Yes, R5 prints lots and lots of Error messages but these shoul really be just warning messages as they do not hinder bulding the graph. So these messages can be ignored in general. That's why the default behavior of r5r::setup_r5() is to use verbose = FALSE. This is probably why @luyuliu you did not see these messagens on the work station, because you did explicitly set verbose = TRUE

[luyuliu]

network_settings.json is also missing. Yeah I think we are seeing the same. I deleted all the libraries in AppData/Local/R and reinstall R. Still have the same issue...

See my warning message below.
waning_output.txt

Really lost now.

[luyuliu]

For the sake of science i just copied the work folder to C:\Users\lzl0116\AppData\Local\R\win-library\4.4\r5r\test_run. Just hoping it's a privilege issue but looks like no. Still have no dat file created. Hmmm.

[e-kotov]

@rafapereirabr here it is, @luyuliu 's log does not have messages on KryoNetworkSerializer saving the file and does not have messages like "DEBUG c.c.r.s.StreetRouter - Routing took 8 msec".

So there must be some problem internally in R5... I guess.

@luyuliu I would strongly recommend trying Docker if you can. At least you will be trying using Linux on the same hardware and then we'll be able to rule out the hardware (if that is even possible) incompatibility.

@luyuliu your log is incomplete, there is no beginning like this:

2024-09-22 12:31:21,733 [main] INFO  c.c.o.OSM - Reading OSM from file 'G:\kotov\r5r-399\data\ohio-latest.osm.pbf'.
2024-09-22 12:31:21,781 [main] INFO  c.c.o.PBFInput - PBF file has a replication timestamp of 2024-09-21T20:20:54Z
2024-09-22 12:31:21,815 [main] INFO  c.c.o.PBFInput - node 1
2024-09-22 12:31:28,889 [main] INFO  c.c.o.PBFInput - node 5.0M
2024-09-22 12:31:34,877 [main] INFO  c.c.o.PBFInput - node 10.0M
2024-09-22 12:31:40,901 [main] INFO  c.c.o.PBFInput - node 15.0M
2024-09-22 12:31:46,876 [main] INFO  c.c.o.PBFInput - node 20.0M
2024-09-22 12:31:52,866 [main] INFO  c.c.o.PBFInput - node 25.0M
2024-09-22 12:31:58,851 [main] INFO  c.c.o.PBFInput - node 30.0M
2024-09-22 12:32:03,073 [main] INFO  c.c.o.PBFInput - way 1
2024-09-22 12:32:08,330 [main] INFO  c.c.o.PBFInput - way 1.0M
2024-09-22 12:32:11,354 [main] INFO  c.c.o.PBFInput - way 2.0M
2024-09-22 12:32:14,764 [main] INFO  c.c.o.PBFInput - way 3.0M
2024-09-22 12:32:18,641 [main] INFO  c.c.o.PBFInput - way 4.0M
2024-09-22 12:32:18,962 [main] INFO  c.c.o.PBFInput - relation 1
2024-09-22 12:32:20,195 [main] INFO  c.c.o.PBFInput - Done parsing PBF.
2024-09-22 12:32:20,195 [main] INFO  c.c.o.PBFInput - Read 33464217 nodes, 4061763 ways, 39056 relations.

I am wondering, how many edges you get when the file is first loaded.

When I look at your log, I see:

2024-09-22 11:39:08,583 [main] INFO  c.c.r.s.StreetLayer - Surrogate LTS:
  1: 272186 edges
  2: 13243 edges
  3: 16070 edges
  4: 38815 edges
  Unknown: 179656 edges

In my log:

2024-09-22 12:33:23,752 [main] INFO  c.c.r.s.StreetLayer - Surrogate LTS:
  1: 4010942 edges
  2: 150189 edges
  3: 717090 edges
  4: 731263 edges
  Unknown: 2577406 edges

I have many more edges.

So I am wondering... What if, @luyuliu your pbf file is just an incomplete/broken download? Can you try to redownload this file (maybe also the gtfs.zip for good measure, though this one is probably ok, as I would expect R5 to fail on broken zip).

[rafapereirabr]

The broken pbf is a valid hypothesis. It is also compatible with the fact that @luyuliu was able to run the code on his personal computer but not on the workstation at Uni.

[e-kotov]

FYI (for pbf file I downloaded today on 2024-09-22):

 tools::md5sum("data/ohio-latest.osm.pbf")
          data/ohio-latest.osm.pbf 
"0ac122403520e97b4d3cc2285432c160" 

update:

file.size("data/ohio-latest.osm.pbf")
[1] 256250570

[luyuliu]

Okay! I had some progress:

1. first, conclusion: I can successfully build the dat file now! Yay! I was using the Columbus subset of data since I ruled out the data factor. I ran several tests on different settings like Florida + Miami gtfs, Ohio + COTA gtfs, Columbus + gtfs, all did not have positive results. So I assumed it is not a data issue and proceed to use the Columbus subset generated by Osmosis.

2. second, I changed the original OSM data I downloaded earlier, and it went through. This is the same data I downloaded yesterday. Very strange.

3. I did md5 check and size check:

tools::md5sum("C:\Luyu\academic\2022_history_access\test_run/ohio-latest.osm.pbf")
C:\Luyu\academic\2022_history_access\test_run/ohio-latest.osm.pbf
"24142fc3783880a9a30e53b30497e83e"
file.size("C:\Luyu\academic\2022_history_access\test_run/ohio-latest.osm.pbf")
[1] 255856527

Different, but weird, since this is the one that actually succeeded. Must be OSM fairy doing magic work here.

4. I think it may be because the clean up I did earlier that worked? I did osmosis multiple time in the per-2.0 era and never encounter anything like this. I am going to redo the cut operation in osmosis again and see what happens.

5. I wanted to change the OSM data in the future analysis to create different scenarios for infrastructure sensitivity analsysis, but this issues, which can be because of the bad data, can really disrupt that plan. Would would you suggest I do? I want to remove and add some links in the OSM network, and maybe change to the walkable tag to make some road passable for pedestrian.

Thank you for the help! Enjoy your weekend!

[e-kotov]

@luyuliu Great to know that it somewhat resolved for you! It does seem like the issue likes in something that osmosis does during the processing...

[luyuliu]

Attaching the osmosis code:

osmosis --rb ohio-latest.osm.pbf --bounding-box left=-83.138667 right=-82.847642 bottom=39.870147 top=40.117322 --wb columbus.osm.pbf

[luyuliu]

columbus.osm.zip

And network file for your reference

[luyuliu]

Did another run with the Columbus pbf on personal PC. Also failed to generate DAT file. So I guess this is the potential cause.

But, this cannot address the issue we had earlier with Ohio and Florida weight. Which operation I did solved that? I honestly am so lost right now lol.

Meanwhile, the prompt we had is kinda confusing too. It would be the best if we have a mechanism to report .dat file output failure. Maybe just add a message here like @e-kotov has pointed out?

r5r/java-r5rcore/src/org/ipea/r5r/Network/NetworkBuilder.java

Lines 68 to 70 in 934c4ac

	try {
	KryoNetworkSerializer.write(tn, new File(dataFolder, "network.dat"));
	writeNetworkSettings(dataFolder, networkConfig);

[e-kotov]

is kinda confusing too

yep, I forgot to suggest that too. Perhaps here:

r5r/r-package/R/setup_r5.R

Line 178 in 934c4ac

message("\nFinished building network.dat at ", dat_file)

just before this line it would make sense to add some code that checks if the file was actually created and provide a different message if it wasn't. Possibly, suggesting to the user that the pbf file might be corrupt.

[e-kotov]

Attaching the osmosis code

@luyuliu
Perhaps you could try using osmium instead? https://osmcode.org/osmium-tool/

It seems like it may have a more robust handling of cropping with its different extract strategies, while osmosis does not seem to even describe how it crops the data. I would then assume that osmosis does a very rough crop, right by the bbox boundary, while osmium acts smarter and therefore is less likely to damage the pbf, at least for it to be useful in R5.

[rafapereirabr]

@luyuliu , I would suggest trying our {rosmosis} package.

[e-kotov]

{rosmosis} package

But it's just a wrapper around command line utility and if it somehow damages the pbfs, the using it through the package will not make a difference.

When I say "damage", I mean that probably in this case the selected bounding box turned out to be a bad choice that produced some invalid geometries in linestrings. However, it seems like osmium's strategies for smarter cropping would probably prevent this. The downside of osmium is that it's not available for Windows, but you can run it through Docker (if you can on your work machine).

[rafapereirabr]

yep, {rosmosis} is a wrapper around command line utility. I was suggesting it just for the sake of convenience :)

[luyuliu]

Thanks for the suggestion. I will see what I can do with osmium. Closing the issue now.

[rafapereirabr]

thank you both for the discussion !