Fatal Java error when running setup_r5 about r5r

[rafapereirabr]

Hi David. Thank you for opening the issue. Could you please run the below below to check what version of Java is being used? Moreover, what version of R are you using? I've received the feedback that upgrading to R 4.0.2 on Macos solved the problemas with Java

library(r5r)
library(rJava)

# get installed version of Java
.jinit()
ver <- .jcall("java.lang.System","S","getProperty","java.version")
ver

[dwachsmuth]

Hi Rafa, thanks for the speedy reply! I'm on R 4.0.2, and my version of Java is 13.0.2.

[rafapereirabr]

David, would mind installing the Java SE Development Kit 11.0.8 so we can check if the problem persists?

[dwachsmuth]

Oops, sorry, I had installed 11.0.8, but evidently it was being superseded by the newer 13.0.2, which was previously installed. I removed 13.0.2, verified with rJava that 11.0.8 was being used, and I'm still getting the same error.

The only difference is that the file in the illegal access warning is now "file:/Library/Frameworks/R.framework/Versions/4.0/Resources/library/r5r/jar/r5r_v4.9.0_20200826.jar", whereas it was "...00825.jar" last time.

[rafapereirabr]

We updated the Jar version today but it shouldn't be related to this issue. Could you please run sessionInfo() and paste the output in a comment below?

[dwachsmuth]

Sure thing:

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rJava_0.9-13   r5r_0.0.1      devtools_2.3.1 usethis_1.6.1 

loaded via a namespace (and not attached):
 [1] fs_1.5.0           sf_0.9-5           satellite_1.0.2    lubridate_1.7.9    bit64_4.0.2        webshot_0.5.2     
 [7] RColorBrewer_1.1-2 httr_1.4.2         rprojroot_1.3-2    mapview_2.9.0      tools_4.0.2        backports_1.1.9   
[13] R6_2.4.1           KernSmooth_2.23-17 DBI_1.1.0          colorspace_1.4-1   raster_3.3-13      withr_2.2.0       
[19] sp_1.4-2           tidyselect_1.1.0   gridExtra_2.3      prettyunits_1.1.1  processx_3.4.3     leaflet_2.0.3     
[25] bit_4.0.4          curl_4.3           compiler_4.0.2     leafem_0.1.3       cli_2.0.2          rvest_0.3.6       
[31] xml2_1.3.2         desc_1.2.0         checkmate_2.0.0    scales_1.1.1       classInt_0.4-3     callr_3.4.3       
[37] digest_0.6.25      base64enc_0.1-3    dichromat_2.0-0    pkgconfig_2.0.3    htmltools_0.5.0    sessioninfo_1.1.1 
[43] htmlwidgets_1.5.1  rlang_0.4.7        rstudioapi_0.11    generics_0.0.2     jsonlite_1.7.0     crosstalk_1.1.0.1 
[49] dplyr_1.0.2        magrittr_1.5       Rcpp_1.0.5         munsell_0.5.0      fansi_0.4.1        abind_1.4-5       
[55] viridis_0.5.1      lifecycle_0.2.0    tmaptools_3.1      pkgbuild_1.1.0     grid_4.0.2         blob_1.2.1        
[61] parallel_4.0.2     crayon_1.3.4       lattice_0.20-41    stars_0.4-3        hms_0.5.3          ps_1.3.4          
[67] pillar_1.4.6       codetools_0.2-16   stats4_4.0.2       pkgload_1.1.0      XML_3.99-0.5       glue_1.4.1        
[73] data.table_1.13.1  RPostgres_1.2.0    remotes_2.2.0      RcppParallel_5.0.2 vctrs_0.3.2        png_0.1-7         
[79] testthat_2.3.2     gtable_0.3.0       purrr_0.3.4        assertthat_0.2.1   ggplot2_3.3.2      xfun_0.16         
[85] lwgeom_0.2-5       e1071_1.7-3        class_7.3-17       viridisLite_0.3.0  tibble_3.0.3       tinytex_0.25      
[91] memoise_1.1.0      units_0.6-7        ellipsis_0.3.1     osmdata_0.1.3 

[rafapereirabr]

Hmm.. Could please install.packages('jdx') and try again?

[dwachsmuth]

Progress! I now get a different error:

Error in rJava::.jnew("org.ipea.r5r.R5RCore", data_path, verbose) : java.io.IOError: java.io.IOException: Wrong index checksum, store was not closed properly and could be corrupted.

I tried deleting the .jar from the the package folder to force a redownload, but I got the same error a second time.

[dwachsmuth]

Sorry, to be clearer, that is the same ultimate error I was receiving before, but I'm no longer getting all the illegal operation warnings before it errors out.

[dwachsmuth]

Another update: after rebooting R and trying again, I'm back to the initial set of warnings. Here is the full function output, in case it is useful:

> r5r_core <- setup_r5(data_path = path)
Using cached version from /Library/Frameworks/R.framework/Versions/4.0/Resources/library/r5r/jar/r5r_v4.9.0_20200826.jar
20:41:15.095 [main] INFO  c.c.r5.transit.TransportNetwork - File 'OTP-2020/build-config.json' is not present. Using default configuration.
20:41:15.108 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20200201_20200212_Montreal_agence-metropolitaine-de-transport-133-20200131.zip
20:41:15.112 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20191222_20200212_Laval_societe-de-transport-de-laval-38-20191222.zip
20:41:15.114 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20200123_20200212_Montreal_agence-metropolitaine-de-transport-140-20200123.zip
20:41:15.115 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20191219_20200212_Montreal_agence-metropolitaine-de-transport-139-20191219.zip
20:41:15.115 [main] WARN  c.c.r5.transit.TransportNetwork - Skipping non-input file 'OTP-2020/router-config.json'
20:41:15.115 [main] WARN  c.c.r5.transit.TransportNetwork - Skipping non-input file 'OTP-2020/points-2.csv'
20:41:15.115 [main] WARN  c.c.r5.transit.TransportNetwork - Skipping non-input file 'OTP-2020/traveltime_matrix.csv'
20:41:15.116 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20191211_20200212_Montreal_agence-metropolitaine-de-transport-143-20191211.zip
20:41:15.116 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20200201_20200212_Montreal_agence-metropolitaine-de-transport-135-20200131.zip
20:41:15.116 [main] WARN  c.c.r5.transit.TransportNetwork - Skipping non-input file 'OTP-2020/montreal_cma.osm.pbf.mapdb'
20:41:15.116 [main] WARN  c.c.r5.transit.TransportNetwork - Skipping non-input file 'OTP-2020/File'
20:41:15.117 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20200121_20200212_Montreal_agence-metropolitaine-de-transport-128-20200121.zip
20:41:15.117 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20200107_20200212_Montreal_agence-metropolitaine-de-transport-137-20200107.zip
20:41:15.117 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20200129_20200212_Longueuil_reseau-de-transport-de-longueuil-37-20200129.zip
20:41:15.118 [main] WARN  c.c.r5.transit.TransportNetwork - Skipping non-input file 'OTP-2020/montreal'
20:41:15.118 [main] WARN  c.c.r5.transit.TransportNetwork - Skipping non-input file 'OTP-2020/ttmfeb.py'
20:41:15.118 [main] WARN  c.c.r5.transit.TransportNetwork - Skipping non-input file 'OTP-2020/otp.jar'
20:41:15.118 [main] WARN  c.c.r5.transit.TransportNetwork - Skipping non-input file 'OTP-2020/montreal_cma.osm.pbf.mapdb.p'
20:41:15.118 [main] INFO  c.c.r5.transit.TransportNetwork - Found OSM file OTP-2020/montreal_cma.osm.pbf
20:41:15.118 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20191219_20200212_Montreal_agence-metropolitaine-de-transport-130-20191219.zip
20:41:15.119 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20200121_20200212_Montreal_agence-metropolitaine-de-transport-1174-20200120.zip
20:41:15.119 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20191217_20200212_Montreal_agence-metropolitaine-de-transport-131-20191216.zip
20:41:15.119 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20191217_20200212_Montreal_agence-metropolitaine-de-transport-136-20191216.zip
20:41:15.120 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20191216_20200212_Montreal_agence-metropolitaine-de-transport-138-20191216.zip
20:41:15.120 [main] INFO  c.c.r5.transit.TransportNetwork - Found GTFS file OTP-2020/20200115_20200212_Montreal_agence-metropolitaine-de-transport-142-20200115.zip
20:41:15.120 [main] INFO  c.c.r5.transit.TransportNetwork - Summarizing builder config: build-config.json
20:41:15.481 [main] INFO  c.c.r5.transit.TransportNetwork - {
  "htmlAnnotations" : false,
  "maxHtmlAnnotationsPerFile" : 1000,
  "transit" : true,
  "useTransfersTxt" : false,
  "parentStopLinking" : false,
  "stationTransfers" : false,
  "subwayAccessTime" : 2.0,
  "streets" : true,
  "embedRouterConfig" : true,
  "areaVisibility" : false,
  "matchBusRoutesToStreets" : false,
  "fetchElevationUS" : false,
  "speeds" : {
    "units" : "mph",
    "values" : {
      "secondary" : 35,
      "tertiary_link" : 25,
      "primary_link" : 25,
      "tertiary" : 25,
      "living_street" : 5,
      "motorway_link" : 35,
      "trunk" : 55,
      "secondary_link" : 25,
      "motorway" : 65,
      "trunk_link" : 35,
      "primary" : 45
    },
    "defaultSpeed" : 25
  },
  "staticBikeRental" : true,
  "staticParkAndRide" : true,
  "staticBikeParkAndRide" : false,
  "bikeRentalFile" : null,
  "analysisFareCalculator" : null
}
20:41:15.580 [main] INFO  com.conveyal.osmlib.OSM - Reading OSM DB from: /Users/dwachsmuth/Documents/Academic/Code/network-analysis/OTP-2020/montreal_cma.osm.pbf.mapdb
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.mapdb.Volume$ByteBufferVol (file:/Library/Frameworks/R.framework/Versions/4.0/Resources/library/r5r/jar/r5r_v4.9.0_20200826.jar) to method java.nio.DirectByteBuffer.cleaner()
WARNING: Please consider reporting this to the maintainers of org.mapdb.Volume$ByteBufferVol
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Error in rJava::.jnew("org.ipea.r5r.R5RCore", data_path, verbose) : 
  java.io.IOError: java.io.IOException: Wrong index checksum, store was not closed properly and could be corrupted.

[rafapereirabr]

David, if would like to force an update of the Jar file, you can run r5r::download_r5(force_update = T). These warnings shouldn't be a problem. This is a temporary Java issue.

From the output above, it seems setup_r5 worked. Does it return an r5r_core object? This error message is strange though. @mvpsaraiva , could have a look into this please?

Error in rJava::.jnew("org.ipea.r5r.R5RCore", data_path, verbose) :
java.io.IOError: java.io.IOException: Wrong index checksum, store was not closed properly and could be corrupted.

[dwachsmuth]

Thanks for your persistence here! Unfortunately forcing the Jar update didn't change anything. Same error, and no, the function doesn't return any results, just the error.

[rafapereirabr]

and thanks for your patience. :)

from the setup_r5 output above, it seems the functions worked but I'm not sure what could have caused that error message. This could be an issue with the input data. To check if this is the case, I would ask you to run the code below. This is a sample data that comes with the r5r pacakge.

library(r5r)

# build transport network
data_path <- system.file("extdata", package = "r5r")
r5r_obj <- setup_r5(data_path = data_path)

# load origin/destination points
points <- read.csv(system.file("extdata/poa_hexgrid.csv", package = "r5r"))[1:5,]

mode <- c("WALK", "TRANSIT")
max_walk_dist = Inf
max_trip_duration <- 120L
departure_datetime <- as.POSIXct("13-03-2019 14:00:00",
                                format = "%d-%m-%Y %H:%M:%S")

# estimate travel time matrix
df <- travel_time_matrix(r5r_obj,
                        origins = points,
                        destinations = points,
                        mode,
                        departure_datetime,
                        max_walk_dist,
                        max_trip_duration)

head(df)

[dwachsmuth]

Yup, everything works fine with the sample data. Output is:

> head(df)
            fromId            toId travel_time
1: 89a901291abffff 89a901291abffff           2
2: 89a901291abffff 89a901295b7ffff          54
3: 89a901291abffff 89a901284a3ffff         117
4: 89a901291abffff 89a9012809bffff          80
5: 89a9012a3cfffff 89a9012a3cfffff           3
6: 89a9012a3cfffff 89a901284a3ffff          58

I will go talk to the project lead (Kevin Manaugh, who says hi!) to see if there's any possibility there are problems with the input data.

[rafapereirabr]

Great. This is indeed an issue with the input data then. R5 is very picky with GTFS data and its error messages often do not make it clear what is wrong with the input data. Sometimes it might be a calendar.txt missing in the GTFS for example. Say hi to Kevin for me!

ps. David, please let me know if you find what could be causing this.

[dwachsmuth]

Ok, a quick update! After removing all the files from our data folder and just adding the GTFS and .pbm file, everything worked. So something else that was in the folder was causing the problem. If I can isolate it, I'll let you know, but for now all is good, and the function worked through 15 GTFS networks in 50 seconds, which is just incredible! Thanks again for your help.

[rafapereirabr]

Thank you for the update, David!

[mvpsaraiva]

Thank you David, for using our package and giving us such helpful feedback. I'm sorry I couldn't help yesterday, but I've been reading through your messages now trying to get some insights on what is going on. For now, my best guess is that this guy is the culprit:

20:41:15.580 [main] INFO com.conveyal.osmlib.OSM - Reading OSM DB from: /Users/dwachsmuth/Documents/Academic/Code/network-analysis/OTP-2020/montreal_cma.osm.pbf.mapdb

We are going to add extra checks to make sure everything is ok in the input folder, and try to output clearer error messages.