Local package install fail when dependencies = TRUE

[mcanouil]

Somehow the codepath completely change when adding dependencies = TRUE.
It seems, for some reason it goes to the default in switch which then change the source to Repository.
The codepath for "local" packages also seems off, the record object contains the RemoteURL which is the path, but it is somehow not used.

Also, the records with the proper information, is overwritten: https://github.com/rstudio/renv/blob/main/R/install.R#L193-L209

I tried to get to the bottom of it, but it seems there is no actual "local" codepath which seems the reason of such weird behaviour. I know that local packages cannot really be restored, but I do think they should still be installed properly even if they are no caching, etc.
Unfortunately, I cannot manage to create a dummy package that reproduce this.

The DESCRIPTION of "mypackage" looks like:

Package: mypackage
Title: My Package
Version: 1.4.1
Authors@R: c(
    person(),
  )
Description: Set of analysis functions, templates, graphics- functions.
License: file LICENSE
URL: https://XXXXXXXXXXX.pages.github.io/mypackage/,
    https://github.com/user/repo/tree/main/src/packages/mypackage
BugReports: 
    https://github.com/user/repo/issues
Depends: 
    R (>= 4.3.0)
Imports:
    data.table,
    myotherpackage (>= 1.1.2)
Suggests:
    cli,
    prompt,
    broom,
    DESeq2,
    devtools
Remotes:
    github::user/repo:src/packages/myotherpackage@myotherpackage-latest,
    bioc::DESeq2
Config/Needs/website: pkgdown
Config/testthat/edition: 3
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.2

This is what the output look like without dependencies:

renv::install("/workspaces/mypackage")

The following package(s) will be installed:
- base64enc   [0.1-3]
...
These packages will be installed into "/workspaces/renv/library/linux-ubuntu-jammy/R-4.4/x86_64-pc-linux-gnu".

Do you want to proceed? [Y/n]: 
# Installing packages --------------------------------------------------------
...
- Installing mypackage ...                           OK [built from source and cached in 24s]
Successfully installed 65 packages in 31 seconds.
Warning message:
failed to resolve remote 'github::user/repo:src/packages/myotherpackage@myotherpackage-latest'; skipping

With dependencies:

renv::install("/workspaces/mypackage", dependencies = TRUE)

# Downloading packages -------------------------------------------------------
- Downloading cards from P3M ...                OK [447.5 Kb in 0.65s]
- Package mypackage [1.5.0] will be installed from the cellar.
- Downloading insight from P3M ...              OK [2.1 Mb in 0.66s]
- Downloading datawizard from P3M ...           OK [1 Mb in 0.71s]
- Downloading performance from P3M ...          OK [3 Mb in 0.59s]
- Downloading mvtnorm from CRAN ...             OK [file is up to date]
- Package mypackage [1.5.0] will be installed from the cellar.
Warning: failed to find source for 'mypackage 1.5.0' in package repositories
Error: failed to retrieve package '[email protected]'
In addition: Warning messages:
1: failed to resolve remote 'github::user/repo:src/packages/myotherpackage@myotherpackage-latest'; skipping 
2: failed to resolve remote 'github::user/repo:src/packages/myotherpackage@myotherpackage-latest'; skipping 
3: failed to resolve remote 'github::user/repo:src/packages/myotherpackage@myotherpackage-latest'; skipping 
Traceback (most recent calls last):
8: renv::install("/workspaces/mypackage", dependencies = TRUE)
7: records <- renv_retrieve_impl(packages) at install.R#209
6: for (package in packages)
     handler(package, renv_retrieve_impl_one(package)) at retrieve.R#120
5: for (package in packages)
     handler(package, renv_retrieve_impl_one(package)) at retrieve.R#120
4: switch(source,
          bioconductor = renv_retrieve_bioconductor(record),
          bitbucket    = renv_retrieve_bitbucket(record),
          git          = renv_retrieve_git(record),
          github       = renv_retrieve_github(record),
          gitlab       = renv_retrieve_gitlab(record),
          repository   = renv_retrieve_repos(record),
          url          = renv_retrieve_url(record),
          renv_retrieve_unknown_source(record)
   ) at retrieve.R#317
3: renv_retrieve_repos(record) at retrieve.R#1258
2: stopf("failed to retrieve package '%s'", remote) at retrieve.R#757
1: stop(sprintf(fmt, ...), call. = call.) at utils-format.R#3

[kevinushey]

Thanks for the bug report, but this is going to be very challenging to fix without a reproducible example. Any chance the project wherein you're seeing this is available publicly, or you'd be able to divine a reproducible example?

[mcanouil]

It should remain private, that is the unfortunate part of the issue as I know too well the requirement for a reproducible example.

I'll keep trying to reproduce with a dummy package and update here.

Meanwhile, why does the switch (from the trace) does not have a "renv_retrieve_local" function?

[kevinushey]

The code isn't entirely clear, but before we attempt to install from the regular remote sources, we try to handle installation "shortcuts" which includes local package installations:

renv/R/retrieve.R

Lines 296 to 310 in 73c899e

	# try some early shortcut methods
	shortcuts <- if (rebuild) c(
	renv_retrieve_cellar
	) else c(
	renv_retrieve_explicit,
	renv_retrieve_cellar,
	if (!renv_tests_running() && config$install.shortcuts())
	renv_retrieve_libpaths
	)
	for (shortcut in shortcuts) {
	retrieved <- catch(shortcut(record))
	if (identical(retrieved, TRUE))
	return(TRUE)
	}

In particular,

renv/R/retrieve.R

Lines 648 to 663 in 73c899e

	renv_retrieve_explicit <- function(record) {
	# try parsing as a local remote
	source <- record$Path %||% record$RemoteUrl %||% ""
	if (nzchar(source)) {
	resolved <- catch(renv_remotes_resolve_path(source))
	if (inherits(resolved, "error"))
	return(FALSE)
	}
	# treat as 'local' source but extract path
	normalized <- renv_path_normalize(source, mustWork = TRUE)
	resolved$Source <- "Local"
	renv_retrieve_successful(resolved, normalized)
	}

will check for RemoteUrl, and if that's the path to a local file, then we use that for installation.

This output:

- Package mypackage [1.5.0] will be installed from the cellar.
Warning: failed to find source for 'mypackage 1.5.0' in package repositories
Error: failed to retrieve package '[email protected]'
In addition: Warning messages:
1: failed to resolve remote 'github::user/repo:src/packages/myotherpackage@myotherpackage-latest'; skipping 
2: failed to resolve remote 'github::user/repo:src/packages/myotherpackage@myotherpackage-latest'; skipping 
3: failed to resolve remote 'github::user/repo:src/packages/myotherpackage@myotherpackage-latest'; skipping 

seems curious; it seems like renv should be choosing to install a local copy of the package from the cellar, but it's still trying to install it from the remote repository for some reason?

[mcanouil]

I'll try next week to come up with a clean small reproducible example mimicking the private environment in which this issue occurs.
(For now, we are basically using workarounds as the issue appears for several users)

[mcanouil]

I recreated a very similar setup but I am unable to reproduce the issue in it (https://github.com/mcanouil/renv-issue1977).
Since the issue seemed tied to renv caching on our real infrastructure, I am not sure I'll be able to figure out the exact combination of things leading to the issue I reported.

Feel free to close as the blocking part of the issue was solved on our end by changing where we stored the cache and by completely purging it in the process.

---

Side note (I can open a separate issue about this if it's indeed a bug):

it seems the following syntax (used to tell renv which remotes belong to which package #1755) is not recognised by all components of renv:

Assume a DESCRIPTION. (studies/study2 in the repository above):

Depends:
    pkg1
Remotes:
    pkg1=github::mcanouil/renv-issue1977:src/packages/pkg1@pkg1-latest

renv::init()

This project contains a DESCRIPTION file.
Which files should renv use for dependency discovery in this project? 

1: Use only the DESCRIPTION file. (explicit mode)
2: Use all files in this project. (implicit mode)

Selection: 1
- Using 'explicit' snapshot type. Please see `?renv::snapshot` for more details.

Error in FUN(X[[i]], ...) : object of type 'closure' is not subsettable
Traceback (most recent calls last):
6: renv::init()
5: hydrate(library = library, repos = repos, prompt = FALSE, report = FALSE, 
       project = project)
4: renv_project_remotes(project, filter = filter, resolve = TRUE)
3: filter(specs, remotes)
2: map_chr(remotes, `[[`, "Package")
1: vapply(x, f, ..., FUN.VALUE = character(1))

[kevinushey]

Thanks! I can reproduce that as well, and it appears to be a real issue. I've filed that in #2055.