Changes in renv/rsconnect - now getting "Local" packages in our CI/CD pipeline

[slodge-work]

Not entirely sure this is an issue - but this seems like the best place to get advice/answers - so please forgive me asking here.

For many of our apps, reports and apis, our repos and CI/CD pipelines work by:

• each repo contains a package and one or more apps
• the package is version 9.9.9 in dev/git
• the build pipeline:
  • stamps an incrementing version number on top
  • uses devtools to build the package (just a zip)
  • uses devtools to install the local package from disk
  • uploads the package to either our test CRAN or to our real CRAN
  • runs rsconnect writeManifest and deploy steps to deploy to either our test Connect or to our prod Connect

This pipeline's been working pretty well for a couple of years and is very stable.

Right now I'm looking at a breaking change in our pipelines following upgrading rsconnect to 1.x. What I'm seeing is that the currently built package is being picked up in the auto-generated manifest.json as:

    "R.Brandy": {
      "Source": "Local",
      "Repository": "OurCo.IMS",
      "description": {
        "Package": "R.Brandy",
        "Type": "Package",
        "Title": "Shiny support for OurCo look and feel including the Unified\nDesign System",
        "Authors@R": "...",
        "Description": "R package with functions exposing Shiny Unified Design.",
        "Version": "0.1.209",
        "Maintainer": "...",
        "License": "file LICENSE",
        "Encoding": "UTF-8",
        "Repository": "OurCo.IMS",
        "Imports": "htmltools, shiny, dplyr, rmarkdown, purrr, magrittr,\ndownloadthis, rlang, jsonlite, stringr, rstudioapi, ggplot2,\nbase64enc, tibble, readr",
        "RoxygenNote": "7.2.1",
        "NeedsCompilation": "no",
        "Packaged": "2023-09-26 15:18:59 UTC; azdagent_azpcontainer",
        "Author": "Support [aut, cre]",
        "Built": "R 4.2.0; ; 2023-09-26 15:19:03 UTC; unix",
        "RemoteType": "local",
        "RemoteUrl": "/__w/1/a/package/R.Brandy_0.1.209.tar.gz"
      }
    },

The problem here are the line:

      "Source": "Local",      
      "Repository": "OurCo.IMS",

These cause the Connect deployment step to fail with lots of messages like...

2023/09/26 15:21:19.810629517 curl: (6) Could not resolve host: OurCo.IMS
2023/09/26 15:21:19.810642129 curl: HTTP 000 http://OurCo.IMS/src/contrib/Archive/R.Brandy/R.Brandy_0.1.210.tar.gz

Under previous versions of rsconnect (which I guess were less renv based) the manifest was written as:

    "R.Brandy": {
      "Source": "OurCo.IMS",
      "Repository": "https:/cran-uat.ourco.app/ims",
      "description": {

which just worked (TM)

I'm currently wondering if there's anything I can toggle inside rsconnect/renv to make it pick up that this package should be looked up in CRAN...

If not then I can change the build system it so that it installs from our CRAN server(s) before doing the deployment. I'm just hesitating before doing this second option, though, as there are subtleties there (and as I've broken production stability enough already this month!)

Just wondering if anyone has any other suggestions/ideas...

[aronatkins]

@slodge-work - Is this project managed by renv? How is R.Brandy tracked in your renv.lock?

@kevinushey - Do you have any suggestions, here? What is the best way to record that the devtools installed from-zip package is also available in the repository?

[slodge-work]

I've been digging some more.

Each repo is renv controlled, yes.

I think this block of code is what controls Repository and Source for non-Local packages - https://github.com/rstudio/rsconnect/blob/41907afefe219a5e00496c0ba99746eca38fbf01/R/bundlePackageRenv.R#L94C1-L120C1

Interestingly, for most of our apps (ones which have multiple packages - including some from outside the current repo), then the Connect installation succeeds - it seems Connect tries the bad OurCo.IMS urls, but then tries all the repo url's it has seen in the manifest.

2023/09/26 19:44:07.770019755 Installing R.MultiPack (0.1.58) ... 
2023/09/26 19:44:08.585154554 curl: HTTP 200 https://packagemanager.rstudio.com/all/__linux__/focal/latest/src/contrib/PACKAGES.rds
2023/09/26 19:44:08.746001131 curl: (6) Could not resolve host: OurCo.IMS
2023/09/26 19:44:08.746342560 curl: HTTP 000 http://OurCo.IMS/src/contrib/PACKAGES.rds
2023/09/26 19:44:08.758770760 curl: (6) Could not resolve host: OurCo.IMS
2023/09/26 19:44:08.759081966 curl: HTTP 000 http://OurCo.IMS/src/contrib/PACKAGES.gz
2023/09/26 19:44:08.770057524 curl: (6) Could not resolve host: OurCo.IMS
2023/09/26 19:44:08.770357443 curl: HTTP 000 http://OurCo.IMS/src/contrib/PACKAGES
2023/09/26 19:44:08.771484448 Warning: unable to access index for repository OurCo.IMS/src/contrib:
2023/09/26 19:44:08.771491906   'curl' call had nonzero exit status
2023/09/26 19:44:08.875471736 curl: HTTP 200 https://cran-uat.OurCo.app/ims/src/contrib/PACKAGES.rds
2023/09/26 19:44:09.628420366 curl: (6) Could not resolve host: OurCo.IMS
2023/09/26 19:44:09.628384051 curl: HTTP 000 http://OurCo.IMS/src/contrib/PACKAGES.rds
2023/09/26 19:44:09.640428122 curl: (6) Could not resolve host: OurCo.IMS
2023/09/26 19:44:09.640888313 curl: HTTP 000 http://OurCo.IMS/src/contrib/PACKAGES.gz
2023/09/26 19:44:09.652598310 curl: (6) Could not resolve host: OurCo.IMS
2023/09/26 19:44:09.653061171 curl: HTTP 000 http://OurCo.IMS/src/contrib/PACKAGES
2023/09/26 19:44:09.654625775 Warning: unable to access index for repository OurCo.IMS/src/contrib:
2023/09/26 19:44:09.654634329   'curl' call had nonzero exit status
2023/09/26 19:44:10.298665668 curl: HTTP 200 https://cran-uat.OurCo.app/ims/src/contrib/R.MultiPack_0.1.58.tar.gz
2023/09/26 19:44:12.824397522 Caching R.MultiPack.
2023/09/26 19:44:13.221714572 Using cached R.MultiPack.

I've also looked back at other deployments which were working a month ago... and some of those have manifest entries starting e.g.

      "Source": "Local",
      "Repository": null,

... so maybe we were just "lucky" this has been working so well all along...

I'm beginning to think I should just bite the bullet and change our build system so it deploys the newly built package(s) from our CRAN instead of from the local disk copy.... although if there is an answer to "What is the best way to record that the devtools installed from-zip package is also available in the repository?" then maybe I'll reconsider...

Thanks for looking, but please don't worry too much... I can try to get this working "properly" using CRANs.

[aronatkins]

The interpretation of Repository=null by Connect and by rsconnect has changed over time. Some time ago, the tools would produce an error because a null repository meant that we could not locate a package in one of the configured repositories. We have relaxed that position, partially because archived packages are installable but not listed in available.packages() results.

I'm surprised that we are leaving {"Source": "Local", "Repository": "OurCo.IMS"} in the manifest.

@hadley - It feels as if we are missing some path in standardizeRenvPackage(). Maybe the "unknown" check should include "Local", or we should attempt to find local packages in some repository.

[kevinushey]

What is the best way to record that the devtools installed from-zip package is also available in the repository?

It's not great, but the escape hatch that renv uses is through the (undocumented) renv.records.override option, e.g.

options(renv.records.override = list(devtools = list(...)))

where you'd fill in the list with the fields you'd normally have in the lockfile entry.

[slodge-work]

Thanks @kevinushey

I gave that a go, but it looks like this override is only used for renv::restore() workflows - https://github.com/search?q=repo%3Arstudio%2Frenv%20renv_lockfile_override&type=code

This has given me better understanding of where "Local" is used - and an idea of how to "override" (aka "hack") insider our own build system though... so I think I'll go with that, especially given the conceptual problems that the PR hit.

[slodge-work]

Workaround implemented - we now find the DESCRIPTION file of the locally installed packages and remove the remote lines:

message("Fixing up the installed DESCRIPTION file to try to pretend it's not a LOCAL package")
renv_path <- renv::paths$library()
package_desc <- file.path(renv_path, package_name, "DESCRIPTION")
message("Reading ", package_desc)
package_desc_data <- read.dcf(package_desc)
package_desc_data_frame <- data.frame(package_desc_data)
message("Read DESCRIPTION was ", package_desc_data_frame)
package_desc_data_frame$RemoteType <- NULL
package_desc_data_frame$RemoteUrl <- NULL
message("Patched DESCRIPTION is ", package_desc_data_frame)
write.dcf(package_desc_data_frame, file=package_desc)

At some point we might need to do something better... e.g. I might need to work out the network and security paths to do proper CRAN installing within our build system. But for now 👍

So closing - thanks all