nasty cargo fetch

[cangyin]

Problem

I 'm in isolated network. I want to make cargo fetch to use alternative crates source, so I configured in PROJECT_DIR/.cargo/config.toml:

[registries]
sf= { index = "http://mirrors.sf.com/nexus/repository/rust-crates" }

[registry]
default = "sf"

[source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index"
replace-with = "sf-crates-proxy"

[source.sf-crates-proxy]
registry = "http://mirrors.sf.com/nexus/repository/rust-crates/"

[net]
git-fetch-with-cli = true

Steps

$ cargo update
    Updating `sf` index

$ cargo fetch -vv
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 1254 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 1277 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 222 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 1257 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 225 ms: Couldn't connect to server)
warning: spurious network error (2 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 216 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 1284 ms: Couldn't connect to server)
warning: spurious network error (2 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 1265 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 225 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 218 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 1234 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 8391 ms: Couldn't connect to server)
warning: spurious network error (2 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 1261 ms: Couldn't connect to server)
warning: spurious network error (2 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 2313 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 221 ms: Couldn't connect to server)
warning: spurious network error (1 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 235 ms: Couldn't connect to server)
warning: spurious network error (1 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 2295 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 4352 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 2311 ms: Couldn't connect to server)
warning: spurious network error (2 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 234 ms: Couldn't connect to server)
warning: spurious network error (2 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 3285 ms: Couldn't connect to server)
warning: spurious network error (3 tries remaining): [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 3324 ms: Couldn't connect to server)
error: failed to download from `https://crates.io/api/v1/crates/libc/0.2.147/download`

Caused by:
  [7] Couldn't connect to server (Failed to connect to crates.io port 443 after 1283 ms: Couldn't connect to server)

So why is cargo fetch still attempting to connect to crates.io ?

What more configuration am I missing?

Version

$ cargo version
cargo 1.72.0-nightly (5b377cece 2023-06-30)

[cangyin]

I added registry = ... to each [[package]] section in Cargo.lock, like:

[[package]]
name = "libc"
version = "0.2.147"
registry = "sf"  <-- this line
source = "registry+http://mirrors.sf.com/nexus/repository/rust-crates"
checksum = "b4668fb0ea861c1df094127ac5f1da3409a82116a4ba74fca2e58ef927159bb3"

but cargo fetch simply silently removes these lines, at least emit some warning and tell the reason !

I modified Cargo.toml as said in doc :

[dependencies]
blake3 = { version = "1.2.0", registry = "sf" }
libc =  { version = "0.2.132", registry = "sf" }

Still no luck. cargo fetch always want to connect to crates.io

[cangyin]

How frustrating!

[cangyin]

This is my second try on building a same rust project. Any help is really appreciated.

[arlosi]

The index has a file called config.json that specifies where to download the .crate files.

My guess is that http://mirrors.sf.com/nexus/repository/rust-crates still has the original config.json that points to https://crates.io/api/v1/crates as it's dl value.

So Cargo is successfully getting the metadata about the crates from your mirror, but when it's actually attempting to download the crate files, it's going to crates.io.

You can read more about config.json here: https://doc.rust-lang.org/cargo/reference/registry-index.html#index-configuration

[cangyin]

The index has a file called config.json that specifies where to download the .crate files.

My guess is that http://mirrors.sf.com/nexus/repository/rust-crates still has the original config.json that points to https://crates.io/api/v1/crates as it's dl value.

So Cargo is successfully getting the metadata about the crates from your mirror, but when it's actually attempting to download the crate files, it's going to crates.io.

You can read more about config.json here: https://doc.rust-lang.org/cargo/reference/registry-index.html#index-configuration

Thanks for the reply. But I have to say this looks not so true for me. As:

$ grep 'crates.io' -r $CARGO_HOME/
<-- no output

$ tcpdump -A -v -i eth0  <-- pre run tcpdump in another terminal

$ cargo fetch
... fail

tcpdump shows that cargo fetch would attempt to connect to crates.io from the very beginning, without downloading the stated config.json from mirror site http://mirrors.sf.com/nexus/repository/rust-crates.

$ tcpdump -A -v -i eth0
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:01:08.101720 IP (tos 0x0, ttl 64, id 12790, offset 0, flags [DF], proto UDP (17), length 55)
    839e41e4590b.60429 > public1.114dns.com.53: 8825+ A? crates.io. (27) <-- this line, want resolving crates.io
E..71.@[email protected].#./"y...........crates.io.....
20:01:08.101826 IP (tos 0x0, ttl 64, id 12791, offset 0, flags [DF], proto UDP (17), length 55)
    839e41e4590b.60429 > public1.114dns.com.53: 23679+ AAAA? crates.io. (27)
E..71.@[email protected].#./\............crates.io.....
20:01:08.129170 IP (tos 0x0, ttl 56, id 58572, offset 0, flags [none], proto UDP (17), length 119)
    public1.114dns.com.53 > 839e41e4590b.60429: 8825 4/0/0 crates.io. A 13.224.167.52, crates.io. A 13.224.167.65, crates.io. A 13.224.167.34, crates.io. A 13.224.167.116 (91)
E..w....8...rrrr.....5...c5{"y...........crates.io....................4...............A..............."...............t

(... omited data)

strace shows cargo fetch did not open any config.json

strace cargo fetch |& grep config.json
<-- no output

[arlosi]

cargo fetch will use a local copy of the index if it's available. Try the same thing with cargo update --dry-run.

If that still doesn't explain it, run with environment variable CARGO_HTTP_DEBUG=true

Also, please post a copy of your config.json within the index repository, just we we can rule that out.

[cangyin]

$ CARGO_HTTP_DEBUG=true cargo update --dry-run
    Updating `sf-crates-proxy` index
warning: not updating lockfile due to dry run

$ export CARGO_HTTP_DEBUG=true
$ cargo update --dry-run
    Updating `sf-crates-proxy` index
warning: not updating lockfile due to dry run

Also, please post a copy of your config.json within the index repository, just we we can rule that out.

How to obtain that? Accessing "http://mirrors.sf.com/nexus/repository/rust-crates/config.json" results in 404 error. Just like accessing "https://rsproxy.cn/crates.io-index/config.json" (this is a similar mirror site)

[arlosi]

How to obtain that?

It's a git repository. For example on rsproxy.cn.

git clone https://rsproxy.cn/crates.io-index
cat crates.io-index/config.json

{
  "dl": "https://rsproxy.cn/api/v1/crates",
  "api": "https://rsproxy.cn"
}

Note that the dl key does not point to crates.io. They also have a sparse endpoint here: https://rsproxy.cn/index/config.json

[cangyin]

well, that config.json is actually pointing to crates.io

$ git clone http://mirrors.sf.com/nexus/repository/rust-crates
$ cd rust-crates
$ cat config.json
{
  "dl": "https://crates.io/api/v1/crates",
  "api": "https://crates.io"
}

that explains! So is there any method to dynamically replace this value to mirror site during cargo fetch?

[arlosi]

So is there any method to dynamically replace this value to mirror site during cargo fetch?

That's something that would need to be done on the registry server. Of course, you'll also need a place to actually host all the .crate files as well.