Build failure: pulumi-resource-aws v6.28.1 (unstable) #300994

jmastr · 2024-04-02T16:06:43Z

Steps To Reproduce

Steps to reproduce the behavior:

build pulumi-bin v3.112.0 from unstable (f3842e0)

Build log

unpacking channels...
building Nix...
building the system configuration...
these 10 derivations will be built:
  /nix/store/9cry4zbp8bdcs9d760h5xwaysn7ll04z-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz.drv
  /nix/store/ncy6ilnjiczrpkgk7dzhgimdmirxza2g-pulumi-3.112.0.drv
  /nix/store/ycknakgqa66d2w6p9zy42chrdf297kh2-home-manager-path.drv
  /nix/store/5l0kd0ka9vkbyq7c4fsdwcxs3zb1imvm-activation-script.drv
  /nix/store/mcj5k69mhh4pgkfhva9y6x57gb92svli-home-manager-generation.drv
  /nix/store/q5winh1vzm9pkcx30x51mw7kifq0l9rr-unit-home-manager-julian.service.drv
  /nix/store/mq68ijc4nipjnmdnhrbw54nprhhc4chg-system-units.drv
  /nix/store/mqw9047aswb4h6y36806d8nv4hhvych7-etc-current-packages.drv
  /nix/store/c373qsdwpbnjpwwii9j4hxxjlrg2hf67-etc.drv
  /nix/store/cn4x3hifiml2my0x6fsvsc47pwif23nf-nixos-system-nixos-23.11.5742.219951b495fc.drv
building '/nix/store/mqw9047aswb4h6y36806d8nv4hhvych7-etc-current-packages.drv'...
building '/nix/store/9cry4zbp8bdcs9d760h5xwaysn7ll04z-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz.drv'...

trying https://api.pulumi.com/releases/plugins/pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   121  100   121    0     0    508      0 --:--:-- --:--:-- --:--:--   508
100  216M  100  216M    0     0  10.7M      0  0:00:20  0:00:20 --:--:-- 11.6M
error: hash mismatch in fixed-output derivation '/nix/store/9cry4zbp8bdcs9d760h5xwaysn7ll04z-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz.drv':
         specified: sha256-pxfmEmc1cbvIygamFiVyEp7mCXyBnqpyydErXVXMKK0=
            got:    sha256-YcvSerK0X3YDLP/IiiKmG3jJ7H4exWV1pZ2cLp9GVfs=
error: 1 dependencies of derivation '/nix/store/ncy6ilnjiczrpkgk7dzhgimdmirxza2g-pulumi-3.112.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/ycknakgqa66d2w6p9zy42chrdf297kh2-home-manager-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/mcj5k69mhh4pgkfhva9y6x57gb92svli-home-manager-generation.drv' failed to build
error: 1 dependencies of derivation '/nix/store/q5winh1vzm9pkcx30x51mw7kifq0l9rr-unit-home-manager-julian.service.drv' failed to build
error: 1 dependencies of derivation '/nix/store/mq68ijc4nipjnmdnhrbw54nprhhc4chg-system-units.drv' failed to build
error: 1 dependencies of derivation '/nix/store/c373qsdwpbnjpwwii9j4hxxjlrg2hf67-etc.drv' failed to build
error: 1 dependencies of derivation '/nix/store/cn4x3hifiml2my0x6fsvsc47pwif23nf-nixos-system-nixos-23.11.5742.219951b495fc.drv' failed to build

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.8.1, NixOS, 23.11 (Tapir), 23.11.5742.219951b495fc`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(julian): `""`
 - channels(root): `"home-manager-23.11.tar.gz, nixos-23.11, nixos-jmastr, nixos-unstable"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Add a 👍 reaction to issues you find important.

The text was updated successfully, but these errors were encountered:

jmastr · 2024-04-02T16:09:54Z

@wrbbz could you have a look?

wrbbz · 2024-04-02T21:17:31Z

I'll take a look at that. However, as I can see, some versions were bumped in that commit already.

wrbbz · 2024-04-02T21:27:27Z

Also, as I can see, nix-prefetch-url of this Pulumi resource shows the same sha:

❯ nix-prefetch-url https://api.pulumi.com/releases/plugins/pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz
path is '/nix/store/b9j1adam9ym2cmqixygnk4f6fb17cg9j-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz'
1b98riamsayir5ram7l1gh4yd7hjf8jid9h6rb4bnw9mcw9fc5x7

My system is:

❯ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.7.11, NixOS, 24.05 (Uakari), 24.05.20240329.d8fe5e6`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - nixpkgs: `/nix/store/cb1gs888vfqxawvc65q1dk6jzbayh3wz-source`

jmastr · 2024-04-03T04:53:23Z

@wrbbz Interesting. On my system it is:

> nix-prefetch-url https://api.pulumi.com/releases/plugins/pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz
path is '/nix/store/5cvlrp9435p04ml8nqxzb3h2pn1vpfnr-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz'
1ysm8sgjx74xlmsnbi8ygvncjy0vlqi8mj7z5h1pcpxln9xd5jv1

jmastr · 2024-04-03T04:56:11Z

If I disconnect from my VPN:

> nmcli con d xyz.vpn         
Connection 'xyz.vpn' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/9)

Then I get your result:

> nix-prefetch-url https://api.pulumi.com/releases/plugins/pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz
path is '/nix/store/b9j1adam9ym2cmqixygnk4f6fb17cg9j-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz'
1b98riamsayir5ram7l1gh4yd7hjf8jid9h6rb4bnw9mcw9fc5x7

So this ticket can be closed.

However I really wonder what NetworkManager does in the background and why it only happens with this package...

wrbbz · 2024-04-03T07:06:53Z

An interesting thing:
If I disconnect from my VPN, I'll get result like yours:

❯ nix-prefetch-url https://api.pulumi.com/releases/plugins/pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz
path is '/nix/store/5cvlrp9435p04ml8nqxzb3h2pn1vpfnr-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz'
1ysm8sgjx74xlmsnbi8ygvncjy0vlqi8mj7z5h1pcpxln9xd5jv1

However, my VPN works on router level, not on NetworkManager

ejiek · 2024-04-03T10:51:08Z

I've test from different locations and got both:

1b98riamsayir5ram7l1gh4yd7hjf8jid9h6rb4bnw9mcw9fc5x7

and

1ysm8sgjx74xlmsnbi8ygvncjy0vlqi8mj7z5h1pcpxln9xd5jv1

While archives are different, files inside are identical:

61cbd27ab2b45f76032cffc88a22a61b78c9ec7e1ec56575a59d9c2e9f4655fb  1y/5cvlrp9435p04ml8nqxzb3h2pn1vpfnr-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz
26f1c56113856e3ffa785ba3600fa53b6cde57d89f62e97db96c49408fe66845  1y/CHANGELOG_OLD.md
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  1y/LICENSE
da8953d965a1b2bb916513833e4ae8a0076617b00e33c5ce42150a97b0d3c544  1y/pulumi-resource-aws
0f5c4bdfa8dc89d42880cd4ea78b3398d17651c3e21d2ec5a3fdd3a7c8ac5904  1y/README.md

a717e612673571bbc8ca06a6162572129ee6097c819eaa72c9d12b5d55cc28ad  1b/b9j1adam9ym2cmqixygnk4f6fb17cg9j-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz
26f1c56113856e3ffa785ba3600fa53b6cde57d89f62e97db96c49408fe66845  1b/CHANGELOG_OLD.md
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  1b/LICENSE
da8953d965a1b2bb916513833e4ae8a0076617b00e33c5ce42150a97b0d3c544  1b/pulumi-resource-aws
0f5c4bdfa8dc89d42880cd4ea78b3398d17651c3e21d2ec5a3fdd3a7c8ac5904  1b/README.md

It seems that archives where created in a different manner. Here's file output for each:

1y/5cvlrp9435p04ml8nqxzb3h2pn1vpfnr-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz: gzip compressed data, max compression, original size modulo 2^32 678070784 gzip compressed data, reserved method, ASCII, has CRC, extra field, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 678070784

1b/b9j1adam9ym2cmqixygnk4f6fb17cg9j-pulumi-resource-aws-v6.28.1-linux-amd64.tar.gz: gzip compressed data, max compression, original size modulo 2^32 678070784 gzip compressed data, reserved method, ASCII, has CRC, extra field, has comment, encrypted, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 678070784

1b has fields has comment, encrypted that 1y doesn't. I haven't figured out how to view this difference. My assumption is that it's some CDN thing with compression settings being different for different ~~endpoints~~ locations.

jmastr · 2024-04-03T10:54:14Z

@ejiek Great investigation and it totally makes sense. Depending which CDN endpoint is closer (either to the physical or to the VPN location) a different file is served...

jmastr · 2024-04-03T10:57:12Z

So probably the "fix" is to ask Pulumi to clear the CDN cache or to wait for the next version and hope the problem disappears

wrbbz · 2024-04-03T11:10:27Z

The next version is already available (6.28.2). And it is already in master branch - commit

Both with VPN and without, they have the same hash: 1bdig7fc2dh5rc1m1i7q74m1kpal8cykhifxp71pg9qx3ddck8sf

ejiek · 2024-04-03T11:20:03Z

I've dug a bit further. To get a better CDN guestimate.

drill api.pulumi.com
. . .
api.pulumi.com.	60	IN	A	13.227.219.5
api.pulumi.com.	60	IN	A	13.227.219.123
api.pulumi.com.	60	IN	A	13.227.219.16
api.pulumi.com.	60	IN	A	13.227.219.63
. . .

whois 13.227.219.5
. . .
Organization:   Amazon.com, Inc. (AMAZON-4)
. . .

Same for the rest of them. It might be safe safe to assume that we are dealing with AWS CDN.
I'm aware that there is GZIP compression option in CloudFront but I doubt it can be used to compress a tar archive.

Something dubious is happening. It might be that pulumi team uploaded a new version of a file while the old one is cached in some CDN locations. Which seems like a bad practice. Or there is something else. Either way it might happen again.

@jmastr @wrbbz Do you know a good way to communicate with pulumi team?

wrbbz · 2024-04-03T11:22:33Z

Only via Issues in their repo, I suppose

wrbbz · 2024-04-03T13:31:56Z

@ejiek, thanks for the issue and your investigation

ejiek · 2024-04-03T19:48:25Z

We have our answers 🥳

PyPI outage caused pulumi team to run CI twice. This situation is an exception so I suggest we just leave it be for the v6.28.1.

Our pulumi issue was moved from pulumi-aws repo to the CI managment one to deal with the fact that pulumi pipelines might generate files inconsistancy in some rare cases.

I suggest closing this one unless we want to ask pulumi team to clean the v6.28.1 release.

jmastr · 2024-04-04T06:10:25Z

@wrbbz and @ejiek thank you for your quick investigations and responses. I too think it’s fine to close this issue and wait for the next version. Cheers

jmastr added the 0.kind: build failure A package fails to build label Apr 2, 2024

ejiek mentioned this issue Apr 3, 2024

Disribution files for the same version may differ api.pulumi.com pulumi/ci-mgmt#874

Open

jmastr closed this as completed Apr 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build failure: pulumi-resource-aws v6.28.1 (unstable) #300994

Build failure: pulumi-resource-aws v6.28.1 (unstable) #300994

jmastr commented Apr 2, 2024

jmastr commented Apr 2, 2024

wrbbz commented Apr 2, 2024

wrbbz commented Apr 2, 2024

jmastr commented Apr 3, 2024

jmastr commented Apr 3, 2024

wrbbz commented Apr 3, 2024

ejiek commented Apr 3, 2024 •

edited

Loading

jmastr commented Apr 3, 2024

jmastr commented Apr 3, 2024

wrbbz commented Apr 3, 2024

ejiek commented Apr 3, 2024

wrbbz commented Apr 3, 2024

wrbbz commented Apr 3, 2024

ejiek commented Apr 3, 2024

jmastr commented Apr 4, 2024

Build failure: pulumi-resource-aws v6.28.1 (unstable) #300994

Build failure: pulumi-resource-aws v6.28.1 (unstable) #300994

Comments

jmastr commented Apr 2, 2024

Steps To Reproduce

Build log

Metadata

jmastr commented Apr 2, 2024

wrbbz commented Apr 2, 2024

wrbbz commented Apr 2, 2024

jmastr commented Apr 3, 2024

jmastr commented Apr 3, 2024

wrbbz commented Apr 3, 2024

ejiek commented Apr 3, 2024 • edited Loading

jmastr commented Apr 3, 2024

jmastr commented Apr 3, 2024

wrbbz commented Apr 3, 2024

ejiek commented Apr 3, 2024

wrbbz commented Apr 3, 2024

wrbbz commented Apr 3, 2024

ejiek commented Apr 3, 2024

jmastr commented Apr 4, 2024

ejiek commented Apr 3, 2024 •

edited

Loading