Fix network-online

[arianvp]

No description provided.

[hjkatz]

@arianvp QQ about network-online.target vs. network.target in NixOS.

We're continuing debugging our networking issues with AMIs from #137 and notice that amazon-init.service wants = [network-online.target] not network.target.

We saw 2 cases where we think this might be related:

1. Instances are coming online and running amazon-init.service without the IMDS being available
2. We had a user service on a host startup and it thought its hostname was localhost rather than ip-x-x-x-x... as expected.

Do you think either of these could be caused by network-online.target?

Is it possible for network-online to report true, launch other dependent targets, but then the network is not actually available?

Ref: https://systemd.io/NETWORK_ONLINE/

[arianvp]

network-online.target should be the right thing yes. But to be honest scripted networking is kind of a mess.

e.g. it's buggy when you have multiple IP addresses or multiple interfaces .

You could try if you have better luck when setting networking.useNetworkd = true;

Do you happen to run in an environment with IPV6 dual stack and/or multiple network interfaces attached and configured to the instance? That could explain the behaviour you're seeing perhaps

[hjkatz]

Do you happen to run in an environment with IPV6 dual stack and/or multiple network interfaces attached and configured to the instance? That could explain the behaviour you're seeing perhaps

Yes to both. I was reading online about this sometimes causing IMDS issues with the default gateway. I'll play around with this more tomorrow in a controlled environment.

networking.useNetworkd = true;

Would we set this in configuration.nix? But we could only do that if we're not using a shebang user-data: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/amazon-init.nix#L26

I noticed that changing configuration.nix in a user-data.sh shell script, then running nixos-rebuild switch did not seem to make nixos happy. But I'll explore this path too.