Federated runtime with TLS: network is not getting updated properly with current FQDN #1265

payalcha · 2025-01-13T07:43:17Z

Describe the bug
Tried to run Federated runtime with tls enabled. During the federation run, system is picking random port and system hostname as fqdn instead of assigned port-50050 and fqdn-localhost.

To Reproduce
Steps to reproduce the behavior:

Create certificates using below steps

2. Run envoys and director using

3. Run the notebook [MNIST_Watermarking.ipynb](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/301_MNIST_Watermarking.ipynb).

Expected behavior
Notebook must run properly without any error.

Screenshots

Desktop (please complete the following information):

OS: [e.g. iOS] - ubuntu
Browser [e.g. chrome, safari] - chrome
Version [e.g. 22] - 22
python - 3.10

Additional context
After hardcoding fqdn and port in default network.yaml it worked.
We understand that it should get updated during runtime instead of hardcoding the same.

scngupta-dsp · 2025-01-14T16:53:12Z

Hi @payalcha

We tried to follow the process that was shared and enabled TLS in the FederatedRuntime using following steps. As you can see from director and envoy logs this runs successfully

Request if you could review the steps below and share the steps you have followed to enable us to reproduce the issue

Install the CA server: fx pki install -p </path/to/ca/dir> --ca-url <host:port>

(env_openfl_latest) scngupta@soc-9TTR9K3:~/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking$ fx pki install -p server/ --ca-url localhost:50050
The password will encrypt some ca files
Enter the password:
Repeat for confirmation:
[16:03:03] INFO Creating CA ca.py:159
CA binaries will be downloaded now [Y/n]: y
[16:03:14] INFO Create CA Config ca.py:247

Generating root certificate...
all done!

Generating intermediate certificate...
all done!

✔ Root certificate: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/server/step_config/certs/root_ca.crt
✔ Root private key: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/server/step_config/secrets/root_ca_key
✔ Root fingerprint: e9575ce274f70944aae7bfb6725bc485f69caa42efabdb05da08d0736853224e
✔ Intermediate certificate: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/server/step_config/certs/intermediate_ca.crt
✔ Intermediate private key: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/server/step_config/secrets/intermediate_ca_key
✔ Database folder: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/server/step_config/db
✔ Default configuration: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/server/step_config/config/defaults.json
✔ Certificate Authority configuration: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/server/step_config/config/ca.json

Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.

FEEDBACK 😍 🍻
The step utility is not instrumented for usage statistics. It does not
phone home. But your feedback is extremely valuable. Any information you
can provide regarding how you’re using step helps. Please send us a
sentence or two, good or bad: [email protected] or join
https://github.com/smallstep/certificates/discussions.
Success! Your step-ca config has been updated. To pick up the new configuration SIGHUP (kill -1 ) or restart the step-ca process.
Your public key has been saved in /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/server/step_config/certs/pub.json.
Your private key has been saved in /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/server/step_config/secrets/priv.json.
Success! Your step-ca config has been updated. To pick up the new configuration SIGHUP (kill -1 ) or restart the step-ca process.

Run the CA Server: fx pki run -p </path/to/ca/dir>

(env_openfl_latest) scngupta@soc-9TTR9K3:~/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking$ fx pki run -p server/
[16:04:02] INFO Up CA server
ca.py:188
badger 2025/01/13 16:04:02 INFO: All 0 tables opened in 0s
2025/01/13 16:04:02 Serving HTTPS on localhost:50050 ...

Generate a token for the director: fx pki get-token -n <director_fqdn> --ca-path </path/to/ca/dir> --ca-url <host:port>

(env_openfl_latest) scngupta@soc-9TTR9K3:~/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking$ fx pki get-token -n soc-9TTR9K3 --ca-path server --ca-url localhost:50050
✔ Provisioner: provisioner (JWK) [kid: IoTD0r5mzGdWnJRapNzVp0FyVNFKuf99Q3NKKKWSORA]
Token:
ZXlKaGJHY2lPaUpGVXpJMU5pSXNJbXRwWkNJNklrbHZWRVF3Y2pWdGVrZGtWMjVLVW1Gd1RucFdjREJHZVZaT1JrdDFaams1VVROT1MwdExWMU5QVWtFaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpoZFdRaU9pSm9kSFJ3Y3pvdkwyeHZZMkZzYUc5emREbzFNREExTUM4eExqQXZjMmxuYmlJc0ltVjRjQ0k2TVRjek5qYzJORGcxT1N3aWFXRjBJam94TnpNMk56WTBOVFU1TENKcGMzTWlPaUp3Y205MmFYTnBiMjVsY2lJc0ltcDBhU0k2SWpsa05qbGxZekEwTW1SaE5Ua3daRGRrWW1WalpqaGpaVEpoTm1FMU1ERXpNV05sT1RBME56TmlNVFZqWkRnM1lURTROV00xWkdWa1pUZzNaamxoTlRnaUxDSnVZbVlpT2pFM016WTNOalExTlRrc0luTmhibk1pT2xzaWMyOWpMVGxVVkZJNVN6TWlYU3dpYzJoaElqb2laVGsxTnpWalpUSTNOR1kzTURrME5HRmhaVGRpWm1JMk56STFZbU0wT0RWbU5qbGpZV0UwTW1WbVlXSmtZakExWkdFd09HUXdOek0yT0RVek1qSTBaU0lzSW5OMVlpSTZJbk52WXkwNVZGUlNPVXN6SW4wLnhlbUdUdUl5bmxleW40LVVuX3hKcmNIZDFDbFNIMkQ3a0NYVEZkUi1PMFRzalc5cWo1S3FzaEQ3T1NoN0NaTjZzSk9vZ3FxVDd3cHIydjdYNzBjcld3.LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRhZ0F3SUJBZ0lSQVAvTHpJWE1WZmJXZ2o5OHErY2Y2Zjh3Q2dZSUtvWkl6ajBFQXdJd0pqRU4KTUFzR0ExVUVDaE1FYm1GdFpURVZNQk1HQTFVRUF4TU1ibUZ0WlNCU2IyOTBJRU5CTUI0WERUSTFNREV4TXpFdwpNek14TkZvWERUTTFNREV4TVRFd016TXhORm93SmpFTk1Bc0dBMVVFQ2hNRWJtRnRaVEVWTUJNR0ExVUVBeE1NCmJtRnRaU0JTYjI5MElFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUVLY1c3Z213aWY5UDQKaVRpaWNUeENETEgxaG5OSWJNNXZETEpBOWFYb3EzVHRtT3kzVHdIOGZUNnFTMFZWRWhGMjJQem1jYURYVS9kTQpObEVpS2NLQVNhTkZNRU13RGdZRFZSMFBBUUgvQkFRREFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUV3CkhRWURWUjBPQkJZRUZJMWdDaEVBWlV0WWJ0cmx2YVliVFE2eWhJMlVNQW9HQ0NxR1NNNDlCQU1DQTBrQU1FWUMKSVFDT1VmWTU4M3ZmMjVlVmRTUmhlTnByRm43MlQvNVdjdHBnbjNXVFJHZ0Q3d0loQUtmeE9QMDRNWDNvZTBvbApUTmRBa0t4Z3pkaVUzMTQrMXVzVGx0MlorODgvCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

✔️ OK

Certify the token for the director: fx pki certify -n <director_fqdn> -t <director_token>

(env_openfl_latest) scngupta@soc-9TTR9K3:~/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/director$ fx pki certify -n soc-9TTR9K3 -t ZXlKaGJHY2lPaUpGVXpJMU5pSXNJbXRwWkNJNklrbHZWRVF3Y2pWdGVrZGtWMjVLVW1Gd1RucFdjREJHZVZaT1JrdDFaams1VVROT1MwdExWMU5QVWtFaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpoZFdRaU9pSm9kSFJ3Y3pvdkwyeHZZMkZzYUc5emREbzFNREExTUM4eExqQXZjMmxuYmlJc0ltVjRjQ0k2TVRjek5qYzJORGcxT1N3aWFXRjBJam94TnpNMk56WTBOVFU1TENKcGMzTWlPaUp3Y205MmFYTnBiMjVsY2lJc0ltcDBhU0k2SWpsa05qbGxZekEwTW1SaE5Ua3daRGRrWW1WalpqaGpaVEpoTm1FMU1ERXpNV05sT1RBME56TmlNVFZqWkRnM1lURTROV00xWkdWa1pUZzNaamxoTlRnaUxDSnVZbVlpT2pFM016WTNOalExTlRrc0luTmhibk1pT2xzaWMyOWpMVGxVVkZJNVN6TWlYU3dpYzJoaElqb2laVGsxTnpWalpUSTNOR1kzTURrME5HRmhaVGRpWm1JMk56STFZbU0wT0RWbU5qbGpZV0UwTW1WbVlXSmtZakExWkdFd09HUXdOek0yT0RVek1qSTBaU0lzSW5OMVlpSTZJbk52WXkwNVZGUlNPVXN6SW4wLnhlbUdUdUl5bmxleW40LVVuX3hKcmNIZDFDbFNIMkQ3a0NYVEZkUi1PMFRzalc5cWo1S3FzaEQ3T1NoN0NaTjZzSk9vZ3FxVDd3cHIydjdYNzBjcld3.LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRhZ0F3SUJBZ0lSQVAvTHpJWE1WZmJXZ2o5OHErY2Y2Zjh3Q2dZSUtvWkl6ajBFQXdJd0pqRU4KTUFzR0ExVUVDaE1FYm1GdFpURVZNQk1HQTFVRUF4TU1ibUZ0WlNCU2IyOTBJRU5CTUI0WERUSTFNREV4TXpFdwpNek14TkZvWERUTTFNREV4TVRFd016TXhORm93SmpFTk1Bc0dBMVVFQ2hNRWJtRnRaVEVWTUJNR0ExVUVBeE1NCmJtRnRaU0JTYjI5MElFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUVLY1c3Z213aWY5UDQKaVRpaWNUeENETEgxaG5OSWJNNXZETEpBOWFYb3EzVHRtT3kzVHdIOGZUNnFTMFZWRWhGMjJQem1jYURYVS9kTQpObEVpS2NLQVNhTkZNRU13RGdZRFZSMFBBUUgvQkFRREFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUV3CkhRWURWUjBPQkJZRUZJMWdDaEVBWlV0WWJ0cmx2YVliVFE2eWhJMlVNQW9HQ0NxR1NNNDlCQU1DQTBrQU1FWUMKSVFDT1VmWTU4M3ZmMjVlVmRTUmhlTnByRm43MlQvNVdjdHBnbjNXVFJHZ0Q3d0loQUtmeE9QMDRNWDNvZTBvbApUTmRBa0t4Z3pkaVUzMTQrMXVzVGx0MlorODgvCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
CA binaries will be downloaded now [Y/n]: y
✔ CA: https://localhost:50050/1.0/sign
✔ Certificate: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/director/cert/soc-9TTR9K3.crt
✔ Private Key: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/director/cert/soc-9TTR9K3.key

✔️ OK

Generate a token for the envoy: fx pki get-token -n <envoy_name> --ca-path </path/to/ca/dir> --ca-url <host:port>

(env_openfl_latest) scngupta@soc-9TTR9K3:~/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/Bangalore$ fx pki get-token -n Bangalore --ca-path ../server/ --ca-url localhost:50050
✔ Provisioner: provisioner (JWK) [kid: IoTD0r5mzGdWnJRapNzVp0FyVNFKuf99Q3NKKKWSORA]
Token:
ZXlKaGJHY2lPaUpGVXpJMU5pSXNJbXRwWkNJNklrbHZWRVF3Y2pWdGVrZGtWMjVLVW1Gd1RucFdjREJHZVZaT1JrdDFaams1VVROT1MwdExWMU5QVWtFaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpoZFdRaU9pSm9kSFJ3Y3pvdkwyeHZZMkZzYUc5emREbzFNREExTUM4eExqQXZjMmxuYmlJc0ltVjRjQ0k2TVRjek5qYzJOVEExTkN3aWFXRjBJam94TnpNMk56WTBOelUwTENKcGMzTWlPaUp3Y205MmFYTnBiMjVsY2lJc0ltcDBhU0k2SWpGbVpqUXlNamMyTW1VellUbGlOMk0yT1dJNVpUZGhZemRpWWpabE9XRTVNemczWWprME1tWTVOR1JtTWpGak0yRm1ObUU1T0dJNFlqazNORGd6TWpNaUxDSnVZbVlpT2pFM016WTNOalEzTlRRc0luTmhibk1pT2xzaVFtRnVaMkZzYjNKbElsMHNJbk5vWVNJNkltVTVOVGMxWTJVeU56Um1OekE1TkRSaFlXVTNZbVppTmpjeU5XSmpORGcxWmpZNVkyRmhOREpsWm1GaVpHSXdOV1JoTURoa01EY3pOamcxTXpJeU5HVWlMQ0p6ZFdJaU9pSkNZVzVuWVd4dmNtVWlmUS56YnFOVWV2SXA4LXA3RkJuQ3lwZTBkOWthV3dhUGxxSWVwSjBvQzh3RExYUGZnZEU5NlVvNHV1cDNHLWtpQzBTT0lILWVVMmF6dkR2V0xoel85dEhEdw==.LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRhZ0F3SUJBZ0lSQVAvTHpJWE1WZmJXZ2o5OHErY2Y2Zjh3Q2dZSUtvWkl6ajBFQXdJd0pqRU4KTUFzR0ExVUVDaE1FYm1GdFpURVZNQk1HQTFVRUF4TU1ibUZ0WlNCU2IyOTBJRU5CTUI0WERUSTFNREV4TXpFdwpNek14TkZvWERUTTFNREV4TVRFd016TXhORm93SmpFTk1Bc0dBMVVFQ2hNRWJtRnRaVEVWTUJNR0ExVUVBeE1NCmJtRnRaU0JTYjI5MElFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUVLY1c3Z213aWY5UDQKaVRpaWNUeENETEgxaG5OSWJNNXZETEpBOWFYb3EzVHRtT3kzVHdIOGZUNnFTMFZWRWhGMjJQem1jYURYVS9kTQpObEVpS2NLQVNhTkZNRU13RGdZRFZSMFBBUUgvQkFRREFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUV3CkhRWURWUjBPQkJZRUZJMWdDaEVBWlV0WWJ0cmx2YVliVFE2eWhJMlVNQW9HQ0NxR1NNNDlCQU1DQTBrQU1FWUMKSVFDT1VmWTU4M3ZmMjVlVmRTUmhlTnByRm43MlQvNVdjdHBnbjNXVFJHZ0Q3d0loQUtmeE9QMDRNWDNvZTBvbApUTmRBa0t4Z3pkaVUzMTQrMXVzVGx0MlorODgvCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

✔️ OK

Certify the token for the envoy: fx pki certify -n <envoy_name> -t <envoy_token>

(env_openfl_latest) scngupta@soc-9TTR9K3:~/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/Bangalore$ fx pki certify -n Bangalore -t ZXlKaGJHY2lPaUpGVXpJMU5pSXNJbXRwWkNJNklrbHZWRVF3Y2pWdGVrZGtWMjVLVW1Gd1RucFdjREJHZVZaT1JrdDFaams1VVROT1MwdExWMU5QVWtFaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpoZFdRaU9pSm9kSFJ3Y3pvdkwyeHZZMkZzYUc5emREbzFNREExTUM4eExqQXZjMmxuYmlJc0ltVjRjQ0k2TVRjek5qYzJOVEExTkN3aWFXRjBJam94TnpNMk56WTBOelUwTENKcGMzTWlPaUp3Y205MmFYTnBiMjVsY2lJc0ltcDBhU0k2SWpGbVpqUXlNamMyTW1VellUbGlOMk0yT1dJNVpUZGhZemRpWWpabE9XRTVNemczWWprME1tWTVOR1JtTWpGak0yRm1ObUU1T0dJNFlqazNORGd6TWpNaUxDSnVZbVlpT2pFM016WTNOalEzTlRRc0luTmhibk1pT2xzaVFtRnVaMkZzYjNKbElsMHNJbk5vWVNJNkltVTVOVGMxWTJVeU56Um1OekE1TkRSaFlXVTNZbVppTmpjeU5XSmpORGcxWmpZNVkyRmhOREpsWm1GaVpHSXdOV1JoTURoa01EY3pOamcxTXpJeU5HVWlMQ0p6ZFdJaU9pSkNZVzVuWVd4dmNtVWlmUS56YnFOVWV2SXA4LXA3RkJuQ3lwZTBkOWthV3dhUGxxSWVwSjBvQzh3RExYUGZnZEU5NlVvNHV1cDNHLWtpQzBTT0lILWVVMmF6dkR2V0xoel85dEhEdw==.LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRhZ0F3SUJBZ0lSQVAvTHpJWE1WZmJXZ2o5OHErY2Y2Zjh3Q2dZSUtvWkl6ajBFQXdJd0pqRU4KTUFzR0ExVUVDaE1FYm1GdFpURVZNQk1HQTFVRUF4TU1ibUZ0WlNCU2IyOTBJRU5CTUI0WERUSTFNREV4TXpFdwpNek14TkZvWERUTTFNREV4TVRFd016TXhORm93SmpFTk1Bc0dBMVVFQ2hNRWJtRnRaVEVWTUJNR0ExVUVBeE1NCmJtRnRaU0JTYjI5MElFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUVLY1c3Z213aWY5UDQKaVRpaWNUeENETEgxaG5OSWJNNXZETEpBOWFYb3EzVHRtT3kzVHdIOGZUNnFTMFZWRWhGMjJQem1jYURYVS9kTQpObEVpS2NLQVNhTkZNRU13RGdZRFZSMFBBUUgvQkFRREFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUV3CkhRWURWUjBPQkJZRUZJMWdDaEVBWlV0WWJ0cmx2YVliVFE2eWhJMlVNQW9HQ0NxR1NNNDlCQU1DQTBrQU1FWUMKSVFDT1VmWTU4M3ZmMjVlVmRTUmhlTnByRm43MlQvNVdjdHBnbjNXVFJHZ0Q3d0loQUtmeE9QMDRNWDNvZTBvbApUTmRBa0t4Z3pkaVUzMTQrMXVzVGx0MlorODgvCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
CA binaries will be downloaded now [Y/n]: y
✔ CA: https://localhost:50050/1.0/sign
✔ Certificate: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/Bangalore/cert/Bangalore.crt
✔ Private Key: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/Bangalore/cert/Bangalore.key

✔️ OK

Same steps repeated for Chandler

Generate token for experiment manager: fx pki get-token -n <manager> --ca-path </path/to/ca/dir> --ca-url <host:port>

✔ Provisioner: provisioner (JWK) [kid: IoTD0r5mzGdWnJRapNzVp0FyVNFKuf99Q3NKKKWSORA]
Token:
ZXlKaGJHY2lPaUpGVXpJMU5pSXNJbXRwWkNJNklrbHZWRVF3Y2pWdGVrZGtWMjVLVW1Gd1RucFdjREJHZVZaT1JrdDFaams1VVROT1MwdExWMU5QVWtFaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpoZFdRaU9pSm9kSFJ3Y3pvdkwyeHZZMkZzYUc5emREbzFNREExTUM4eExqQXZjMmxuYmlJc0ltVjRjQ0k2TVRjek5qYzJOVE13T0N3aWFXRjBJam94TnpNMk56WTFNREE0TENKcGMzTWlPaUp3Y205MmFYTnBiMjVsY2lJc0ltcDBhU0k2SW1WbVpHTmlaV1JtWTJOa01UWmhPV0kzTUdRMVlUbGlOV0V6T1dJM05UbGhNMkprWkRkbVpHUTFNelF3WVdGaE1EUTJOall3WkRCa1pUWXhObVZsTWpraUxDSnVZbVlpT2pFM016WTNOalV3TURnc0luTmhibk1pT2xzaVRXRnVZV2RsY2lKZExDSnphR0VpT2lKbE9UVTNOV05sTWpjMFpqY3dPVFEwWVdGbE4ySm1ZalkzTWpWaVl6UTROV1kyT1dOaFlUUXlaV1poWW1SaU1EVmtZVEE0WkRBM016WTROVE15TWpSbElpd2ljM1ZpSWpvaVRXRnVZV2RsY2lKOS5MaldiVDYxWmVmWVlsWGdmVF82OTRKWHdFR3hzRU1PT2NmZFIxOW1hYlBLSTRwMUlxZHJ3X3FIeWlIR1I3cUhLbk9iZTJzd2FNMVNnWkgyMVhTT3l5dw==.LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRhZ0F3SUJBZ0lSQVAvTHpJWE1WZmJXZ2o5OHErY2Y2Zjh3Q2dZSUtvWkl6ajBFQXdJd0pqRU4KTUFzR0ExVUVDaE1FYm1GdFpURVZNQk1HQTFVRUF4TU1ibUZ0WlNCU2IyOTBJRU5CTUI0WERUSTFNREV4TXpFdwpNek14TkZvWERUTTFNREV4TVRFd016TXhORm93SmpFTk1Bc0dBMVVFQ2hNRWJtRnRaVEVWTUJNR0ExVUVBeE1NCmJtRnRaU0JTYjI5MElFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUVLY1c3Z213aWY5UDQKaVRpaWNUeENETEgxaG5OSWJNNXZETEpBOWFYb3EzVHRtT3kzVHdIOGZUNnFTMFZWRWhGMjJQem1jYURYVS9kTQpObEVpS2NLQVNhTkZNRU13RGdZRFZSMFBBUUgvQkFRREFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUV3CkhRWURWUjBPQkJZRUZJMWdDaEVBWlV0WWJ0cmx2YVliVFE2eWhJMlVNQW9HQ0NxR1NNNDlCQU1DQTBrQU1FWUMKSVFDT1VmWTU4M3ZmMjVlVmRTUmhlTnByRm43MlQvNVdjdHBnbjNXVFJHZ0Q3d0loQUtmeE9QMDRNWDNvZTBvbApUTmRBa0t4Z3pkaVUzMTQrMXVzVGx0MlorODgvCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

Certify the token for the experiment manager: fx pki certify -n <manager> -t <manager_token>

(env_openfl_latest) scngupta@soc-9TTR9K3:~/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/workspace$ fx pki certify -n Manager -t ZXlKaGJHY2lPaUpGVXpJMU5pSXNJbXRwWkNJNklrbHZWRVF3Y2pWdGVrZGtWMjVLVW1Gd1RucFdjREJHZVZaT1JrdDFaams1VVROT1MwdExWMU5QVWtFaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpoZFdRaU9pSm9kSFJ3Y3pvdkwyeHZZMkZzYUc5emREbzFNREExTUM4eExqQXZjMmxuYmlJc0ltVjRjQ0k2TVRjek5qYzJOVE13T0N3aWFXRjBJam94TnpNMk56WTFNREE0TENKcGMzTWlPaUp3Y205MmFYTnBiMjVsY2lJc0ltcDBhU0k2SW1WbVpHTmlaV1JtWTJOa01UWmhPV0kzTUdRMVlUbGlOV0V6T1dJM05UbGhNMkprWkRkbVpHUTFNelF3WVdGaE1EUTJOall3WkRCa1pUWXhObVZsTWpraUxDSnVZbVlpT2pFM016WTNOalV3TURnc0luTmhibk1pT2xzaVRXRnVZV2RsY2lKZExDSnphR0VpT2lKbE9UVTNOV05sTWpjMFpqY3dPVFEwWVdGbE4ySm1ZalkzTWpWaVl6UTROV1kyT1dOaFlUUXlaV1poWW1SaU1EVmtZVEE0WkRBM016WTROVE15TWpSbElpd2ljM1ZpSWpvaVRXRnVZV2RsY2lKOS5MaldiVDYxWmVmWVlsWGdmVF82OTRKWHdFR3hzRU1PT2NmZFIxOW1hYlBLSTRwMUlxZHJ3X3FIeWlIR1I3cUhLbk9iZTJzd2FNMVNnWkgyMVhTT3l5dw==.LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRhZ0F3SUJBZ0lSQVAvTHpJWE1WZmJXZ2o5OHErY2Y2Zjh3Q2dZSUtvWkl6ajBFQXdJd0pqRU4KTUFzR0ExVUVDaE1FYm1GdFpURVZNQk1HQTFVRUF4TU1ibUZ0WlNCU2IyOTBJRU5CTUI0WERUSTFNREV4TXpFdwpNek14TkZvWERUTTFNREV4TVRFd016TXhORm93SmpFTk1Bc0dBMVVFQ2hNRWJtRnRaVEVWTUJNR0ExVUVBeE1NCmJtRnRaU0JTYjI5MElFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUVLY1c3Z213aWY5UDQKaVRpaWNUeENETEgxaG5OSWJNNXZETEpBOWFYb3EzVHRtT3kzVHdIOGZUNnFTMFZWRWhGMjJQem1jYURYVS9kTQpObEVpS2NLQVNhTkZNRU13RGdZRFZSMFBBUUgvQkFRREFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUV3CkhRWURWUjBPQkJZRUZJMWdDaEVBWlV0WWJ0cmx2YVliVFE2eWhJMlVNQW9HQ0NxR1NNNDlCQU1DQTBrQU1FWUMKSVFDT1VmWTU4M3ZmMjVlVmRTUmhlTnByRm43MlQvNVdjdHBnbjNXVFJHZ0Q3d0loQUtmeE9QMDRNWDNvZTBvbApUTmRBa0t4Z3pkaVUzMTQrMXVzVGx0MlorODgvCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
CA binaries will be downloaded now [Y/n]: y
✔ CA: https://localhost:50050/1.0/sign
✔ Certificate: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/workspace/cert/Manager.crt
✔ Private Key: /home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/workspace/cert/Manager.key

✔️ OK

Start Director: fx director start -c director_config.yaml -rc cert/root_ca.crt -pk cert/soc-9TTR9K3.key -oc cert/soc-9TTR9K3.crt
Settings modified to below to avoid failure in connection on envoys

settings:
listen_host: 0.0.0.0
listen_port: 50050
envoy_health_check_period: 5 # in seconds

Start Bangalore: fx envoy start -n Bangalore --envoy-config-path Bangalore_config.yaml -dh soc-9TTR9K3 -dp 50050 -rc cert/root_ca.crt -pk cert/Bangalore.key -oc cert/Bangalore.crt

Start Chandler: fx envoy start -n Chandler --envoy-config-path Chandler_config.yaml -dh soc-9TTR9K3 -dp 50050 -rc cert/root_ca.crt -pk cert/Chandler.key -oc cert/Chandler.crt

Jupyter Notebook: edited code as follows

director_info = {
    'director_node_fqdn':'soc-9TTR9K3',
    'director_port':50050,
    'cert_chain': '/home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/workspace/cert/root_ca.crt',
    'api_cert': '/home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/workspace/cert/Manager.crt',
    'api_private_key': '/home/scngupta/openfl_latest/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/workspace/cert/Manager.key',    
}

authorized_collaborators = ['Bangalore', 'Chandler']

federated_runtime = FederatedRuntime(
    collaborators=authorized_collaborators,
    director=director_info, 
    notebook_path='./MNIST_Watermarking.ipynb',
    tls=True
)

LOGS
director_with_tls.log
Bangalore_with_tls.log
Chandler_with_tls.log

ishant162 · 2025-01-17T12:16:38Z

Initial analysis of logs shared by @payalcha and @noopurintel, indicates that there is a hostname mismatch between the Aggregator and the Director.

Here's a detailed breakdown:

Director's Hostname:

The Director is configured to use localhost as its hostname.

(tls_fix) ishant@soc-PF2F2E02:~/tls_issue/openfl/openfl-tutorials/experimental/workflow/FederatedRuntime/301_MNIST_Watermaking/director$ fx director start -c director_config.yaml -rc cert/root_ca.crt -pk cert/localhost.key -oc cert/localhost.crt
/home/ishant/miniforge3/envs/tls_fix/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
[12:15:01] INFO     🧿 Starting the Director Service.                                                                                             director.py:86
           INFO     Starting director server on localhost:50050                                                                           director_server.py:146
[12:15:04] INFO     Envoy Bangalore is attempting to connect                                                                              director_server.py:179
           INFO     Envoy Bangalore is connected                                                                                          director_server.py:182
[12:15:33] INFO     Experiment FederatedFlow_MNIST_Watermarking registered                                                                director_server.py:314

Aggregator's Hostname:

During plan parsing the Aggregator's hostname resolves to the machine's FQDN.

INFO     FL-Plan hash is 4794523320c407e0031c301f6b13c63c5d5775c829163efeb254226fc9104fd31c1c82cafea463f5e0a77d16aa9dca1b                 plan.py:267
           INFO     FL-Plan hash is 9ef6330c565bbde25bfddf17852707863d0bd2ecee6a001082cc4ca1dc6af00be38e8db81cd01b8568156dfb1a9e0321                 plan.py:267
           INFO     Parsing Federated Learning Plan : SUCCESS : plan/plan.yaml.                                                                      plan.py:161
           INFO     aggregator:                                                                                                                      plan.py:166
                      settings:
                        rounds_to_train: 1
                      template: openfl.experimental.workflow.component.Aggregator
                    collaborator:
                      settings: {}
                      template: openfl.experimental.workflow.component.Collaborator
                    federated_flow:
                      settings:
                        checkpoint: true
                        model: src.experiment.model
                        optimizer: src.experiment.optimizer
                        watermark_pretrain_optimizer: src.experiment.watermark_pretrain_optimizer
                        watermark_retrain_optimizer: src.experiment.watermark_retrain_optimizer
                      template: src.experiment.FederatedFlow_MNIST_Watermarking
                    network:
                      settings:
                        agg_addr: soc-PF2F2E02.clients.intel.com  // Machine's FQDN
                        agg_port: 58646
                        cert_folder: cert
                        client_reconnect_interval: 5
                        disable_client_auth: false
                        hash_salt: auto
                        tls: true
                      template: openfl.federation.Network

Subsequently when collaborator tries to connect to the aggregator the connection fails probably due to hostname discrepency.

Next Steps: Need to analyze the hostnames used in aggregator server and collaborator clients.

ishant162 self-assigned this Jan 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Federated runtime with TLS: network is not getting updated properly with current FQDN #1265

Federated runtime with TLS: network is not getting updated properly with current FQDN #1265

payalcha commented Jan 13, 2025 •

edited

Loading

scngupta-dsp commented Jan 14, 2025

ishant162 commented Jan 17, 2025

Federated runtime with TLS: network is not getting updated properly with current FQDN #1265

Federated runtime with TLS: network is not getting updated properly with current FQDN #1265

Comments

payalcha commented Jan 13, 2025 • edited Loading

scngupta-dsp commented Jan 14, 2025

ishant162 commented Jan 17, 2025

payalcha commented Jan 13, 2025 •

edited

Loading