Charlie/dev/v0.7.0 #2236
Charlie/dev/v0.7.0 #2236
Conversation
[CoreEngine] make the server status work.
Alexleung/dev v070 for refactor
Fix Docker Issues
Alexleung/dev branch latest
[Deploy] Try to convert the gpu_topology value type to int.
[Deploy] Fix timezone issue when using pandas
…ntainer inference logs to the single dir.
[CoreEngine] In order to make the inference logs work, we save the co…
In order to make the inference logs work, we save the container inference logs to the single dir
Dev/v0.7.0
Alexleung/dev v070 for refactor
[Deploy] Avoid re-download the same model serving package.
Enabling grpc to work with docker containers.
Parameterizing deploy host, port.
[Deploy] Edge Case Handling.
[fixbug]1.read inference proxy port from config file 2.occupy_gpu_ids fail,get gpu ids from cache may return [] instead of None
… with init gpu ids because of the system gpu resource may change) and logging in hardware and job utilities
…U IDs when reset available gpus
…nd initial_available_gpu_ids
Merge pull request #2233 from FedML-AI/charlie/dev/v0.7.0
Revert "Merge pull request #2233 from FedML-AI/charlie/dev/v0.7.0"
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| + " [email protected] -n fedml-devops-aggregator-" | ||
| + self.version | ||
| ) | ||
| logging.info("Create secret cmd: " + registry_secret_cmd) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 2 months ago
To fix the problem, we need to avoid logging sensitive information in clear text. Instead of logging the entire registry_secret_cmd, we can log a sanitized version of it that omits the sensitive parts. This way, we still get useful logging information without exposing sensitive data.
- Identify the lines where sensitive information is being logged.
- Replace the logging statements with sanitized versions that exclude sensitive data.
- Ensure that the functionality of the code remains unchanged.
| @@ -83,3 +83,19 @@ | ||
| ) | ||
| logging.info("Create secret cmd: " + registry_secret_cmd) | ||
| sanitized_registry_secret_cmd = ( | ||
| "kubectl create namespace fedml-devops-aggregator-" | ||
| + self.version | ||
| + ";kubectl -n fedml-devops-aggregator-" | ||
| + self.version | ||
| + " delete secret secret-" | ||
| + self.cloud_server_name | ||
| + " ;kubectl create secret docker-registry secret-" | ||
| + self.cloud_server_name | ||
| + " --docker-server=" | ||
| + self.agent_config["docker_config"]["registry_server"] | ||
| + " --docker-username=***" | ||
| + " --docker-password=***" | ||
| + " [email protected] -n fedml-devops-aggregator-" | ||
| + self.version | ||
| ) | ||
| logging.info("Create secret cmd: " + sanitized_registry_secret_cmd) | ||
| os.system(registry_secret_cmd) |
| except Exception: | ||
| logging.error("Failed to connect to the docker daemon, please ensure that you have " | ||
| "installed Docker Desktop or Docker Engine, and the docker is running") | ||
| return "", "", None, None, None | ||
|
|
||
| # Pull the inference image | ||
| logging.info(f"Start pulling the inference image {inference_image_name}... with policy {image_pull_policy}") |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 2 months ago
To fix the problem, we should ensure that sensitive information such as passwords is not logged. We can achieve this by sanitizing the inference_image_name before logging it or by avoiding logging it altogether if it contains sensitive data. In this case, we will sanitize the inference_image_name to ensure it does not contain sensitive information before logging it.
- Identify the logging statement that includes potentially sensitive information.
- Sanitize the
inference_image_nameto ensure it does not contain sensitive data. - Update the logging statement to use the sanitized version of
inference_image_name.
| @@ -144,3 +144,4 @@ | ||
| # Pull the inference image | ||
| logging.info(f"Start pulling the inference image {inference_image_name}... with policy {image_pull_policy}") | ||
| sanitized_inference_image_name = inference_image_name.split('/')[-1] # Extract the image name without registry info | ||
| logging.info(f"Start pulling the inference image {sanitized_inference_image_name}... with policy {image_pull_policy}") | ||
| ContainerUtils.get_instance().pull_image_with_policy(image_pull_policy, inference_image_name) |
| except Exception as e: | ||
| inference_response = {"error": True, "message": f"{traceback.format_exc()}"} | ||
|
|
||
| return inference_response |
Check warning
Code scanning / CodeQL
Information exposure through an exception Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 2 months ago
To fix the problem, we need to ensure that stack traces are not exposed to end users. Instead, we should log the stack trace on the server and return a generic error message to the user. This can be achieved by modifying the exception handling code to log the stack trace and return a generic error message.
- Modify the exception handling code in the
predict,predict_openai,predict_with_end_point_id, andcustom_inferencefunctions to log the stack trace and return a generic error message. - Ensure that the logging configuration is set up to capture the stack trace information.
| @@ -117,3 +117,4 @@ | ||
| except Exception as e: | ||
| response = {"error": True, "message": f"{traceback.format_exc()}"} | ||
| logging.error(traceback.format_exc()) | ||
| response = {"error": True, "message": "An internal error has occurred!"} | ||
|
|
||
| @@ -141,3 +142,4 @@ | ||
| except Exception as e: | ||
| response = {"error": True, "message": f"{traceback.format_exc()}, exception {e}"} | ||
| logging.error(traceback.format_exc()) | ||
| response = {"error": True, "message": "An internal error has occurred!"} | ||
|
|
||
| @@ -167,3 +169,4 @@ | ||
| except Exception as e: | ||
| inference_response = {"error": True, "message": f"{traceback.format_exc()}"} | ||
| logging.error(traceback.format_exc()) | ||
| inference_response = {"error": True, "message": "An internal error has occurred!"} | ||
|
|
||
| @@ -183,3 +186,4 @@ | ||
| except Exception as e: | ||
| inference_response = {"error": True, "message": f"{traceback.format_exc()}"} | ||
| logging.error(traceback.format_exc()) | ||
| inference_response = {"error": True, "message": "An internal error has occurred!"} | ||
|
|
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 11782617 | Triggered | Generic High Entropy Secret | 0491bb7 | .github/workflows/registry-runners/Dockerfile | View secret |
| 5451874 | Triggered | Generic Password | 87ae30a | python/fedml/computing/scheduler/model_scheduler/master_job_runner.py | View secret |
| 11782618 | Triggered | Generic High Entropy Secret | a5bbcd2 | .github/workflows/registry-runners/windows.ps1 | View secret |
| 5692101 | Triggered | Generic High Entropy Secret | a932082 | python/fedml/computing/scheduler/model_scheduler/device_model_deployment.py | View secret |
| 9453265 | Triggered | Generic High Entropy Secret | 87ae30a | python/fedml/api/api_test.py | View secret |
| 8762943 | Triggered | Generic Password | 87ae30a | python/fedml/computing/scheduler/scheduler_core/compute_cache_manager.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
[bugfix]1.Enhance GPU cache management by setting initial available GPU IDs when reset available gpus 2.set shm_size to 8G if not specified.