GitHub Actions / Unit Test Results
failed
Feb 5, 2024 in 0s
1 fail, 5 skipped, 6 pass in 2m 58s
Annotations
Check warning on line 0 in tests.regression_tests.benchmark.test_model_performance
github-actions / Unit Test Results
All 4 runs failed: test_performance[sarcos.gbm.yaml] (tests.regression_tests.benchmark.test_model_performance)
artifacts/Unit Test Results (Python 3.8 distributed)/pytest.xml
artifacts/Unit Test Results (Python 3.8 not distributed)/pytest.xml
artifacts/Unit Test Results (Python 3.9 distributed)/pytest.xml
artifacts/Unit Test Results (Python 3.9 not distributed)/pytest.xml
Raw output
config_filename = 'sarcos.gbm.yaml'
tmpdir = local('/tmp/pytest-of-runner/pytest-1/test_performance_sarcos_gbm_ya0')
@pytest.mark.benchmark
@pytest.mark.parametrize("config_filename", get_test_config_filenames())
def test_performance(config_filename, tmpdir):
update_skipped_configs_issues(config_filename)
if config_filename in SKIPPED_CONFIG_ISSUES:
pytest.skip(reason=SKIPPED_CONFIG_ISSUES[config_filename])
return
benchmark_directory = "/".join(__file__.split("/")[:-1])
config_path = os.path.join(benchmark_directory, "configs", config_filename)
expected_test_statistics_fp = os.path.join(benchmark_directory, "expected_metrics", config_filename)
dataset_name = get_dataset_from_config_path(config_path)
if not os.path.exists(expected_test_statistics_fp):
raise FileNotFoundError(
"""No corresponding expected metrics found for benchmarking config '{config_path}'.
Please add a new metrics YAML file '{expected_test_statistics_fp}'. Suggested content:
metrics:
- output_feature_name: <YOUR_OUTPUT_FEATURE e.g. SalePrice>
metric_name: <YOUR METRIC NAME e.g. accuracy>
expected_value: <A FLOAT VALUE>
tolerance_percent: 0.15"""
)
expected_metrics_dict = load_yaml(expected_test_statistics_fp)
benchmarking_config = {
"experiment_name": "regression_test",
"export": {"export_artifacts": True, "export_base_path": tmpdir},
"experiments": [{"dataset_name": dataset_name, "config_path": config_path}],
}
benchmarking_artifacts = benchmark(benchmarking_config)
experiment_artifact, err = benchmarking_artifacts[dataset_name]
if err is not None:
> raise err
tests/regression_tests/benchmark/test_model_performance.py:75:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ludwig/benchmarking/benchmark.py:130: in benchmark
benchmark_one(experiment)
ludwig/benchmarking/benchmark.py:71: in benchmark_one
dataset = load_from_module(dataset_module, model_config["output_features"][0])
ludwig/benchmarking/utils.py:46: in load_from_module
dataset = dataset_module.load(split=False)
ludwig/datasets/loaders/dataset_loader.py:301: in load
self._download_and_process(kaggle_username=kaggle_username, kaggle_key=kaggle_key)
ludwig/datasets/loaders/dataset_loader.py:274: in _download_and_process
self.verify()
ludwig/datasets/loaders/dataset_loader.py:344: in verify
digest = _sha256_digest(os.path.join(self.raw_dataset_dir, filename))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
file_path = '/home/runner/.ludwig_cache/sarcos_1.0/raw/sarcos_inv_test.mat'
def _sha256_digest(file_path) -> str:
"""Returns the sha256 digest for the specified file."""
hash = hashlib.sha256()
buffer = bytearray(hash.block_size * 1024) # Attempts to read in multiples of the hash block size (64KB).
mv = memoryview(buffer)
> with open(file_path, "rb", buffering=0) as f:
E FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.ludwig_cache/sarcos_1.0/raw/sarcos_inv_test.mat'
ludwig/datasets/loaders/dataset_loader.py:82: FileNotFoundError
Check notice on line 0 in .github
github-actions / Unit Test Results
5 skipped tests found
There are 5 skipped tests, see "Raw output" for the full list of skipped tests.
Raw output
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[ames_housing.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[ames_housing.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[mercedes_benz_greener.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[mercedes_benz_greener.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[sarcos.ecd.yaml]
Check notice on line 0 in .github
github-actions / Unit Test Results
12 tests found
There are 12 tests, see "Raw output" for the full list of tests.
Raw output
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[adult_census_income.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[adult_census_income.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[ames_housing.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[ames_housing.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[mercedes_benz_greener.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[mercedes_benz_greener.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[sarcos.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[sarcos.gbm.yaml]
tests.regression_tests.model.test_old_models ‑ test_model_loaded_from_old_config_prediction_works
tests.regression_tests.model.test_old_models ‑ test_predict_deprecated_model[respiratory]
tests.regression_tests.model.test_old_models ‑ test_predict_deprecated_model[titanic]
tests.regression_tests.model.test_old_models ‑ test_predict_deprecated_model[twitter_bots]
Loading