Skip to content

Merge branch 'ludwig-ai:master' into improve-ludwig-feature-dict

Sign in for the full log view
GitHub Actions / Unit Test Results failed Feb 5, 2024 in 0s

1 fail, 5 skipped, 6 pass in 2m 58s

  4 files   -   2    4 suites   - 2   2m 58s ⏱️ - 1m 35s
12 tests ±  0    6 ✔️ ±0    5 💤 ±  0  1 ±0 
40 runs   - 20  16 ✔️  - 8  20 💤  - 10  4  - 2 

Results for commit acfa198. ± Comparison against earlier commit 23a2da7.

Annotations

Check warning on line 0 in tests.regression_tests.benchmark.test_model_performance

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results

All 4 runs failed: test_performance[sarcos.gbm.yaml] (tests.regression_tests.benchmark.test_model_performance)

artifacts/Unit Test Results (Python 3.8 distributed)/pytest.xml
artifacts/Unit Test Results (Python 3.8 not distributed)/pytest.xml
artifacts/Unit Test Results (Python 3.9 distributed)/pytest.xml
artifacts/Unit Test Results (Python 3.9 not distributed)/pytest.xml
Raw output
config_filename = 'sarcos.gbm.yaml'
tmpdir = local('/tmp/pytest-of-runner/pytest-1/test_performance_sarcos_gbm_ya0')

    @pytest.mark.benchmark
    @pytest.mark.parametrize("config_filename", get_test_config_filenames())
    def test_performance(config_filename, tmpdir):
        update_skipped_configs_issues(config_filename)
        if config_filename in SKIPPED_CONFIG_ISSUES:
            pytest.skip(reason=SKIPPED_CONFIG_ISSUES[config_filename])
            return
    
        benchmark_directory = "/".join(__file__.split("/")[:-1])
        config_path = os.path.join(benchmark_directory, "configs", config_filename)
        expected_test_statistics_fp = os.path.join(benchmark_directory, "expected_metrics", config_filename)
        dataset_name = get_dataset_from_config_path(config_path)
    
        if not os.path.exists(expected_test_statistics_fp):
            raise FileNotFoundError(
                """No corresponding expected metrics found for benchmarking config '{config_path}'.
                Please add a new metrics YAML file '{expected_test_statistics_fp}'. Suggested content:
    
                metrics:
                  - output_feature_name: <YOUR_OUTPUT_FEATURE e.g. SalePrice>
                    metric_name: <YOUR METRIC NAME e.g. accuracy>
                    expected_value: <A FLOAT VALUE>
                    tolerance_percent: 0.15"""
            )
        expected_metrics_dict = load_yaml(expected_test_statistics_fp)
    
        benchmarking_config = {
            "experiment_name": "regression_test",
            "export": {"export_artifacts": True, "export_base_path": tmpdir},
            "experiments": [{"dataset_name": dataset_name, "config_path": config_path}],
        }
        benchmarking_artifacts = benchmark(benchmarking_config)
        experiment_artifact, err = benchmarking_artifacts[dataset_name]
        if err is not None:
>           raise err

tests/regression_tests/benchmark/test_model_performance.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ludwig/benchmarking/benchmark.py:130: in benchmark
    benchmark_one(experiment)
ludwig/benchmarking/benchmark.py:71: in benchmark_one
    dataset = load_from_module(dataset_module, model_config["output_features"][0])
ludwig/benchmarking/utils.py:46: in load_from_module
    dataset = dataset_module.load(split=False)
ludwig/datasets/loaders/dataset_loader.py:301: in load
    self._download_and_process(kaggle_username=kaggle_username, kaggle_key=kaggle_key)
ludwig/datasets/loaders/dataset_loader.py:274: in _download_and_process
    self.verify()
ludwig/datasets/loaders/dataset_loader.py:344: in verify
    digest = _sha256_digest(os.path.join(self.raw_dataset_dir, filename))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

file_path = '/home/runner/.ludwig_cache/sarcos_1.0/raw/sarcos_inv_test.mat'

    def _sha256_digest(file_path) -> str:
        """Returns the sha256 digest for the specified file."""
        hash = hashlib.sha256()
        buffer = bytearray(hash.block_size * 1024)  # Attempts to read in multiples of the hash block size (64KB).
        mv = memoryview(buffer)
>       with open(file_path, "rb", buffering=0) as f:
E       FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.ludwig_cache/sarcos_1.0/raw/sarcos_inv_test.mat'

ludwig/datasets/loaders/dataset_loader.py:82: FileNotFoundError

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results

5 skipped tests found

There are 5 skipped tests, see "Raw output" for the full list of skipped tests.
Raw output
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[ames_housing.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[ames_housing.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[mercedes_benz_greener.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[mercedes_benz_greener.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[sarcos.ecd.yaml]

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results

12 tests found

There are 12 tests, see "Raw output" for the full list of tests.
Raw output
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[adult_census_income.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[adult_census_income.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[ames_housing.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[ames_housing.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[mercedes_benz_greener.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[mercedes_benz_greener.gbm.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[sarcos.ecd.yaml]
tests.regression_tests.benchmark.test_model_performance ‑ test_performance[sarcos.gbm.yaml]
tests.regression_tests.model.test_old_models ‑ test_model_loaded_from_old_config_prediction_works
tests.regression_tests.model.test_old_models ‑ test_predict_deprecated_model[respiratory]
tests.regression_tests.model.test_old_models ‑ test_predict_deprecated_model[titanic]
tests.regression_tests.model.test_old_models ‑ test_predict_deprecated_model[twitter_bots]