Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure csr matrices are in "canonical format" before impact calculation #893

Merged
merged 15 commits into from
Jul 17, 2024

Update unit tests for matrix pruning and checks

71ce00e
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
Merged

Ensure csr matrices are in "canonical format" before impact calculation #893

Update unit tests for matrix pruning and checks
71ce00e
Select commit
Loading
Failed to load commit list.
GitHub Actions / Petals / Unit Test Results (3.10) failed Jun 21, 2024 in 0s

5 fail, 187 pass in 4m 43s

192 tests  ±0   187 ✅ ±0   4m 43s ⏱️ +2s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     5 ❌ ±0 

Results for commit 71ce00e. ± Comparison against earlier commit c58b48d.

Annotations

Check warning on line 0 in climada_petals.hazard.test.test_low_flow.TestLowFlowDummyData

See this annotation in the file changed.

@github-actions github-actions / Petals / Unit Test Results (3.10)

test_events_from_clusters_default (climada_petals.hazard.test.test_low_flow.TestLowFlowDummyData) failed

climada_petals/tests_xml/tests.xml [took 0s]
Raw output
AttributeError: 'lil_matrix' object has no attribute 'check_format'
self = <climada_petals.hazard.test.test_low_flow.TestLowFlowDummyData testMethod=test_events_from_clusters_default>

    def test_events_from_clusters_default(self):
        """Test events_from_clusters: creation of events and computation of intensity based on clusters,
        requires: identify_clusters, Centroids, also tests correct intensity sum"""
        haz = LowFlow()
        haz.lowflow_df = init_test_data_clustering()
        haz.identify_clusters(clus_thresh_xy=1.5, clus_thresh_t=1, min_samples=1)
        centroids = init_test_centroids(haz.lowflow_df)
>       haz.events_from_clusters(centroids)

climada_petals/hazard/test/test_low_flow.py:121: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
climada_petals/hazard/low_flow.py:397: in events_from_clusters
    self.intensity = self._intensity_loop(uniq_ev, centroids.coord, res_centr, num_centr)
../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/hazard/base.py:221: in intensity
    u_check.prune_csr_matrix(self._intensity)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

matrix = <4x36 sparse matrix of type '<class 'numpy.float64'>'
	with 11 stored elements in List of Lists format>

    def prune_csr_matrix(matrix: sparse.csr_matrix):
        """Ensure that the matrix is in the "canonical format".
    
        Depending on how the matrix was instantiated or modified, it might be in a
        "non-canonical" state. This only relates to its internal storage. In this state,
        multiple values might be stored for a single "apparent" value in the matrix.
        Also, the matrix might store zeros explicitly, which could be removed.
        Calling this function makes sure that the matrix is in the "canonical state", and
        brings it into this state, if possible.
    
        See Also
        --------
        `csr_matrix.has_canonical_format
        <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.has_canonical_format.html#scipy.sparse.csr_matrix.has_canonical_format>`_
    
        Parameters
        ----------
        matrix : csr_matrix
            The matrix to check. It will be modified *inplace*. No apparent matrix values
            will change.
    
        Raises
        ------
        ValueError
            If
            `csr_matrix.check_format
            <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.check_format.html#scipy.sparse.csr_matrix.check_format>`_
            fails
        """
>       matrix.check_format()
E       AttributeError: 'lil_matrix' object has no attribute 'check_format'

../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/util/checker.py:196: AttributeError

Check warning on line 0 in climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF

See this annotation in the file changed.

@github-actions github-actions / Petals / Unit Test Results (3.10)

test_combine_nc (climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF) failed

climada_petals/tests_xml/tests.xml [took 4s]
Raw output
AttributeError: 'lil_matrix' object has no attribute 'check_format'
self = <climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF testMethod=test_combine_nc>

    def test_combine_nc(self):
        """Test combining two chunked data files (2001-2003 combined with 2004-2005)"""
>       haz = LowFlow.from_netcdf(input_dir=INPUT_DIR, percentile=2.5,
                         yearrange=(2001, 2005), yearrange_ref=(2001, 2005),
                         gh_model='h08', cl_model='gfdl-esm2m',
                         scenario='historical', scenario_ref='historical', soc='histsoc',
                         soc_ref='histsoc', fn_str_var=FN_STR_DEMO, keep_dis_data=True,
                         yearchunks=['2001_2003', '2004_2005'])

climada_petals/hazard/test/test_low_flow.py:197: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
climada_petals/hazard/low_flow.py:275: in from_netcdf
    haz.set_intensity_from_clusters(centroids, min_intensity, min_number_cells,
climada_petals/hazard/low_flow.py:286: in set_intensity_from_clusters
    self.events_from_clusters(centroids)
climada_petals/hazard/low_flow.py:397: in events_from_clusters
    self.intensity = self._intensity_loop(uniq_ev, centroids.coord, res_centr, num_centr)
../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/hazard/base.py:221: in intensity
    u_check.prune_csr_matrix(self._intensity)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

matrix = <66x513 sparse matrix of type '<class 'numpy.float64'>'
	with 793 stored elements in List of Lists format>

    def prune_csr_matrix(matrix: sparse.csr_matrix):
        """Ensure that the matrix is in the "canonical format".
    
        Depending on how the matrix was instantiated or modified, it might be in a
        "non-canonical" state. This only relates to its internal storage. In this state,
        multiple values might be stored for a single "apparent" value in the matrix.
        Also, the matrix might store zeros explicitly, which could be removed.
        Calling this function makes sure that the matrix is in the "canonical state", and
        brings it into this state, if possible.
    
        See Also
        --------
        `csr_matrix.has_canonical_format
        <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.has_canonical_format.html#scipy.sparse.csr_matrix.has_canonical_format>`_
    
        Parameters
        ----------
        matrix : csr_matrix
            The matrix to check. It will be modified *inplace*. No apparent matrix values
            will change.
    
        Raises
        ------
        ValueError
            If
            `csr_matrix.check_format
            <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.check_format.html#scipy.sparse.csr_matrix.check_format>`_
            fails
        """
>       matrix.check_format()
E       AttributeError: 'lil_matrix' object has no attribute 'check_format'

../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/util/checker.py:196: AttributeError

Check warning on line 0 in climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF

See this annotation in the file changed.

@github-actions github-actions / Petals / Unit Test Results (3.10)

test_filter_events (climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF) failed

climada_petals/tests_xml/tests.xml [took 3s]
Raw output
AttributeError: 'lil_matrix' object has no attribute 'check_format'
self = <climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF testMethod=test_filter_events>

    def test_filter_events(self):
        """test if the right events are being filtered out"""
>       haz = LowFlow.from_netcdf(input_dir=INPUT_DIR, percentile=2.5, min_intensity=10,
                        min_number_cells=10, min_days_per_month=10,
                        yearrange=(2001, 2003), yearrange_ref=(2001, 2003),
                        gh_model='h08', cl_model='gfdl-esm2m',
                        scenario='historical', scenario_ref='historical', soc='histsoc',
                        soc_ref='histsoc', fn_str_var=FN_STR_DEMO, keep_dis_data=True,
                        yearchunks=['2001_2003'])

climada_petals/hazard/test/test_low_flow.py:232: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
climada_petals/hazard/low_flow.py:275: in from_netcdf
    haz.set_intensity_from_clusters(centroids, min_intensity, min_number_cells,
climada_petals/hazard/low_flow.py:286: in set_intensity_from_clusters
    self.events_from_clusters(centroids)
climada_petals/hazard/low_flow.py:397: in events_from_clusters
    self.intensity = self._intensity_loop(uniq_ev, centroids.coord, res_centr, num_centr)
../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/hazard/base.py:221: in intensity
    u_check.prune_csr_matrix(self._intensity)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

matrix = <23x513 sparse matrix of type '<class 'numpy.float64'>'
	with 381 stored elements in List of Lists format>

    def prune_csr_matrix(matrix: sparse.csr_matrix):
        """Ensure that the matrix is in the "canonical format".
    
        Depending on how the matrix was instantiated or modified, it might be in a
        "non-canonical" state. This only relates to its internal storage. In this state,
        multiple values might be stored for a single "apparent" value in the matrix.
        Also, the matrix might store zeros explicitly, which could be removed.
        Calling this function makes sure that the matrix is in the "canonical state", and
        brings it into this state, if possible.
    
        See Also
        --------
        `csr_matrix.has_canonical_format
        <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.has_canonical_format.html#scipy.sparse.csr_matrix.has_canonical_format>`_
    
        Parameters
        ----------
        matrix : csr_matrix
            The matrix to check. It will be modified *inplace*. No apparent matrix values
            will change.
    
        Raises
        ------
        ValueError
            If
            `csr_matrix.check_format
            <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.check_format.html#scipy.sparse.csr_matrix.check_format>`_
            fails
        """
>       matrix.check_format()
E       AttributeError: 'lil_matrix' object has no attribute 'check_format'

../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/util/checker.py:196: AttributeError

Check warning on line 0 in climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF

See this annotation in the file changed.

@github-actions github-actions / Petals / Unit Test Results (3.10)

test_load_FR_all (climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF) failed

climada_petals/tests_xml/tests.xml [took 3s]
Raw output
AttributeError: 'lil_matrix' object has no attribute 'check_format'
self = <climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF testMethod=test_load_FR_all>

    def test_load_FR_all(self):
        """Test defining low flow hazard from demo file (France 2001-2003)
        and keep monthly data"""
    
        # init test hazard instance from trimmed ISIMIP output netcdf file
>       haz = LowFlow.from_netcdf(input_dir=INPUT_DIR, percentile=2.5,
                    yearrange=(2001, 2003), yearrange_ref=(2001, 2003),
                    gh_model='h08', cl_model='gfdl-esm2m',
                    scenario='historical', scenario_ref='historical', soc='histsoc',
                    soc_ref='histsoc', fn_str_var=FN_STR_DEMO, keep_dis_data=True,
                    yearchunks=['2001_2003'])

climada_petals/hazard/test/test_low_flow.py:164: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
climada_petals/hazard/low_flow.py:275: in from_netcdf
    haz.set_intensity_from_clusters(centroids, min_intensity, min_number_cells,
climada_petals/hazard/low_flow.py:286: in set_intensity_from_clusters
    self.events_from_clusters(centroids)
climada_petals/hazard/low_flow.py:397: in events_from_clusters
    self.intensity = self._intensity_loop(uniq_ev, centroids.coord, res_centr, num_centr)
../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/hazard/base.py:221: in intensity
    u_check.prune_csr_matrix(self._intensity)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

matrix = <43x513 sparse matrix of type '<class 'numpy.float64'>'
	with 564 stored elements in List of Lists format>

    def prune_csr_matrix(matrix: sparse.csr_matrix):
        """Ensure that the matrix is in the "canonical format".
    
        Depending on how the matrix was instantiated or modified, it might be in a
        "non-canonical" state. This only relates to its internal storage. In this state,
        multiple values might be stored for a single "apparent" value in the matrix.
        Also, the matrix might store zeros explicitly, which could be removed.
        Calling this function makes sure that the matrix is in the "canonical state", and
        brings it into this state, if possible.
    
        See Also
        --------
        `csr_matrix.has_canonical_format
        <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.has_canonical_format.html#scipy.sparse.csr_matrix.has_canonical_format>`_
    
        Parameters
        ----------
        matrix : csr_matrix
            The matrix to check. It will be modified *inplace*. No apparent matrix values
            will change.
    
        Raises
        ------
        ValueError
            If
            `csr_matrix.check_format
            <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.check_format.html#scipy.sparse.csr_matrix.check_format>`_
            fails
        """
>       matrix.check_format()
E       AttributeError: 'lil_matrix' object has no attribute 'check_format'

../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/util/checker.py:196: AttributeError

Check warning on line 0 in climada_petals.hazard.test.test_wildfire.TestMethodsFirms

See this annotation in the file changed.

@github-actions github-actions / Petals / Unit Test Results (3.10)

test_calc_bright_pass (climada_petals.hazard.test.test_wildfire.TestMethodsFirms) failed

climada_petals/tests_xml/tests.xml [took 3s]
Raw output
AttributeError: 'lil_matrix' object has no attribute 'check_format'
self = <climada_petals.hazard.test.test_wildfire.TestMethodsFirms testMethod=test_calc_bright_pass>

        def test_calc_bright_pass(self):
            """ Test _calc_brightness """
    #        from pathos.pools import ProcessPool as Pool
    #        pool = Pool()
            wf = WildFire()
            firms = wf._clean_firms_df(TEST_FIRMS)
            firms['datenum'].values[100] = 7000
            firms = wf._firms_cons_days(firms)
            firms = wf._firms_clustering(firms, DEF_CENTROIDS[1]/2/15)
            wf._firms_fire(firms)
            firms.latitude[8169] = firms.loc[16]['latitude']
            firms.longitude[8169] = firms.loc[16]['longitude']
>           wf._calc_brightness(firms, DEF_CENTROIDS[0], DEF_CENTROIDS[1])

climada_petals/hazard/test/test_wildfire.py:226: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
climada_petals/hazard/wildfire.py:902: in _calc_brightness
    self.intensity = sparse.lil_matrix(np.zeros((num_ev, num_centr)))
../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/hazard/base.py:221: in intensity
    u_check.prune_csr_matrix(self._intensity)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

matrix = <7x19454 sparse matrix of type '<class 'numpy.float64'>'
	with 0 stored elements in List of Lists format>

    def prune_csr_matrix(matrix: sparse.csr_matrix):
        """Ensure that the matrix is in the "canonical format".
    
        Depending on how the matrix was instantiated or modified, it might be in a
        "non-canonical" state. This only relates to its internal storage. In this state,
        multiple values might be stored for a single "apparent" value in the matrix.
        Also, the matrix might store zeros explicitly, which could be removed.
        Calling this function makes sure that the matrix is in the "canonical state", and
        brings it into this state, if possible.
    
        See Also
        --------
        `csr_matrix.has_canonical_format
        <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.has_canonical_format.html#scipy.sparse.csr_matrix.has_canonical_format>`_
    
        Parameters
        ----------
        matrix : csr_matrix
            The matrix to check. It will be modified *inplace*. No apparent matrix values
            will change.
    
        Raises
        ------
        ValueError
            If
            `csr_matrix.check_format
            <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.check_format.html#scipy.sparse.csr_matrix.check_format>`_
            fails
        """
>       matrix.check_format()
E       AttributeError: 'lil_matrix' object has no attribute 'check_format'

../../../../micromamba/envs/climada_env_3.10/lib/python3.10/site-packages/climada/util/checker.py:196: AttributeError