Fix amount <> interval clash #163

nilesh-skit · 2022-10-20T19:55:40Z

Sometimes (randomly?) this test on application collections combined is failing and that's resulting in a failed CI/CD pipeline.

This patch currently bypasses the dim=amount-of-money and type=interval coupling. (amount-of-money expects type to be value, not interval).

nilesh-skit · 2022-10-20T19:56:27Z

Reference 1: here

nilesh-skit · 2022-10-20T20:02:17Z

Log of make test on SLU succeeding and then make test failing, without any change to the codebase.

(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$ make test

========================================== test session starts ==========================================
platform linux -- Python 3.9.13, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/nilesh/slu-test-acc-v1/application-collections-entity-improvements, configfile: pytest.ini
plugins: cov-2.12.1, anyio-3.6.1
collected 8 items                                                                                       

tests/test_controller/test_predict_api.py ........                                                [100%]

---------- coverage: platform linux, python 3.9.13-final-0 -----------
Name                                   Stmts   Miss  Cover
----------------------------------------------------------
slu/dev/cli.py                            74     74     0%
slu/dev/dev.py                            69     69     0%
slu/dev/dir_setup.py                      18     18     0%
slu/dev/io/mp/__init__.py                 16     16     0%
slu/dev/io/reader/csv.py                  35     35     0%
slu/dev/io/reader/pickle.py               15     15     0%
slu/dev/io/reader/sqlite.py               17     17     0%
slu/dev/prepare_entity_tags.py           101    101     0%
slu/dev/repl.py                           50     50     0%
slu/dev/test.py                          105     77    27%
slu/dev/train.py                         114    114     0%
slu/src/api/__init__.py                    3      3     0%
slu/src/api/endpoints.py                  78     78     0%
slu/src/api/models.py                      9      9     0%
slu/src/controller/custom_plugins.py     597    161    73%
slu/src/controller/prediction.py          73     24    67%
slu/src/controller/processors.py          53      5    91%
slu/utils/calibration.py                  16     16     0%
slu/utils/config.py                       97     21    78%
slu/utils/decorators.py                   22     22     0%
slu/utils/error.py                         4      4     0%
slu/utils/error_response.py               19     19     0%
slu/utils/ignore.py                        3      3     0%
slu/utils/make_test_cases.py              23     23     0%
slu/utils/s3.py                           20     20     0%
slu/utils/sentry.py                        4      4     0%
slu/utils/slack.py                        30     30     0%
----------------------------------------------------------
TOTAL                                   1819   1028    43%

10 files skipped due to complete coverage.
Coverage HTML written to dir htmlcov


========================================== 8 passed in 53.35s ===========================================
The tests pass!
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$ 
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$ 
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$ 
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$ make test
========================================== test session starts ==========================================
platform linux -- Python 3.9.13, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/nilesh/slu-test-acc-v1/application-collections-entity-improvements, configfile: pytest.ini
plugins: cov-2.12.1, anyio-3.6.1
collected 8 items                                                                                       

tests/test_controller/test_predict_api.py .......F                                                [100%]

=============================================== FAILURES ================================================
___________________________________ test_classifier_on_training_data ____________________________________

cls = <class 'dialogy.types.entity.deserialize.EntityDeserializer'>
duckling_entity_dict = {'body': '3-4', 'dim': 'amount-of-money', 'end': 23, 'latent': False, ...}
alternative_index = 0, reference_time = 1666295139132, timezone = 'Asia/Kolkata'
duration_cast_operator = None, constraints = None

    @classmethod
    def deserialize_duckling(
        cls,
        duckling_entity_dict: Dict[str, Any],
        alternative_index: int,
        reference_time: Optional[int] = None,
        timezone: str = "UTC",
        duration_cast_operator: Optional[str] = None,
        constraints: Optional[Dict[str, Any]] = None,
    ) -> Optional[BaseEntity]:
        cls.validate(duckling_entity_dict)
        entity_class_name = cls.get_entity_class_str(duckling_entity_dict)
        EntityClass: BaseEntity = cls.entitiy_classes[entity_class_name]
    
        try:
>           entity = EntityClass.from_duckling(
                duckling_entity_dict,
                alternative_index,
                constraints=constraints,
                duration_cast_operator=duration_cast_operator,
                timezone=timezone,
                reference_time=reference_time,
            )

../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/types/entity/deserialize/__init__.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'dialogy.types.entity.amount_of_money.CurrencyEntity'>
d = {'body': '3-4', 'dim': 'amount-of-money', 'end': 23, 'latent': False, ...}, alternative_index = 0
kwargs = {'constraints': None, 'duration_cast_operator': None, 'reference_time': 1666295139132, 'timezone': 'Asia/Kolkata'}

    @classmethod
    def from_duckling(
        cls, d: Dict[str, Any], alternative_index: int, **kwargs: Any
    ) -> CurrencyEntity:
>       value = d[const.VALUE][const.VALUE]
E       KeyError: 'value'

../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/types/entity/amount_of_money/__init__.py:61: KeyError

The above exception was the direct cause of the following exception:

slu_api = <function get_predictions.<locals>.predict at 0x7f8bd36428b0>

    def test_classifier_on_training_data(slu_api):
        """
        Evaluate the workflow with all the embedded plugins.
    
        Plugins can be evaluated individually for fine-tuning but since there are interactions
        between them, we need to evaluate them all together. This helps in cases where these interactions
        are a cause of a model's poor performance.
    
        This method doesn't mutate the given test dataset, instead we produce results with the same `id_`
        so that they can be joined and studied together.
        """
        project_config_map = YAMLLocalConfig().generate()
        config: Config = list(project_config_map.values()).pop()
        dataset = config.get_dataset(const.CLASSIFICATION, f"{const.TRAIN}.csv")
        test_df = pd.read_csv(dataset).sample(n=100)
    
        predictions = []
        config.tasks.classification.threshold = 0
    
        for _, row in tqdm(test_df.iterrows(), total=test_df.shape[0]):
>           output = slu_api(
                **{
                    const.ALTERNATIVES: json.loads(row[const.ALTERNATIVES]),
                    const.CONTEXT: {},
                    const.LANG: "en",
                    "ignore_test_case": True,
                }
            )

tests/test_controller/test_predict_api.py:146: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
slu/src/controller/prediction.py:110: in predict
    _, output = workflow.run(input_)
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/workflow/workflow.py:285: in run
    return self.execute().flush()
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/workflow/workflow.py:260: in execute
    plugin(self)
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/base/plugin/__init__.py:292: in __call__
    value = self.utility(workflow.input, workflow.output)
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/plugins/text/duckling_plugin/__init__.py:992: in utility
    return self.parse(
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/plugins/text/duckling_plugin/__init__.py:970: in parse
    entities = self.apply_entity_classes(
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/plugins/text/duckling_plugin/__init__.py:907: in apply_entity_classes
    self._reshape(
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/plugins/text/duckling_plugin/__init__.py:788: in _reshape
    entity = EntityDeserializer.deserialize_duckling(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'dialogy.types.entity.deserialize.EntityDeserializer'>
duckling_entity_dict = {'body': '3-4', 'dim': 'amount-of-money', 'end': 23, 'latent': False, ...}
alternative_index = 0, reference_time = 1666295139132, timezone = 'Asia/Kolkata'
duration_cast_operator = None, constraints = None

    @classmethod
    def deserialize_duckling(
        cls,
        duckling_entity_dict: Dict[str, Any],
        alternative_index: int,
        reference_time: Optional[int] = None,
        timezone: str = "UTC",
        duration_cast_operator: Optional[str] = None,
        constraints: Optional[Dict[str, Any]] = None,
    ) -> Optional[BaseEntity]:
        cls.validate(duckling_entity_dict)
        entity_class_name = cls.get_entity_class_str(duckling_entity_dict)
        EntityClass: BaseEntity = cls.entitiy_classes[entity_class_name]
    
        try:
            entity = EntityClass.from_duckling(
                duckling_entity_dict,
                alternative_index,
                constraints=constraints,
                duration_cast_operator=duration_cast_operator,
                timezone=timezone,
                reference_time=reference_time,
            )
        except KeyError as e:
>           raise ValueError(
                f"Failed to deserialize {EntityClass} "
                f"from duckling response: {duckling_entity_dict}. Exception: {e}"
            ) from e
E           ValueError: Failed to deserialize <class 'dialogy.types.entity.amount_of_money.CurrencyEntity'> from duckling response: {'body': '3-4', 'start': 20, 'value': {'to': {'value': 4, 'unit': 'unknown'}, 'from': {'value': 3, 'unit': 'unknown'}, 'type': 'interval'}, 'end': 23, 'dim': 'amount-of-money', 'latent': False}. Exception: 'value'

../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/types/entity/deserialize/__init__.py:75: ValueError
----------------------------------------- Captured stdout call ------------------------------------------
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = reason_merchant_issue
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = reason_service_issue_other
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = reason_other
PaymentHesitationReasonFiller: rlabel = reason_service_issue_complaints_and_customer_care
PaymentHesitationReasonFiller: rlabel = reason_another_company
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = reason_no_reminder
PaymentHesitationReasonFiller: rlabel = reason_service_issue_other
PaymentHesitationReasonFiller: rlabel = reason_loan_not_taken
PaymentHesitationReasonFiller: rlabel = reason_other
PaymentHesitationReasonFiller: rlabel = reason_service_issue_no_statement
PaymentHesitationReasonFiller: rlabel = reason_service_issue_complaints_and_customer_care
PaymentHesitationReasonFiller: rlabel = reason_other
PaymentHesitationReasonFiller: rlabel = reason_i_was_not_available
PaymentHesitationReasonFiller: rlabel = reason_service_issue_no_statement
----------------------------------------- Captured stderr call ------------------------------------------
/home/nilesh/slu-test-acc-v1/application-collections-entity-improvements/tests/test_controller/test_predict_api.py:140: DtypeWarning: Columns (3,6,13,14,15,17,18,19,20,21,22,23,24,25,28,29,30,32,34,36,45,48,49) have mixed types. Specify dtype option on import or set low_memory=False.
  test_df = pd.read_csv(dataset).sample(n=100)
 79%|███████▉  | 79/100 [00:28<00:07,  2.74it/s]

---------- coverage: platform linux, python 3.9.13-final-0 -----------
Name                                   Stmts   Miss  Cover
----------------------------------------------------------
slu/dev/cli.py                            74     74     0%
slu/dev/dev.py                            69     69     0%
slu/dev/dir_setup.py                      18     18     0%
slu/dev/io/mp/__init__.py                 16     16     0%
slu/dev/io/reader/csv.py                  35     35     0%
slu/dev/io/reader/pickle.py               15     15     0%
slu/dev/io/reader/sqlite.py               17     17     0%
slu/dev/prepare_entity_tags.py           101    101     0%
slu/dev/repl.py                           50     50     0%
slu/dev/test.py                          105     84    20%
slu/dev/train.py                         114    114     0%
slu/src/api/__init__.py                    3      3     0%
slu/src/api/endpoints.py                  78     78     0%
slu/src/api/models.py                      9      9     0%
slu/src/controller/custom_plugins.py     597    165    72%
slu/src/controller/prediction.py          73     23    68%
slu/src/controller/processors.py          53      5    91%
slu/utils/calibration.py                  16     16     0%
slu/utils/config.py                       97     21    78%
slu/utils/decorators.py                   22     22     0%
slu/utils/error.py                         4      4     0%
slu/utils/error_response.py               19     19     0%
slu/utils/ignore.py                        3      3     0%
slu/utils/make_test_cases.py              23     23     0%
slu/utils/s3.py                           20     20     0%
slu/utils/sentry.py                        4      4     0%
slu/utils/slack.py                        30     30     0%
----------------------------------------------------------
TOTAL                                   1819   1038    43%

10 files skipped due to complete coverage.
Coverage HTML written to dir htmlcov

======================================== short test summary info ========================================
FAILED tests/test_controller/test_predict_api.py::test_classifier_on_training_data - ValueError: Faile...
===================================== 1 failed, 7 passed in 46.13s ======================================
make: *** [Makefile:15: test] Error 1
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$

nilesh-skit · 2022-10-20T20:08:16Z

Log of the same pipeline:

failing: here
succeeding: here

nilesh-skit added 2 commits October 21, 2022 00:54

handle interval+amount separately

faf2ce5

add comment

6b9328c

nilesh-skit requested review from dakshvar22 and ltbringer October 20, 2022 20:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix amount <> interval clash #163

Fix amount <> interval clash #163

nilesh-skit commented Oct 20, 2022

nilesh-skit commented Oct 20, 2022

nilesh-skit commented Oct 20, 2022

nilesh-skit commented Oct 20, 2022

Fix amount <> interval clash #163

Are you sure you want to change the base?

Fix amount <> interval clash #163

Conversation

nilesh-skit commented Oct 20, 2022

nilesh-skit commented Oct 20, 2022

nilesh-skit commented Oct 20, 2022

nilesh-skit commented Oct 20, 2022