Looping 2.5 | Repeating Blocks for List Items (#1106)

* Initial repeating blocks test schema * Schema for repeating blocks and section summary incomplete + ListCollector Question handling many add types by enum. * Repeating blocks are presented to the user and can be answered however are not posted correctly and navigation not tested. * List Collector now always enters add_block & both add_block and edit_block go to repeating_blocks if existing. * make format * edit_block does not lead to repeating_blocks. All repeating_blocks show on list collector summary. * Blocks in repeating_blocks calculated upfront. Answers from repeating_blocks shown on list collector summary underneath the relevant list item. * Reverting list_collector handler and fixing invalid use of set * Return to summary reused throughout ListActions * Adding list item completion progress to progress_store and loading this upon block GET. * Looping 2.5 - Simple Repeating Blocks in List Collector (No Item Progress Tracking) (#1100) * Schema for repeating blocks and section summary incomplete + ListCollector Question handling many add types by enum. * Repeating blocks are presented to the user and can be answered however are not posted correctly and navigation not tested. * List Collector now always enters add_block & both add_block and edit_block go to repeating_blocks if existing. * make format * edit_block does not lead to repeating_blocks. All repeating_blocks show on list collector summary. * Blocks in repeating_blocks calculated upfront. Answers from repeating_blocks shown on list collector summary underneath the relevant list item. * Reverting list_collector handler and fixing invalid use of set * Return to summary reused throughout ListActions * Fixing existing questionnaire store tests broken by adding list item progress. * Added remove list item progress to progress store and called from questionnaire store updater. * List Item Progress tracked with the section and list item id keyed progresses. * List Item Progress starts with list item add, adding the add block to the list items list of completed locations * Make response expiry date mandatory (#1104) * Reverting change to serialize() method naming + removing dev only chnage. * Bind additional contexts to flush requests (#1108) * Addressing comments to simplify progress evaluation logic + adding cached property for repeating block ids. * Schemas v3.56.0 (#1110) * Addressing comments regarding common usage of first incomplete repeating blocks to evaluate completeness. * Augmenting comment in capture_dependent_sections_for_list(self, list_name) * Bug fix to repeating block cache, list section deps reverted to dict of lists, renamed method for removing list items. * Removing get_completed_block_ids from questionnaire store updater as it is duplicate of router.is_block_complete * Preliminary impl for list item row icons - breaks tests * Feat/looping 2.5 list item progress (#1105) * Schema for repeating blocks and section summary incomplete + ListCollector Question handling many add types by enum. * Repeating blocks are presented to the user and can be answered however are not posted correctly and navigation not tested. * List Collector now always enters add_block & both add_block and edit_block go to repeating_blocks if existing. * make format * edit_block does not lead to repeating_blocks. All repeating_blocks show on list collector summary. * Blocks in repeating_blocks calculated upfront. Answers from repeating_blocks shown on list collector summary underneath the relevant list item. * Reverting list_collector handler and fixing invalid use of set * Return to summary reused throughout ListActions * Adding list item completion progress to progress_store and loading this upon block GET. * Fixing existing questionnaire store tests broken by adding list item progress. * Added remove list item progress to progress store and called from questionnaire store updater. * List Item Progress tracked with the section and list item id keyed progresses. * List Item Progress starts with list item add, adding the add block to the list items list of completed locations * Reverting change to serialize() method naming + removing dev only chnage. * Addressing comments to simplify progress evaluation logic + adding cached property for repeating block ids. * Addressing comments regarding common usage of first incomplete repeating blocks to evaluate completeness. * Augmenting comment in capture_dependent_sections_for_list(self, list_name) * Bug fix to repeating block cache, list section deps reverted to dict of lists, renamed method for removing list items. * Removing get_completed_block_ids from questionnaire store updater as it is duplicate of router.is_block_complete * Fixing tests broken by list collector icon render changes * Fixing incorrect formatting and rewording a type hint * Cleaned up evaluation of list item icon. * Schemas v3.57.0 (#1113) * Reverting accidental reversion of a file * Schemas v3.57.1 (#1115) * Schemas v3.58.0 (#1116) * Implement "progress" value source (#1044) Co-authored-by: Rhys Berrow <[email protected] * Fix handling of invalid values in form numerical inputs (#1111) * Passing only section id and bool for repeating blocks for list context and using kwargs where possible. * Adding missing list context constructor arg * Enforcing kwargs on _build_list_items_context * Merge conflict resolution with main. * Running make format * Adding list collector with repeating blocks fixture and testing the repeating blocks are mapped by id correctly when schema loads. * Completing unit testing of additions to QuestionnaireSchema * Update Chromedriver to version 113 (#1118) * update chromedriver * temporarily comment out line, to be fixed separately * ListContext unit tests * Updating list collector summary to correctly acquire related answers and avoid breaks when list collector with same name appears twice. * Enforcing kwargs on update_section_status and make format * Feat/Grand calculated summary (#1107) * initial ideas * Implementation closer to calculated summaries * better test schemas and routing evaluation * get routing working * dont need section id * make routing better, start adding tests * start fixing test schemas * Two further schema fixes * Fix remaining schema issue * Add integration test * Add functional test * Functional tests for routing and answers not on the path * Changing an answer updates GCS progress * Add schema for overlapping answers * Functionally test skip-question in grand calculated summary * Add tests for GCS routing * Fix errors caused by merge * type hints, comments, remove some duplication * fix formatting * Update translation templates * remove accidental update * Fix some type hints * Use validator branch and update test description * refactor GCS answer format * Fix lint and test errors * Address PR comments * improve context and coverage of test schemas * Fix invalid schema and add invalid routing test * Fix line length in test * test schema linting * Fix routing bug * Make schema easier to follow * Linting * partial fix to routing * Add addtional routing test case * Remove arg from new test case * Routing to next incomplete block for summaries * linting error * remove arg from router tests * PR comments * PR Comment * Amend comment and improve dependencies function * Handle merge errors * fix gcs routing within same section * Add test for GCS with progress value source * Remove backtick * update chromedriver * Revert "update chromedriver" This reverts commit a2e3698. * Fix schema labels * Revert validator branch to latest * Integration tests started * Fix dynamic answers functional test (#1121) * Repeating blocks integration test suite complete * Schemas v3.59.0 (#1130) * Make format * running and fixing mypy * Merge with main and format and lint * Fixing linting errors and pointing to relevant validator * Fixing validator tag typo * Schemas v3.60.0 (#1133) * Fixing mocker patch of renamed method * make format * Update to chromedriver v114 (#1134) * Functional test progress * Add DESNZ theme (#1131) * Schemas v3.61.0 (#1139) * Adding functional test for editing repeating block answers from section summary * Asserting completing list items in func tests and editing from section summary. * Misc type hinting * Misc type hinting * Formatting * Consolidating duplicate func test helper method and fixing typos. * Linting, formatting and merging with feat prepop. * Fixing broken tests * Linting, formatting and cleaning up merge with main * ListRepeatingBlock now extends to ListAction and does not invoke parent handle_post. * Addressing PR comments 16-6-23 * Reverting local test changes. * Reverting local test changes by formatting * Reverting local test changes by formatting * Adding additional docs and type hints to ProgressStore * Addressing PR comments. * Fixing list collector (no RP) icons * Removing person icon change. * Adding repeating blocks to the submission payload and renaming list name in test schema. * Adding test coverage for repeating block answers in submission payload. * Reformat * Adding func test and schema for a repeating blocks LC on a hub questionnaire. * Fixing test broken in prev commit. * Reformatting. * Updating naming of progress store methods follwoing convos. * Updating comment * Updating comment * Formatting * Addressing comments * Remvovign whitespace from schema * Amending naming and type hinting in list collector block summary and adding context to list collector RP schema. * Changing to unit and currency in simple RB schema. * Formatting * Removing redundant test schema * Using latest validator --------- Co-authored-by: petechd <[email protected]> Co-authored-by: Mebin Abraham <[email protected]> Co-authored-by: Guilhem <[email protected]> Co-authored-by: Katie Gardner_ONS <[email protected]> Co-authored-by: Rhys Berrow <[email protected]>
ONSdigital · Jun 26, 2023 · 4671845 · 4671845
1 parent 4a085d7
commit 4671845
Show file tree

Hide file tree

Showing 53 changed files with 2,880 additions and 317 deletions.
diff --git a/app/data_models/progress_store.py b/app/data_models/progress_store.py
@@ -1,10 +1,10 @@
 from dataclasses import astuple, dataclass
-from typing import Iterable, Iterator, MutableMapping, Optional
+from typing import Iterable, Iterator, MutableMapping
 
 from app.data_models.progress import Progress, ProgressDictType
 from app.questionnaire.location import Location
 
-SectionKeyType = tuple[str, Optional[str]]
+ProgressKeyType = tuple[str, str | None]
 
 
 @dataclass
@@ -20,34 +20,52 @@ def __iter__(self) -> Iterator[tuple[str]]:
 
 class ProgressStore:
     """
-    An object that stores and updates references to sections and blocks
+    An object that stores and updates references to sections and list items
     that have been started.
     """
 
     def __init__(
-        self, in_progress_sections: Optional[Iterable[ProgressDictType]] = None
+        self,
+        in_progress_sections_and_repeating_blocks: Iterable[ProgressDictType]
+        | None = None,
     ) -> None:
         """
-        Instantiate a ProgressStore object that tracks the status of sections and its completed blocks
+        Instantiate a ProgressStore object that tracks the progress status of Sections & Repeating Sections,
+        and their completed blocks, as well as Repeating Blocks for List Items.
+            - Standard Sections are keyed by Section ID, and a None List Item ID
+            - Repeating Sections (dynamic Sections created for List Items that have been added using a List Collector)
+                are keyed by their Section ID, and the List Item ID of the item it is the section for.
+            - Repeating Blocks for List Items are keyed by the Section ID for the Section in which their List Collector
+                appears, and the List Item ID. Repeating Blocks progress is only tracked if the List Collector
+                that created the List Item has Repeating Blocks, and progress of the Repeating Blocks for a List Item
+                indicates if all required Repeating Blocks from the List Collector have been completed for the List Item.
         Args:
-            in_progress_sections: A list of hierarchical dict containing the section status and completed blocks
+            in_progress_sections_and_repeating_blocks: A list of hierarchical dict containing the completion status
+                and completed blocks of Sections, Repeating Sections and List Items
         """
         self._is_dirty: bool = False
         self._is_routing_backwards: bool = False
-        self._progress: MutableMapping[SectionKeyType, Progress] = self._build_map(
-            in_progress_sections or []
-        )
+        self._section_and_repeating_blocks_progress: MutableMapping[
+            ProgressKeyType, Progress
+        ] = self._build_map(in_progress_sections_and_repeating_blocks or [])
 
-    def __contains__(self, section_key: SectionKeyType) -> bool:
-        return section_key in self._progress
+    def __contains__(
+        self, section_and_repeating_blocks_progress_key: ProgressKeyType
+    ) -> bool:
+        return (
+            section_and_repeating_blocks_progress_key
+            in self._section_and_repeating_blocks_progress
+        )
 
     @staticmethod
-    def _build_map(section_progress_list: Iterable[ProgressDictType]) -> MutableMapping:
+    def _build_map(
+        section_and_repeating_blocks_progress_list: Iterable[ProgressDictType],
+    ) -> MutableMapping:
         """
-        Builds the progress_store's data structure from a list of progress dictionaries.
+        Builds the ProgressStore's data structure from a list of progress dictionaries.
 
-        The `section_key` is tuple consisting of `section_id` and the `list_item_id`.
-        The `section_progress` is a mutableMapping created from the Progress object.
+        The `progress_key` is tuple consisting of `section_id` and the `list_item_id`.
+        The `progress` is a mutableMapping created from the Progress object.
 
         Example structure:
         {
@@ -62,10 +80,10 @@ def _build_map(section_progress_list: Iterable[ProgressDictType]) -> MutableMapp
 
         return {
             (
-                section_progress["section_id"],
-                section_progress.get("list_item_id"),
-            ): Progress.from_dict(section_progress)
-            for section_progress in section_progress_list
+                progress["section_id"],
+                progress.get("list_item_id"),
+            ): Progress.from_dict(progress)
+            for progress in section_and_repeating_blocks_progress_list
         }
 
     @property
@@ -76,91 +94,130 @@ def is_dirty(self) -> bool:
     def is_routing_backwards(self) -> bool:
         return self._is_routing_backwards
 
-    def is_section_complete(
-        self, section_id: str, list_item_id: Optional[str] = None
+    def is_section_or_repeating_blocks_progress_complete(
+        self, section_id: str, list_item_id: str | None = None
     ) -> bool:
-        return (section_id, list_item_id) in self.section_keys(
+        """
+        Return True if the CompletionStatus of the Section or List Item specified by the given section_id and
+         list_item_id is COMPLETED or INDIVIDUAL_RESPONSE_REQUESTED, else False.
+        """
+        return (
+            section_id,
+            list_item_id,
+        ) in self.section_and_repeating_blocks_progress_keys(
             statuses={
                 CompletionStatus.COMPLETED,
                 CompletionStatus.INDIVIDUAL_RESPONSE_REQUESTED,
             }
         )
 
-    def section_keys(
+    def section_and_repeating_blocks_progress_keys(
         self,
-        statuses: Optional[Iterable[str]] = None,
-        section_ids: Optional[Iterable[str]] = None,
-    ) -> list[SectionKeyType]:
+        statuses: Iterable[str] | None = None,
+        section_ids: Iterable[str] | None = None,
+    ) -> list[ProgressKeyType]:
+        """
+        Return the Keys of the Section and Repeating Blocks progresses stored in this ProgressStore.
+        """
         if not statuses:
             statuses = {*CompletionStatus()}
 
-        section_keys = [
+        progress_keys = [
             section_key
-            for section_key, section_progress in self._progress.items()
+            for section_key, section_progress in self._section_and_repeating_blocks_progress.items()
             if section_progress.status in statuses
         ]
 
         if section_ids is None:
-            return section_keys
+            return progress_keys
 
         return [
-            section_key
-            for section_key in section_keys
-            if any(section_id in section_key for section_id in section_ids)
+            progress_key
+            for progress_key in progress_keys
+            if any(section_id in progress_key for section_id in section_ids)
         ]
 
-    def update_section_status(
-        self, section_status: str, section_id: str, list_item_id: Optional[str] = None
+    def update_section_or_repeating_blocks_progress_completion_status(
+        self,
+        completion_status: str,
+        section_id: str,
+        list_item_id: str | None = None,
     ) -> bool:
+        """
+        Updates the completion status of the section or repeating blocks for a list item specified by the key based on the given section id and list item id.
+        """
         updated = False
         section_key = (section_id, list_item_id)
-        if section_key in self._progress:
-            if self._progress[section_key].status != section_status:
+        if section_key in self._section_and_repeating_blocks_progress:
+            if (
+                self._section_and_repeating_blocks_progress[section_key].status
+                != completion_status
+            ):
                 updated = True
-                self._progress[section_key].status = section_status
+                self._section_and_repeating_blocks_progress[
+                    section_key
+                ].status = completion_status
                 self._is_dirty = True
 
-        elif section_status == CompletionStatus.INDIVIDUAL_RESPONSE_REQUESTED:
-            self._progress[section_key] = Progress(
+        elif completion_status == CompletionStatus.INDIVIDUAL_RESPONSE_REQUESTED:
+            self._section_and_repeating_blocks_progress[section_key] = Progress(
                 section_id=section_id,
                 list_item_id=list_item_id,
                 block_ids=[],
-                status=section_status,
+                status=completion_status,
             )
             self._is_dirty = True
 
         return updated
 
-    def get_section_status(
-        self, section_id: str, list_item_id: Optional[str] = None
+    def get_section_or_repeating_blocks_progress_status(
+        self, section_id: str, list_item_id: str | None = None
     ) -> str:
-        section_key = (section_id, list_item_id)
-        if section_key in self._progress:
-            return self._progress[section_key].status
+        """
+        Return the CompletionStatus of the Section or Repeating Blocks for a list item,
+        specified by the given section_id and list_item_id.
+        Returns NOT_STARTED if the progress does not exist
+        """
+        progress_key = (section_id, list_item_id)
+        if progress_key in self._section_and_repeating_blocks_progress:
+            return self._section_and_repeating_blocks_progress[progress_key].status
 
         return CompletionStatus.NOT_STARTED
 
     def get_block_status(
         self, *, block_id: str, section_id: str, list_item_id: str | None = None
     ) -> str:
-        section_blocks = self.get_completed_block_ids(
+        """
+        Return the completion status of the block specified by the given block_id,
+        if it is part of the progress of the given Section or Repeating Blocks for list item
+        specified by the given section_id or list_item_id
+        """
+        blocks = self.get_completed_block_ids(
             section_id=section_id, list_item_id=list_item_id
         )
-        if block_id in section_blocks:
+        if block_id in blocks:
             return CompletionStatus.COMPLETED
 
         return CompletionStatus.NOT_STARTED
 
     def get_completed_block_ids(
         self, *, section_id: str, list_item_id: str | None = None
     ) -> list[str]:
-        section_key = (section_id, list_item_id)
-        if section_key in self._progress:
-            return self._progress[section_key].block_ids
+        """
+        Return the block ids recorded as part of the progress for the Section or Repeating Blocks
+        for list item specified by the given section_id and list_item_id
+        """
+        progress_key = (section_id, list_item_id)
+        if progress_key in self._section_and_repeating_blocks_progress:
+            return self._section_and_repeating_blocks_progress[progress_key].block_ids
 
         return []
 
     def add_completed_location(self, location: Location) -> None:
+        """
+        Adds the block from the given Location, to the progress specified by the
+        section id and list item id within the Location.
+        """
         section_id = location.section_id
         list_item_id = location.list_item_id
 
@@ -171,12 +228,14 @@ def add_completed_location(self, location: Location) -> None:
         if location.block_id not in completed_block_ids:
             completed_block_ids.append(location.block_id)  # type: ignore
 
-            section_key = (section_id, list_item_id)
+            progress_key = (section_id, list_item_id)
 
-            if section_key in self._progress:
-                self._progress[section_key].block_ids = completed_block_ids
+            if progress_key in self._section_and_repeating_blocks_progress:
+                self._section_and_repeating_blocks_progress[
+                    progress_key
+                ].block_ids = completed_block_ids
             else:
-                self._progress[section_key] = Progress(
+                self._section_and_repeating_blocks_progress[progress_key] = Progress(
                     section_id=section_id,
                     list_item_id=list_item_id,
                     block_ids=completed_block_ids,
@@ -186,15 +245,24 @@ def add_completed_location(self, location: Location) -> None:
             self._is_dirty = True
 
     def remove_completed_location(self, location: Location) -> bool:
-        section_key = (location.section_id, location.list_item_id)
+        """
+        Removes the block in the given Location, from the progress specified by the
+        section id and list item id within the Location if it exists in the store.
+        """
+        progress_key = (location.section_id, location.list_item_id)
         if (
-            section_key in self._progress
-            and location.block_id in self._progress[section_key].block_ids
+            progress_key in self._section_and_repeating_blocks_progress
+            and location.block_id
+            in self._section_and_repeating_blocks_progress[progress_key].block_ids
         ):
-            self._progress[section_key].block_ids.remove(location.block_id)
+            self._section_and_repeating_blocks_progress[progress_key].block_ids.remove(
+                location.block_id
+            )
 
-            if not self._progress[section_key].block_ids:
-                self._progress[section_key].status = CompletionStatus.IN_PROGRESS
+            if not self._section_and_repeating_blocks_progress[progress_key].block_ids:
+                self._section_and_repeating_blocks_progress[
+                    progress_key
+                ].status = CompletionStatus.IN_PROGRESS
 
             self._is_dirty = True
             return True
@@ -208,32 +276,32 @@ def remove_progress_for_list_item_id(self, list_item_id: str) -> None:
         *Not efficient.*
         """
 
-        section_keys_to_delete = [
+        progress_keys_to_delete = [
             (section_id, progress_list_item_id)
-            for section_id, progress_list_item_id in self._progress
+            for section_id, progress_list_item_id in self._section_and_repeating_blocks_progress
             if progress_list_item_id == list_item_id
         ]
 
-        for section_key in section_keys_to_delete:
-            del self._progress[section_key]
+        for progress_key in progress_keys_to_delete:
+            del self._section_and_repeating_blocks_progress[progress_key]
 
             self._is_dirty = True
 
     def serialize(self) -> list[Progress]:
-        return list(self._progress.values())
+        return list(self._section_and_repeating_blocks_progress.values())
 
     def remove_location_for_backwards_routing(self, location: Location) -> None:
         self.remove_completed_location(location=location)
         self._is_routing_backwards = True
 
     def clear(self) -> None:
-        self._progress.clear()
+        self._section_and_repeating_blocks_progress.clear()
         self._is_dirty = True
 
-    def started_section_keys(
-        self, section_ids: Optional[Iterable[str]] = None
-    ) -> list[SectionKeyType]:
-        return self.section_keys(
+    def started_section_and_repeating_blocks_progress_keys(
+        self, section_ids: Iterable[str] | None = None
+    ) -> list[ProgressKeyType]:
+        return self.section_and_repeating_blocks_progress_keys(
             statuses={CompletionStatus.COMPLETED, CompletionStatus.IN_PROGRESS},
             section_ids=section_ids,
         )