Implement episode and season search templates (#3732)

* Added the episode and season search templates to the tv_shows table. Implemented basic parsing of the templates for searching providers. * Mako changes for start using Vue for editShow.mako. * Small fixes to existing Vue.js code in config. * Moved sceneExceptions table from cache to main.db. * Fixed tvmaze debug log. ! Should be cherry picked to develop. * Fixed old bug where trakt is not using indexer to compare. * Fix bug in artwork, passing indexer_id while should pass object. ! Could be cherry picked. * Refactored sceneExceptions, returning now a named tuple in stead of tuple. This is needed because we add episode_search_template and season_search_template to the sceneExceptions. * Refactored get_scene_exceptions_by_name to use the exception_cache, and not hit the db. Made sure it returned a set of TitleExceptions (named tuples). * Fix typo * Changed most of the editShow.mako to use vue for form handling. * Added some show config items to apiv2. * Added indexer config items. * Updated theme slim src. * Added apiv2 routes for patching ignored and required words. * Added new vue.js component to editShow, for configuring lists of strings. * Added pencil_add.png img. * Disabled some css. Need to clean this up. * Moved anidb to helper. * Added anidb exception handling. * Added dogpile for anidb get release groups. * editShow.mako: Replacing get seriesObj from mako python to use apiv2. * Fixed quotes and removed json.dumps. * Styled the anidbReleaseGroupUi.mako component. Connected anidbReleaseGroupUi component properly to editShow.mako. * Fixed the propagation of the list of strings for the requiredWords and ignoredWords ui select list component. * cleaned up vue and removed python imports * Remove old components. * Add convert method to main.mako, so it can be used in other templates using namespaces. * Added new vue component name-pattern. * No need to create a new variable here. * Make use of the namespace for the convert method. * Remove the fillExamples methods, as i'm in the process of changing that to native vue. * config_Postprocessing.mako: Started implementing vue for the episode naming tab. * Replaced all the different custom naming types (sports, air by date, anime) with one component. * Add axios clients for apiv1 and webRout rest calls. * Move convert method in mako to namespace. * Get exceptions from show. * Api changes maps. * Introduced a new component for name patterns. * Remove token from api client. Will get overwritten by commit later. * Added onChangePattern method, to process name-pattern component updates. * Added postprocessing items to the apiv2 config. * Updated apiv2 config attributes for postprocessing. * Added MultiEp style select. * Improved name-pattern component for use with anime. * Made sure values are passed as properties. * Fixed emit object. * Also update animeNamingType (add absolute number) support to naming-pattern component. * Update muli-ep exaples when changing multi-ep style. * Fix db migration bugs after conflicts. * Added search_templates.py * Initialize an object on Series.py * Create search_template strings * Get search_template strings from main.db * Fix duplicate creation of strings * Separated read_from_db from generate method. This is needed, to create default search strings, but also to prevent overwriting the "enabled" property. * Add search_templates to apiv2 series result * Update table search_templates * Updated the fields * Fix regression in name_parser/parser.py * Move db connection to constructor. * Added save method * Added update method * Add config.templates and config.searchTemplates to apiv2. For saving new values. * Added @properties for the templates. Used for performing actions on saving through apiv2. * Added first UI's for creating search templates. * Add property aliases_to_json because of the new structure of exceptions. * Fix aliases in edit-show.vue because of new structure. * Yarn dev and eslint --fix * Fix enabling/disabling and saving default search templates. Fix saving scene exceptions * Add flag to templates for specifying season searches. * Add new tab to editShow "search templates" * Add new component for adding search templates. * Create new component for managing scene exceptions. * Component has support for adding season scene exceptions. * Remove unused import. * Add seasoned scene exceptions. * Fixed a number of small bugs, related to the renaming of seriesName (and series_name) to title. * Fixed issue for deleting scene exceptions. * Fix adding/removing scene exceptions. * removeEmpty is not used anymore. * Added newnlines between class. * Remove getAllSceneExceptions, as this is replaced by aliases. * Fixed display of Scene names in the show-header.vue * Fixed display of season scene names in display-show.vue (table headers) * Fixed bug in NameParser. * Updated api-description. * Updated apiv2 * Removed allSceneExceptions, as this is replaced by aliases. * Updated api-description.yml * Updated show-detailed.json * updated fixture with alias examples. * Added newline. * Fix stylelint * Fix test use TitleException. * yarn dev * Fix test_parse_anime tests. * Fix api-description * Fix flake warnings * Utilize the field "custom" to specify if a scene exception is a custom added exception. * Do not allow to remove non-custom added exceptions. * Small visual enhancement. * Add custom field for guessit tests. * Fix flake warnings * Fix typo * Flake indentation? * Let's try this * Show Medusa icon for exceptions that are not custom. * Show tooltip with more info. * Vue lint and css-lint fixes. * Flake, we keep trying. * Try to fix api tests. * hanging indent * Finally fixing this sh$t. * Hail to vis code with py (and decent flake8) support! * Change tooltip message * yarn dev * Add db constraint: Make sure template + season is always unique. * Rename show_name to title. * Add template_id. * Re-generate search templates on modifactions to scene_exceptions. * Prevent adding scene_exceptions that already exist. * This was previously a combination of scene_exception + season. But that doesn't make sense. * Add id to search_templates. * Do not pass search_templates, as it's already included in show.config... * Fix lint errors * hide/show templates tab. Bundle runtime and vendors * Fix merge conflicts and linting * Fix adding custom search templates. * Fixed merge conflicts * Make sure added search templates are not cleaned after restart * Decect duplicate template combinations * Add first version of basic searching using the search templates * Don't search irrelevant scene seasons * Improve default template creation for season scene exceptions * Remove debounce, not needed. * Fix debounced error * Place back debounce * yarn dev * Remove unique constraint * Use %0XE * No need to re-recreate all default templates when requesting from api. * Added info for displaying info on show / season exception. * Only get search_templates when detailed. * Add test-guessit component * Fixed show merging errors * Asume season 1, when no season number parsed * Add parsing of special episodes by episode title. * Fixed issue with scene_numbering for special episodes. * Change to staticmethod * Change check to startsWith * Rename remove -> remove_custom. * Add animation to red cross * update snapshot * Fix linting issues * Fix flake * this image is not used * Update changelog Co-authored-by: Alexis Tyler <[email protected]>
pymedusa · Jan 25, 2022 · 3b6bbbe · 3b6bbbe
1 parent ee25124
commit 3b6bbbe
Show file tree

Hide file tree

Showing 36 changed files with 7,191 additions and 3,310 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,7 @@
 - Add support for banner and background images to indexer tvmaze ([10234](https://github.com/pymedusa/Medusa/pull/10234))
 - Add option for using ffprobe to validate postprocessed media ([10132](https://github.com/pymedusa/Medusa/pull/10132))
 - Add change indexer page to change the current indexer for shows in bulk ([9862](https://github.com/pymedusa/Medusa/pull/9862))
+- Add search templates feature. ([3732](https://github.com/pymedusa/Medusa/pull/3732))
 
 #### Improvements
 - Add column sorting for the add new show page search results ([10217](https://github.com/pymedusa/Medusa/pull/10217))

diff --git a/medusa/databases/main_db.py b/medusa/databases/main_db.py
@@ -995,3 +995,37 @@ def execute(self):
             self.addColumn('history', 'part_of_batch', 'INTEGER')
 
         self.inc_minor_version()
+
+
+class AddSearchTemplates(AddHistoryFDHFields):
+    """Create a new table search_templates in main.db."""
+
+    def test(self):
+        """
+        Test if the version is at least 44.19
+        """
+        return self.connection.version >= (44, 19)
+
+    def execute(self):
+        utils.backup_database(self.connection.path, self.connection.version)
+
+        log.info(u'Creating a new table search_templates in the main.db database.')
+
+        self.connection.action(
+            """CREATE TABLE "search_templates" (
+            `search_template_id` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
+            `template`	TEXT,
+            `title`     TEXT,
+            `indexer`	INTEGER,
+            `series_id`	INTEGER,
+            `season`	INTEGER,
+            `enabled`	INTEGER DEFAULT 1,
+            `default`	INTEGER DEFAULT 1,
+            `season_search` INTEGER DEFAULT 0);"""
+        )
+
+        log.info(u'Adding new templates field in the tv_shows table')
+        if not self.hasColumn('tv_shows', 'templates'):
+            self.addColumn('tv_shows', 'templates', 'NUMERIC', 0)
+
+        self.inc_minor_version()
diff --git a/medusa/name_cache.py b/medusa/name_cache.py
@@ -89,7 +89,6 @@ def build_name_cache(series_obj=None):
     """Build internal name cache.
 
     :param series_obj: Specify series to build name cache for, if None, just do all series
-    :param force: Force the build name cache. Do not depend on the scene_exception_refresh table.
     """
     def _cache_name(cache_series_obj):
         """Build the name cache for a single show."""

diff --git a/medusa/name_parser/parser.py b/medusa/name_parser/parser.py
@@ -279,6 +279,12 @@ def _parse_series(result):
         new_absolute_numbers = []
 
         ex_season = scene_exceptions.get_season_from_name(result.series, result.series_name) or result.season_number
+        if ex_season is None:
+            ex_season = 1
+            log.info(
+                "For the show {name} we could not parse a season number. We did match the title, so we'll asume season 1",
+                {'name': result.series.name}
+            )
 
         for episode_number in result.episode_numbers:
             season = ex_season
@@ -300,6 +306,33 @@ def _parse_series(result):
 
         return new_episode_numbers, new_season_numbers, new_absolute_numbers
 
+    @staticmethod
+    def _parse_special(result):
+        new_episode_numbers = []
+        new_season_numbers = []
+
+        episode_title = result.guess.get('episode_title')
+        if not episode_title:
+            log.info(
+                '{name}: This episode is marked as a special. We could not find an episode title. And we need that to map it to an episode in the library.',
+                {'name': result.series.name}
+            )
+            return new_episode_numbers, new_season_numbers
+
+        # Sanitize the episode title.
+        episode_title = episode_title.lower()
+        if episode_title.startswith('special'):
+            episode_title = episode_title.split('special')[-1].strip()
+
+        all_episodes = result.series.get_all_episodes(season=0)
+        for special_episode in all_episodes:
+            if special_episode.name.lower() == episode_title:
+                new_season_numbers.append(0)
+                new_episode_numbers.append(special_episode.episode)
+                return new_episode_numbers, new_season_numbers
+
+        return [], []
+
     def _parse_string(self, name):
         guess = guessit.guessit(name, dict(show_type=self.show_type))
 
@@ -318,18 +351,22 @@ def _parse_string(self, name):
         new_season_numbers = []
         new_absolute_numbers = []
 
+        # Try to map special episodes without an episode number using the episode title.
+        if result.is_episode_special and not result.episode_numbers:
+            new_episode_numbers, new_season_numbers = self._parse_special(result)
+
         # if we have an air-by-date show and the result is air-by-date,
         # then get the real season/episode numbers
-        if result.series.air_by_date and result.is_air_by_date:
+        elif result.series.air_by_date and result.is_air_by_date:
             new_episode_numbers, new_season_numbers = self._parse_air_by_date(result)
 
         elif result.series.is_anime or result.is_anime:
             new_episode_numbers, new_season_numbers, new_absolute_numbers = self._parse_anime(result)
 
-        elif result.season_number is not None:
+        else:
             new_episode_numbers, new_season_numbers, new_absolute_numbers = self._parse_series(result)
 
-        else:
+        if not new_season_numbers and not new_episode_numbers:
             raise InvalidNameException('The result that was found ({result_name}) is not yet supported by Medusa '
                                        'and will be skipped. Sorry.'.format(result_name=result.original_name))
 
@@ -468,14 +505,15 @@ def to_parse_result(self, name, guess):
                            episode_numbers=helpers.ensure_list(guess.get('episode')),
                            ab_episode_numbers=helpers.ensure_list(guess.get('absolute_episode')),
                            air_date=guess.get('date'), release_group=guess.get('release_group'),
-                           proper_tags=helpers.ensure_list(guess.get('proper_tag')), version=guess.get('version', -1))
+                           proper_tags=helpers.ensure_list(guess.get('proper_tag')), version=guess.get('version', -1),
+                           episode_details=helpers.ensure_list(guess.get('episode_details')))
 
 
 class ParseResult(object):
     """Represent the release information for a given name."""
 
     def __init__(self, guess, series_name=None, season_number=None, episode_numbers=None, ab_episode_numbers=None,
-                 air_date=None, release_group=None, proper_tags=None, version=None, original_name=None):
+                 air_date=None, release_group=None, proper_tags=None, version=None, original_name=None, episode_details=None):
         """Initialize the class.
 
         :param guess:
@@ -498,6 +536,8 @@ def __init__(self, guess, series_name=None, season_number=None, episode_numbers=
         :type version: int
         :param original_name:
         :type original_name: str
+        :param episode_details:
+        :type episode_details: list of str
         """
         self.original_name = original_name
         self.series_name = series_name
@@ -512,6 +552,7 @@ def __init__(self, guess, series_name=None, season_number=None, episode_numbers=
         self.proper_tags = proper_tags
         self.guess = guess
         self.total_time = None
+        self.episode_details = episode_details
 
     def __eq__(self, other):
         """Equal implementation.

diff --git a/medusa/naming.py b/medusa/naming.py
@@ -83,11 +83,11 @@ def check_valid_naming(pattern=None, multi=None, anime_type=None):
     if anime_type is None:
         anime_type = app.NAMING_ANIME
 
-    logger.log(u'Checking whether the pattern ' + pattern + ' is valid for a single episode', logger.DEBUG)
+    logger.log(f'Checking whether the pattern {pattern} is valid for a single episode', logger.DEBUG)
     valid = validate_name(pattern, None, anime_type)
 
     if multi is not None:
-        logger.log(u'Checking whether the pattern ' + pattern + ' is valid for a multi episode', logger.DEBUG)
+        logger.log(f'Checking whether the pattern {pattern} is valid for a multi episode', logger.DEBUG)
         valid = valid and validate_name(pattern, multi, anime_type)
 
     return valid
@@ -169,6 +169,7 @@ def validate_name(pattern, multi=None, anime_type=None, file_only=False, abd=Fal
         if parse_result.season_number != ep.season:
             logger.log(u"Season number incorrect in parsed episode, pattern isn't valid", logger.DEBUG)
             return False
+        # If the template is a season search string, we don't need to check for episode.
         if parse_result.episode_numbers != [x.episode for x in [ep] + ep.related_episodes]:
             logger.log(u"Episode numbering incorrect in parsed episode, pattern isn't valid", logger.DEBUG)
             return False

diff --git a/medusa/providers/generic_provider.py b/medusa/providers/generic_provider.py
@@ -666,6 +666,21 @@ def _get_episode_search_strings(self, episode, add_string=''):
             'Episode': []
         }
 
+        # If show.use_templates is enabled, where using those instead of creating them here.
+        if episode.series.use_templates:
+            for template in episode.series.search_templates.templates:
+
+                # Only limit to episode search templates. And filter out disabled search templates.
+                if template.season_search or not template.enabled:
+                    continue
+
+                # Make sure we only use a season search template when searched for that season.
+                if episode.scene_season is not None and template.season != -1 and episode.scene_season != template.season:
+                    continue
+
+                search_string['Episode'].append(episode.formatted_search_string(template.template, title=template.title))
+            return [search_string]
+
         all_possible_show_names = episode.series.get_all_possible_names()
         if episode.scene_season is not None:
             all_possible_show_names = all_possible_show_names.union(
@@ -697,6 +712,18 @@ def _get_season_search_strings(self, episode):
             'Season': []
         }
 
+        # If show.use_templates is enabled, where using those instead of creating them here.
+        if episode.series.use_templates:
+            for template in episode.series.search_templates.templates:
+                if not template.season_search or not template.enabled:
+                    continue
+
+                if episode.scene_season and template.season != -1 and episode.scene_season != template.season:
+                    continue
+
+                search_string['Season'].append(episode.formatted_search_string(template.template, title=template.title))
+            return [search_string]
+
         for show_name in episode.series.get_all_possible_names(season=episode.scene_season):
             episode_string = show_name + self.search_separator
 

diff --git a/medusa/queues/show_queue.py b/medusa/queues/show_queue.py
@@ -689,6 +689,10 @@ def message_step(new_step):
             message_step('add scene numbering')
             self.show.add_scene_numbering()
 
+            # Load search templates
+            message_step('generate search templates')
+            self.show.init_search_templates()
+
             if self.show_dir:
                 # If a show dir was passed, this was added as an existing show.
                 # For new shows we should have any files on disk.

diff --git a/medusa/scene_exceptions.py b/medusa/scene_exceptions.py
@@ -169,7 +169,7 @@ def get_season_from_name(series_obj, exception_name):
 
 def get_all_scene_exceptions(series_obj):
     """
-    Get all scene exceptions for a show ID.
+    Get all scene exceptions for a show object using indexer and series_id.
 
     :param series_obj: series object.
     :return: dict of exceptions (e.g. exceptions_cache[season][exception_name])
@@ -239,6 +239,8 @@ def update_scene_exceptions(series_obj, scene_exceptions):
                 [series_obj.indexer, series_obj.series_id, exception['title'], exception['season'], exception['custom']]
             )
 
+    refresh_exceptions_cache(series_obj)
+
 
 def retrieve_exceptions(force=False, exception_type=None):
     """
@@ -283,7 +285,7 @@ def retrieve_exceptions(force=False, exception_type=None):
                 for scene_exception, season in iteritems(exception_dict):
                     if scene_exception not in existing_exceptions:
                         queries.append([
-                            'INSERT OR IGNORE INTO scene_exceptions '
+                            'INSERT OR IGNORE INTO scene_exceptions'
                             '(indexer, series_id, title, season, custom) '
                             'VALUES (?,?,?,?,?)',
                             [indexer, series_id, scene_exception, season, False]

diff --git a/medusa/scene_numbering.py b/medusa/scene_numbering.py
@@ -358,7 +358,7 @@ def get_indexer_numbering(series_obj, scene_episode, scene_season=None):
     """
     # Try to get a mapping from scene_numbering.
     season, episode = get_custom_numbering_from_scene(series_obj, scene_episode, scene_season)
-    if all((season, episode)):
+    if all((season is not None, episode)):
         return season, episode
 
     if series_obj.is_scene: