Added initial IMDB indexer. (#3603)

* Added initial IMDB indexer.
Working: Searching + basic adding of show.

* Don't add video games and tv short.

* Fixed setting the imdb_id external.

* Added images (only poster for now).

* Added show status and firstaired using releases route.

* Added better support for poster and poster_thumbs.

* Added routes that still need to be added to imdbpie.py

* Fix show_url

* Fix network and images.

* Added episode rating, overview and airdate.

* Fixed bug where adding show without a summary, throwed an error.

* Added actors.
* Don't display imdb icon, when it's an imdb show, as there is already an icon shown.
* added cross-env to package.json for building.

* Added fanart and production_art (as fanart).

* Use the poster initially provided with the show, as the images api, doesn't have any ratings attached to the images.
Use the production art as fanart.
* Rename the exceptions to be used as imdb exceptions.

* Refactor indexers.
* Use get_nested_value function for tvmaze.

* Use get_nested_value() function for tmdb.
* Fixed get_nested_value for when it's getting 0 or empty strings.

* tvmaze exceptions got removed.

* Get airs day of week, from last 10 airdates.

* Added episode thumbnails with kodi_12plus.

* Fix issues with shows missing info.
* correct merge conflict.

* Fix exception for when search doesn't return any results.
The raise, was removed somewhere.

* Use templating for the show_url.
* Imdb requires the tt with appended 0's in front of the id.

* Change the addShows addShows/searchIndexersForShowName function to return a key/value pair of values.
In stead of list.
* Refactored mako for the refactor of the indexer_api to api.

* Fixed searching for The Tick.
* Search was giving duplicate results. Filtered them by unique indexer and seriesId.
* Fixed issue where it cannot get a startyear, for a distribution. My guess is, this happens with stream providers like amazon and netflix.

* Disable the delay on ShowUpdater.

* Update imdbpie to latest develop.

* Modify get_last_updated_series parameters.
Add the cache parameter, for retrieving lastUpdates of specific show+season.

* Start changing show_updater.py to add the ability of adding using per season updates when no the indexer does not provide a list of recently updated shows.

* Added per season calculated intervals for show updates.

* restored the show update delay.
Fixed bug.

* Changed per season update to show updates for imdb.

* Moved the show search in the exception handling block.

* Fix conflicts

* Fix adding shows using imdb

* And it loads the displayShow

* Fixed a number of bugs

* Fix status tmdb

* Fix method call get_last_seasons for tvdb

* Fix show_updater

* Fix get external lookups for imdb.

* Fix getting rating/contentrating from imdb.

* Fix show imdb icon for non-imdb shows

* Fix regressions on providers with absolute numbering

* fixing old merge conflicts

* Not using this

* Fix more indexer merge conflicts

* Fix more merge conflicts

* More merge conflicts

* revert the show_updater scheduler

* Fix change-indexer

* Fix season updating for imdb

* comment show_updater

* fix kodi metadata actor error

* Fix get origin_country

* Fix change-indexer

* Lots of fixes:
* Refactored the imdb id thing
* Fixed recommended shows (adding imdb)

* Fixed bug with actor without role?

* Add search imdb by id.

* fix: Add existing shows

* Make searching more flexible

* added externals component

* Get the thumb version for poster.

* Fix imdb status

* Extended tmdb exception handling.
For AttributeError: 'NoneType' object has no attribute 'raise_for_status'

* Fix error with lastaired

* Adding log for when tt id's are cleaned from indexer_mapping.

* Fix get_episode error

* Clean recommended_db

* Improve searching

* add logs

* simplify clean imdb tt

* Fix parsing images tmdb

* Fix exception handling for show_updater when updating through indexers without a per season update list like tvmaze.

* Get posters and banners by aspect_ratio

* New imdb sorting logic

* Improve the show season updates.

* Limit the amount of info pulled for checking if we need a season update.
* Only pull the season that we need.

* Also applied to the other indexers

* Refactored the get_last_updated_seasons method.

* comment

* Fix imdb id parse error

* Fix diverse imdb id mapping

* Add login response check

* Added imdb exception handling

* Fix flake warnings

* Refactor and fix flake

* Fix jest tests

Co-authored-by: supergonkas <[email protected]>

medusa/__main__.py

@@ -1201,6 +1201,7 @@ def initialize(self, console_logging=True):
             # initialize the recommended shows database
             recommended_db_con = db.DBConnection('recommended.db')
             db.upgradeDatabase(recommended_db_con, recommended_db.InitialSchema)
+            db.sanityCheckDatabase(recommended_db_con, recommended_db.RecommendedSanityCheck)
 
             # Performs a vacuum on cache.db
             logger.debug(u'Performing a vacuum on the CACHE database')

medusa/classes.py

@@ -23,17 +23,21 @@
 
 from dateutil import parser
 
+
 from medusa import app, ws
 from medusa.common import (
     MULTI_EP_RESULT,
     Quality,
     SEASON_RESULT,
 )
+from medusa.helper.common import sanitize_filename
 from medusa.logger.adapters.style import BraceAdapter
 from medusa.search import SearchType
 
 from six import itervalues
 
+from trans import trans
+
 log = BraceAdapter(logging.getLogger(__name__))
 log.logger.addHandler(logging.NullHandler())
 
@@ -365,6 +369,22 @@ def select_series(self, all_series):
         search_results = []
         series_names = []
 
+        def searchterm_in_result(search_term, search_result):
+            norm_search_term = sanitize_filename(search_term.lower())
+            norm_result = sanitize_filename(search_result.lower())
+
+            if norm_search_term in norm_result:
+                return True
+
+            # translates national characters into similar sounding latin characters
+            # For ex. Физрук -> Fizruk
+            search_term_alpha = trans(self.config['searchterm'])
+
+            if search_term_alpha != search_term and search_term_alpha in norm_result:
+                return True
+
+            return False
+
         # get all available shows
         if all_series:
             if 'searchterm' in self.config:
@@ -382,8 +402,11 @@ def select_series(self, all_series):
                     if search_term.isdigit():
                         series_names.append(search_term)
 
+                    if search_term.startswith('tt'):
+                        series_names.append(search_term)
+
                     for name in series_names:
-                        if search_term.lower() in name.lower():
+                        if searchterm_in_result(search_term, name):
                             if 'firstaired' not in cur_show:
                                 default_date = parser.parse('1900-01-01').date()
                                 cur_show['firstaired'] = default_date.strftime(dateFormat)

medusa/databases/cache_db.py

@@ -16,10 +16,14 @@
 # Add new migrations at the bottom of the list
 # and subclass the previous migration.
 class InitialSchema(db.SchemaUpgrade):
+    """Cache.db initial schema class."""
+
     def test(self):
+        """Test db version."""
         return self.hasTable('db_version')
 
     def execute(self):
+        """Execute."""
         queries = [
             ('CREATE TABLE lastUpdate (provider TEXT, time NUMERIC);',),
             ('CREATE TABLE lastSearch (provider TEXT, time NUMERIC);',),
@@ -229,3 +233,19 @@ def test(self):
     def execute(self):
         self.connection.action('DROP TABLE IF EXISTS scene_exceptions;')
         self.inc_major_version()
+
+
+class AddSeasonUpdatesTable(RemoveSceneExceptionsTable):  # pylint:disable=too-many-ancestors
+    def test(self):
+        return self.hasTable('season_updates')
+
+    def execute(self):
+        self.connection.action(
+            """CREATE TABLE "season_updates" (
+              `season_updates_id`	INTEGER,
+              `indexer`	    INTEGER NOT NULL,
+              `series_id`	INTEGER NOT NULL,
+              `season`	    INTEGER,
+              `time`	    INTEGER,
+              PRIMARY KEY(season_updates_id))"""
+        )

medusa/databases/main_db.py

@@ -36,6 +36,7 @@ def check(self):
         self.fix_show_nfo_lang()
         self.fix_subtitle_reference()
         self.clean_null_indexer_mappings()
+        self.clean_imdb_tt_ids()
 
     def clean_null_indexer_mappings(self):
         log.debug(u'Checking for null indexer mappings')
@@ -219,6 +220,13 @@ def fix_subtitles_codes(self):
     def fix_show_nfo_lang(self):
         self.connection.action("UPDATE tv_shows SET lang = '' WHERE lang = 0 OR lang = '0';")
 
+    def clean_imdb_tt_ids(self):
+        # Get all records with 'tt'
+        log.debug(u'Cleaning indexer_mapping table, removing references to same indexer')
+        self.connection.action('DELETE from indexer_mapping WHERE indexer = mindexer')
+        log.debug(u'Cleaning indexer_mapping table from tt indexer ids')
+        self.connection.action("DELETE FROM indexer_mapping where indexer_id like '%tt%' or mindexer_id like '%tt%'")
+
 
 # ======================
 # = Main DB Migrations =

medusa/databases/recommended_db.py

@@ -12,6 +12,19 @@
 log.logger.addHandler(logging.NullHandler())
 
 
+class RecommendedSanityCheck(db.DBSanityCheck):
+    """Sanity check class."""
+
+    def check(self):
+        """Check functions."""
+        self.remove_imdb_tt()
+
+    def remove_imdb_tt(self):
+        """Remove tt from imdb id's."""
+        log.debug(u'Remove shows added with an incorrect imdb id.')
+        self.connection.action("DELETE FROM shows WHERE source = 10 AND series_id like '%tt%'")
+
+
 # Add new migrations at the bottom of the list
 # and subclass the previous migration.
 class InitialSchema(db.SchemaUpgrade):

medusa/helpers/trakt.py

@@ -3,6 +3,7 @@
 import logging
 
 from medusa.helpers import get_title_without_year
+from medusa.indexers.imdb.api import ImdbIdentifier
 from medusa.logger.adapters.style import BraceAdapter
 
 from requests.exceptions import RequestException
@@ -70,8 +71,11 @@ def create_show_structure(show_obj):
         'ids': {}
     }
     for valid_trakt_id in ['tvdb_id', 'trakt_id', 'tmdb_id', 'imdb_id']:
-        if show_obj.externals.get(valid_trakt_id):
-            show['ids'][valid_trakt_id[:-3]] = show_obj.externals.get(valid_trakt_id)
+        external = show_obj.externals.get(valid_trakt_id)
+        if external:
+            if valid_trakt_id == 'imdb_id':
+                external = ImdbIdentifier(external).imdb_id
+            show['ids'][valid_trakt_id[:-3]] = external
     return show
 
 

medusa/indexers/api.py

@@ -32,9 +32,12 @@ def indexer(self, *args, **kwargs):
     def config(self):
         if self.indexer_id:
             return indexerConfig[self.indexer_id]
-        # Sort and put the default language first
-        init_config['valid_languages'].sort(key=lambda i: '\0' if i == app.INDEXER_DEFAULT_LANGUAGE else i)
-        return init_config
+        _ = init_config
+        if app.INDEXER_DEFAULT_LANGUAGE in _:
+            del _[_['valid_languages'].index(app.INDEXER_DEFAULT_LANGUAGE)]
+        _['valid_languages'].sort()
+        _['valid_languages'].insert(0, app.INDEXER_DEFAULT_LANGUAGE)
+        return _
 
     @property
     def name(self):

medusa/indexers/base.py

@@ -20,13 +20,14 @@
     IndexerSeasonNotFound,
     IndexerSeasonUpdatesNotSupported,
     IndexerShowNotFound,
+    IndexerShowUpdatesNotSupported,
 )
 from medusa.indexers.ui import BaseUI, ConsoleUI
 from medusa.logger.adapters.style import BraceAdapter
 from medusa.session.core import IndexerSession
 from medusa.statistics import weights
 
-from six import integer_types, itervalues, string_types, text_type, viewitems
+from six import integer_types, itervalues, string_types, viewitems
 
 
 log = BraceAdapter(logging.getLogger(__name__))
@@ -57,24 +58,18 @@ def __init__(self,
         """Pass these arguments on as args from the subclass."""
         self.shows = ShowContainer()  # Holds all Show classes
         self.corrections = {}  # Holds show-name to show_id mapping
-
-        self.config = {}
-
-        self.config['debug_enabled'] = debug  # show debugging messages
-
-        self.config['custom_ui'] = custom_ui
-
-        self.config['interactive'] = interactive  # prompt for correct series?
-
-        self.config['select_first'] = select_first
-
-        self.config['search_all_languages'] = search_all_languages
-
-        self.config['use_zip'] = use_zip
-
-        self.config['dvdorder'] = dvdorder
-
-        self.config['proxy'] = proxy
+        self.name = None
+
+        self.config = {
+            'debug_enabled': debug,
+            'custom_ui': custom_ui,
+            'interactive': interactive,
+            'select_first': select_first,
+            'search_all_languages': search_all_languages,
+            'use_zip': use_zip,
+            'dvdorder': dvdorder,
+            'proxy': proxy
+        }
 
         if cache is True:
             self.config['cache_enabled'] = True
@@ -93,6 +88,7 @@ def __init__(self,
         self.config['banners_enabled'] = banners
         self.config['image_type'] = image_type
         self.config['actors_enabled'] = actors
+        self.config['limit_seasons'] = []
 
         if self.config['debug_enabled']:
             warnings.warn('The debug argument to tvdbv2_api.__init__ will be removed in the next version. '
@@ -127,7 +123,46 @@ def __init__(self,
             else:
                 self.config['language'] = language
 
-    def _get_temp_dir(self):  # pylint: disable=no-self-use
+    def get_nested_value(self, value, config):
+        """
+        Get a nested value from a dictionary using a dot separated string.
+
+        For example the config 'plot.summaries[0].text' will return the value for dict['plot']['summaries'][0].
+        :param value: Dictionary you want to get a value from.
+        :param config: Dot separated string.
+        :return: The value matching the config.
+        """
+        # Remove a level
+        split_config = config.split('.')
+        check_key = split_config[0]
+
+        if check_key.endswith(']'):
+            list_index = int(check_key.split('[')[-1].rstrip(']'))
+            check_key = check_key.split('[')[0]
+            check_value = value.get(check_key)
+            if check_value and list_index < len(check_value):
+                check_value = check_value[list_index]
+        else:
+            check_value = value.get(check_key)
+        next_keys = '.'.join(split_config[1:])
+
+        if check_value is None:
+            return None
+
+        if isinstance(check_value, dict) and next_keys:
+            return self.get_nested_value(check_value, next_keys)
+        else:
+            try:
+                # Some object have a __dict__ attr. Let's try that.
+                # It shouldn't match basic types like strings, integers or floats.
+                parse_dict = check_value.__dict__
+            except AttributeError:
+                return check_value
+            else:
+                return self.get_nested_value(parse_dict, next_keys)
+
+    @staticmethod
+    def _get_temp_dir():  # pylint: disable=no-self-use
         """Return the [system temp dir]/tvdb_api-u501 (or tvdb_api-myuser)."""
         if hasattr(os, 'getuid'):
             uid = 'u{0}'.format(os.getuid())  # pylint: disable=no-member
@@ -145,19 +180,21 @@ def _get_show_data(self, sid, language):
         return None
 
     def _get_series(self, series):
-        """Search for the series name.
+        """Search indexer for the series name.
 
         If a custom_ui UI is configured, it uses this to select the correct
         series. If not, and interactive == True, ConsoleUI is used, if not
         BaseUI is used to select the first result.
 
         :param series: the query for the series name
-        :return: A list of series mapped to a UI (for example: a BaseUI or custom_ui).
+        :return: A list of series mapped to a UI (for example: a BaseUi or custom_ui).
         """
         all_series = self.search(series)
         if not all_series:
             log.debug('Series result returned zero')
-            raise IndexerShowNotFound('Show search returned zero results (cannot find show on Indexer)')
+            raise IndexerShowNotFound(
+                'Show search for {series} returned zero results (cannot find show on Indexer)'.format(series=series)
+            )
 
         if not isinstance(all_series, list):
             all_series = [all_series]
@@ -184,7 +221,7 @@ def _set_show_data(self, sid, key, value):
 
     def __repr__(self):
         """Indexer representation, returning representation of all shows indexed."""
-        return text_type(self.shows)
+        return str(self.shows)
 
     def _set_item(self, sid, seas, ep, attrib, value):  # pylint: disable=too-many-arguments
         """Create a new episode, creating Show(), Season() and Episode()s as required.
@@ -391,14 +428,14 @@ def _save_images(self, series_id, images):
             self._save_images_by_type(img_type, series_id, images_by_type)
 
     def __getitem__(self, key):
-        """Handle tvdbv2_instance['seriesname'] calls. The dict index should be the show id."""
+        """Handle indexer['seriesname'] calls. The dict index should be the show id."""
         if isinstance(key, (integer_types, int)):
             # Item is integer, treat as show id
             if key not in self.shows:
                 self._get_show_data(key, self.config['language'])
             return self.shows[key]
 
-        key = text_type(key).lower()
+        key = str(key).lower()
         self.config['searchterm'] = key
         selected_series = self._get_series(key)
         if isinstance(selected_series, dict):
@@ -409,19 +446,14 @@ def __getitem__(self, key):
                 self._set_show_data(show['id'], k, v)
         return selected_series
 
-    def get_last_updated_series(self, from_time, weeks=1, filter_show_list=None):
-        """Retrieve a list with updated shows.
+    def get_last_updated_series(self, *args, **kwargs):
+        """Retrieve a list with updated shows."""
+        raise IndexerShowUpdatesNotSupported('Method get_last_updated_series not implemented by this indexer')
 
-        :param from_time: epoch timestamp, with the start date/time
-        :param weeks: number of weeks to get updates for.
-        :param filter_show_list: Optional list of show objects, to use for filtering the returned list.
-        """
+    def get_last_updated_seasons(self, *args, **kwargs):
+        """Retrieve a list with updated show seasons."""
         raise IndexerSeasonUpdatesNotSupported('Method get_last_updated_series not implemented by this indexer')
 
-    def get_episodes_for_season(self, show_id, *args, **kwargs):
-        self._get_episodes(show_id, *args, **kwargs)
-        return self.shows[show_id]
-
 
 class ShowContainer(dict):
     """Simple dict that holds a series of Show instances."""
@@ -502,7 +534,7 @@ def __bool__(self):
 
     def aired_on(self, date):
         """Search and return a list of episodes with the airdates."""
-        ret = self.search(text_type(date), 'firstaired')
+        ret = self.search(str(date), 'firstaired')
         if len(ret) == 0:
             raise IndexerEpisodeNotFound('Could not find any episodes that aired on {0}'.format(date))
         return ret
@@ -631,13 +663,13 @@ def search(self, term=None, key=None):
         if term is None:
             raise TypeError('must supply string to search for (contents)')
 
-        term = text_type(term).lower()
+        term = str(term).lower()
         for cur_key, cur_value in viewitems(self):
-            cur_key, cur_value = text_type(cur_key).lower(), text_type(cur_value).lower()
+            cur_key, cur_value = str(cur_key).lower(), str(cur_value).lower()
             if key is not None and cur_key != key:
                 # Do not search this key
                 continue
-            if cur_value.find(text_type(term).lower()) > -1:
+            if cur_value.find(str(term).lower()) > -1:
                 return self