Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added initial IMDB indexer. #3603

Merged
merged 90 commits into from
Mar 1, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
c8e276e
Added initial IMDB indexer.
p0psicles Jan 12, 2018
b76d78e
Don't add video games and tv short.
p0psicles Jan 12, 2018
6912013
Fixed setting the imdb_id external.
p0psicles Jan 13, 2018
e94dd24
Added images (only poster for now).
p0psicles Jan 13, 2018
81f06ef
Added show status and firstaired using releases route.
p0psicles Jan 14, 2018
e325619
Added better support for poster and poster_thumbs.
p0psicles Jan 14, 2018
dc25c36
Added routes that still need to be added to imdbpie.py
p0psicles Jan 14, 2018
6222031
Fix show_url
duramato Jan 14, 2018
8b4e251
Merge remote-tracking branch 'remotes/origin/develop' into feature/ad…
p0psicles Feb 8, 2018
2223f45
Fix network and images.
p0psicles Feb 8, 2018
80a607b
Added episode rating, overview and airdate.
p0psicles Feb 9, 2018
cad5dea
Fixed bug where adding show without a summary, throwed an error.
p0psicles Feb 9, 2018
d3a4e7d
Added actors.
p0psicles Feb 9, 2018
06a99f4
Added fanart and production_art (as fanart).
p0psicles Feb 10, 2018
730620a
Use the poster initially provided with the show, as the images api, d…
p0psicles Feb 10, 2018
eace45e
Refactor indexers.
p0psicles Feb 10, 2018
4d9986b
Use get_nested_value() function for tmdb.
p0psicles Feb 11, 2018
817e695
tvmaze exceptions got removed.
p0psicles Feb 11, 2018
f3e87f3
Get airs day of week, from last 10 airdates.
p0psicles Feb 11, 2018
20b0d48
Added episode thumbnails with kodi_12plus.
p0psicles Feb 11, 2018
203ffbd
Merge remote-tracking branch 'remotes/origin/develop' into feature/ad…
p0psicles Feb 18, 2018
f1bfbc1
Fix issues with shows missing info.
p0psicles Feb 18, 2018
f956eec
Fix exception for when search doesn't return any results.
p0psicles Feb 19, 2018
65fb5d6
Use templating for the show_url.
p0psicles Feb 19, 2018
1800961
Change the addShows addShows/searchIndexersForShowName function to re…
p0psicles Feb 19, 2018
05d08c6
Fixed searching for The Tick.
p0psicles Feb 19, 2018
8d601eb
Disable the delay on ShowUpdater.
p0psicles Mar 15, 2018
81304ae
Update imdbpie to latest develop.
p0psicles Mar 15, 2018
4521d70
Modify get_last_updated_series parameters.
p0psicles Mar 15, 2018
6d1e970
Start changing show_updater.py to add the ability of adding using per…
p0psicles Mar 15, 2018
9a907b2
Merge remote-tracking branch 'remotes/origin/develop' into feature/ad…
p0psicles Mar 16, 2018
1ccbbea
Added per season calculated intervals for show updates.
p0psicles Mar 16, 2018
1b90fe9
restored the show update delay.
p0psicles Mar 16, 2018
920156f
Changed per season update to show updates for imdb.
p0psicles Mar 16, 2018
4ec0fde
Moved the show search in the exception handling block.
p0psicles Mar 17, 2018
36dfa7b
Merge remote-tracking branch 'origin/develop' into feature/add-imdb-i…
p0psicles Feb 18, 2022
6ed906a
Fix conflicts
p0psicles Feb 18, 2022
e48c213
Fix adding shows using imdb
p0psicles Feb 18, 2022
8a4a62a
And it loads the displayShow
p0psicles Feb 18, 2022
2c57175
Fixed a number of bugs
p0psicles Feb 19, 2022
4a0f630
Fix status tmdb
p0psicles Feb 20, 2022
ea901f3
Fix method call get_last_seasons for tvdb
p0psicles Feb 20, 2022
7858ba5
Fix show_updater
p0psicles Feb 20, 2022
8a6e227
Fix get external lookups for imdb.
p0psicles Feb 20, 2022
ad540a2
Fix getting rating/contentrating from imdb.
p0psicles Feb 21, 2022
2f6123e
Fix show imdb icon for non-imdb shows
p0psicles Feb 21, 2022
34d9b68
Fix regressions on providers with absolute numbering
p0psicles Feb 21, 2022
bdf374e
fixing old merge conflicts
p0psicles Feb 21, 2022
76c644e
Not using this
p0psicles Feb 21, 2022
7992b6d
Fix more indexer merge conflicts
p0psicles Feb 21, 2022
7fc2c80
Fix more merge conflicts
p0psicles Feb 21, 2022
5ff8c06
More merge conflicts
p0psicles Feb 21, 2022
e678649
revert the show_updater scheduler
p0psicles Feb 21, 2022
4025d71
Fix change-indexer
p0psicles Feb 21, 2022
8f6a1ab
Fix season updating for imdb
p0psicles Feb 21, 2022
6d3c8be
comment show_updater
p0psicles Feb 21, 2022
a572ca2
fix kodi metadata actor error
p0psicles Feb 21, 2022
4bd36a0
Fix get origin_country
p0psicles Feb 21, 2022
7f13fe2
Fix change-indexer
p0psicles Feb 22, 2022
57c1271
Lots of fixes:
p0psicles Feb 22, 2022
52a77e5
Fixed bug with actor without role?
p0psicles Feb 22, 2022
667203a
Add search imdb by id.
p0psicles Feb 22, 2022
ee0ba44
fix: Add existing shows
p0psicles Feb 22, 2022
ff23600
Make searching more flexible
p0psicles Feb 22, 2022
bf3ad02
added externals component
p0psicles Feb 22, 2022
8ec6e04
Get the thumb version for poster.
p0psicles Feb 22, 2022
fb6874f
Fix imdb status
p0psicles Feb 23, 2022
1303532
Extended tmdb exception handling.
p0psicles Feb 23, 2022
fd5f013
Fix error with lastaired
p0psicles Feb 23, 2022
76b8246
Adding log for when tt id's are cleaned from indexer_mapping.
p0psicles Feb 23, 2022
b8c133b
Fix get_episode error
p0psicles Feb 23, 2022
baaf043
Clean recommended_db
p0psicles Feb 23, 2022
e14be33
Improve searching
p0psicles Feb 23, 2022
d783c2a
add logs
p0psicles Feb 23, 2022
63b7444
simplify clean imdb tt
p0psicles Feb 23, 2022
77e7632
Fix parsing images tmdb
p0psicles Feb 24, 2022
41a688b
Fix exception handling for show_updater when updating through indexer…
p0psicles Feb 24, 2022
a6a8226
Get posters and banners by aspect_ratio
p0psicles Feb 25, 2022
b8e6fff
New imdb sorting logic
p0psicles Feb 25, 2022
19e3363
Improve the show season updates.
p0psicles Feb 25, 2022
6db36ff
Also applied to the other indexers
p0psicles Feb 25, 2022
aaefba0
Refactored the get_last_updated_seasons method.
p0psicles Feb 26, 2022
f68957b
comment
p0psicles Feb 26, 2022
5c273af
Fix imdb id parse error
p0psicles Feb 28, 2022
1cc83c8
Fix diverse imdb id mapping
p0psicles Feb 28, 2022
e23ebf6
Add login response check
p0psicles Feb 28, 2022
b3d7a8d
Added imdb exception handling
p0psicles Feb 28, 2022
5e8096d
Fix flake warnings
p0psicles Mar 1, 2022
763289e
Refactor and fix flake
p0psicles Mar 1, 2022
983f51b
Fix jest tests
p0psicles Mar 1, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added episode rating, overview and airdate.
  • Loading branch information
p0psicles committed Feb 9, 2018
commit 80a607bc0d1848635622ab44bcf19348c808cfe2
172 changes: 72 additions & 100 deletions medusa/indexers/imdb/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@

from __future__ import unicode_literals

import re
from datetime import datetime
from itertools import chain
import logging
from collections import OrderedDict
from imdbpie import imdbpie
from time import time
from six import text_type

import locale
from six import string_types, text_type
from medusa.bs4_parser import BS4Parser
from medusa.indexers.indexer_base import (Actor, Actors, BaseIndexer)
from medusa.indexers.indexer_exceptions import (
IndexerError,
Expand All @@ -30,6 +30,10 @@ def __init__(self, imdb_id):
self._series_id = None
self.imdb_id = imdb_id

def _clean(self, imdb_id):
if isinstance(imdb_id, string_types):
return imdb_id.strip('/').split('/')[-1]

@property
def series_id(self):
return self._series_id
Expand All @@ -44,9 +48,9 @@ def imdb_id(self):

@imdb_id.setter
def imdb_id(self, value):
if isinstance(value, text_type) and 'tt' in value:
self._imdb_id = value
self.series_id = int(value.split('tt')[-1])
if isinstance(value, string_types) and 'tt' in value:
self._imdb_id = self._clean(value)
self.series_id = int(self._imdb_id.split('tt')[-1])
else:
self._imdb_id = 'tt{0}'.format(text_type(value).zfill(7))
self.series_id = int(value)
Expand Down Expand Up @@ -137,6 +141,8 @@ def _map_results(self, imdb_response, key_mappings=None, list_separator='|'):
# return_dict['id'] = ImdbIdentifier(item.pop('imdb_id')).series_id
for key, config in self.series_map:
value = self.get_nested_value(item, config)
if not value:
continue
if key == 'id' and value:
value = ImdbIdentifier(value.rstrip('/')).series_id
if key == 'contentrating':
Expand Down Expand Up @@ -239,7 +245,7 @@ def _get_episodes(self, imdb_id, specials=False, aired_season=None): # pylint:
log.debug('Getting all episodes of {0}', imdb_id)

series_id = imdb_id
imdb_id = 'tt{0}'.format(text_type(imdb_id).zfill(7))
imdb_id = ImdbIdentifier(imdb_id).imdb_id
results = self.imdb_api.get_title_episodes(imdb_id)

if not results or not results.get('seasons'):
Expand All @@ -250,19 +256,75 @@ def _get_episodes(self, imdb_id, specials=False, aired_season=None): # pylint:
season_no, episode_no = episode.get('season'), episode.get('episode')

if season_no is None or episode_no is None:
log.warning('An episode has incomplete season/episode number (season: {0!r}, episode: {1!r})', seasnum, epno)
log.warning('An episode has incomplete season/episode number (season: {0!r}, episode: {1!r})',
season_no, episode_no)
continue # Skip to next episode

for k, config in self.episode_map:
v = self.get_nested_value(episode, config)
if v is not None:
if k == 'id':
v = ImdbIdentifier(v.rstrip('/')).series_id
v = ImdbIdentifier(v).series_id
if k == 'firstaired':
v = '{year}-01-01'.format(year=v)

self._set_item(series_id, season_no, episode_no, k, v)

if season.get('season'):
# Enrich episode for the current season.
self._enrich_episodes(imdb_id, season['season'])

def _enrich_episodes(self, imdb_id, season):
"""Enrich the episodes with additional information for a specific season."""
episodes_url = 'http://www.imdb.com/title/{imdb_id}/episodes?season={season}'
series_id = ImdbIdentifier(imdb_id).series_id
try:
response = self.config['session'].get(episodes_url.format(imdb_id=ImdbIdentifier(imdb_id).imdb_id, season=season))
with BS4Parser(response.text, 'html5lib') as html:
for episode in html.find_all('div', class_='list_item'):
try:
episode_no = int(episode.find('meta')['content'])
except AttributeError:
pass
try:
first_aired_raw = episode.find('div', class_='airdate').get_text(strip=True)
except AttributeError:
pass

lc = locale.setlocale(locale.LC_TIME)
try:
locale.setlocale(locale.LC_ALL, 'C')
first_aired = datetime.strptime(first_aired_raw.replace('.', ''), '%d %b %Y').strftime('%Y-%m-%d')
except (AttributeError, ValueError):
first_aired = None
finally:
locale.setlocale(locale.LC_TIME, lc)

try:
episode_rating = float(episode.find('span', class_='ipl-rating-star__rating').get_text(strip=True))
except AttributeError:
episode_rating = None

try:
episode_votes = int(episode.find('span', class_='ipl-rating-star__total-votes').get_text(strip=True).strip('()').replace(',', ''))
except AttributeError:
episode_votes = None

try:
synopsis = episode.find('div', class_='item_description').get_text(strip=True)
if 'Know what this is about?' in synopsis:
synopsis = ''
except AttributeError:
synopsis = ''

self._set_item(series_id, season, episode_no, 'firstaired', first_aired)
self._set_item(series_id, season, episode_no, 'rating', episode_rating)
self._set_item(series_id, season, episode_no, 'votes', episode_votes)
self._set_item(series_id, season, episode_no, 'overview', synopsis)

except Exception as error:
log.exception('Error while trying to enrich imdb series {0}, {1}', series_id, error)

def _parse_images(self, imdb_id):
"""Parse Show and Season posters.

Expand Down Expand Up @@ -393,28 +455,6 @@ def get_resolution(image):
# Save the image
self._set_show_data(series_id, img_type, img_url)

def _parse_season_images(self, imdb_id):
"""Parse Show and Season posters."""
seasons = {}
if imdb_id:
log.debug('Getting all show data for {0}', imdb_id)
try:
seasons = self.imdb_api.show_seasons(maze_id=imdb_id)
except BaseError as e:
log.warning('Getting show seasons for the season images failed. Cause: {0}', e)

_images = {'season': {'original': {}}}
# Get the season posters
for season in seasons.keys():
if not getattr(seasons[season], 'image', None):
continue
if season not in _images['season']['original']:
_images['season']['original'][season] = {seasons[season].id: {}}
_images['season']['original'][season][seasons[season].id]['_bannerpath'] = seasons[season].image['original']
_images['season']['original'][season][seasons[season].id]['rating'] = 1

return _images

def _parse_actors(self, imdb_id):
"""Parsers actors XML, from
http://theimdb.com/api/[APIKEY]/series/[SERIES ID]/actors.xml
Expand Down Expand Up @@ -509,71 +549,3 @@ def _get_show_data(self, imdb_id, language='en'): # pylint: disable=too-many-br
self._parse_actors(imdb_id)

return True

# def _get_all_updates(self, start_date=None, end_date=None):
# """Retrieve all updates (show,season,episode) from TVMaze."""
# results = []
# try:
# updates = self.imdb_api.show_updates()
# except (ShowIndexError, UpdateNotFound):
# return results
# except BaseError as e:
# log.warning('Getting show updates failed. Cause: {0}', e)
# return results
#
# if getattr(updates, 'updates', None):
# for show_id, update_ts in updates.updates.items():
# if start_date < update_ts.seconds_since_epoch < (end_date or int(time())):
# results.append(int(show_id))
#
# return results
#
# # Public methods, usable separate from the default api's interface api['show_id']
# def get_last_updated_series(self, from_time, weeks=1, filter_show_list=None):
# """Retrieve a list with updated shows
#
# :param from_time: epoch timestamp, with the start date/time
# :param weeks: number of weeks to get updates for.
# :param filter_show_list: Optional list of show objects, to use for filtering the returned list.
# """
# total_updates = []
# updates = self._get_all_updates(from_time, from_time + (weeks * 604800)) # + seconds in a week
#
# if updates and filter_show_list:
# new_list = []
# for show in filter_show_list:
# if show.indexerid in total_updates:
# new_list.append(show.indexerid)
# updates = new_list
#
# return updates

# def get_id_by_external(self, **kwargs):
# """Search imdb for a show, using an external id.
#
# Accepts as kwargs, so you'l need to add the externals as key/values.
# :param tvrage: The tvrage id.
# :param thetvdb: The tvdb id.
# :param imdb: An imdb id (inc. tt).
# :returns: A dict with externals, including the imdb id.
# """
# mapping = {'thetvdb': 'tvdb_id', 'tvrage': 'tvrage_id', 'imdb': 'imdb_id'}
# for external_id in mapping.values():
# if kwargs.get(external_id):
# try:
# result = self.imdb_api.get_show(**{external_id: kwargs.get(external_id)})
# if result:
# externals = {mapping[imdb_external_id]: external_value
# for imdb_external_id, external_value
# in result.externals.items()
# if external_value and mapping.get(imdb_external_id)}
# externals['imdb_id'] = result.maze_id
# return externals
# except ShowNotFound:
# log.debug('Could not get imdb externals using external key {0} and id {1}',
# external_id, kwargs.get(external_id))
# continue
# except BaseError as e:
# log.warning('Could not get imdb externals. Cause: {0}', e)
# continue
# return {}
2 changes: 1 addition & 1 deletion medusa/indexers/tvdbv2/tvdbv2_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ def _get_episodes(self, tvdb_id, specials=False, aired_season=None):

def _get_episodes_info(self, tvdb_id, episodes, season=None):
"""Add full episode information for existing episodes."""
series = Show.find_by_id(app.showList, self.indexer, tvdb_id)
series = Show.find_by_id(app.showList, 1, tvdb_id)
if not series:
return episodes

Expand Down