Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patches #59

Merged
merged 4 commits into from
Apr 20, 2016
Merged

Patches #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f96989d
Use request helper function in LayersScraper
kashav Apr 20, 2016
095f577
Patch json getters for Buildings
kashav Apr 20, 2016
26fe823
Fix infinite loop in request helper
kashav Apr 20, 2016
fac88e8
Add max attempts for exam requests
kashav Apr 20, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 4 additions & 7 deletions uoftscrapers/scrapers/buildings/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
from bs4 import BeautifulSoup
from collections import OrderedDict
from decimal import *
import json
import os
import re

Expand Down Expand Up @@ -83,13 +82,12 @@ def get_map_json(campus):
Scraper.get(Buildings.host)

headers = {'Referer': Buildings.host}
html = Scraper.get('%s%s%s' % (
data = Scraper.get('%s%s%s' % (
Buildings.host,
'data/map/',
campus
), headers=headers)
), headers=headers, json=True)

data = json.loads(html)
return data

@staticmethod
Expand All @@ -99,11 +97,10 @@ def get_regions_json(campus):
Scraper.get(Buildings.host)

headers = {'Referer': Buildings.host}
html = Scraper.get('%s%s%s' % (
data = Scraper.get('%s%s%s' % (
Buildings.host,
'data/regions/',
campus
), headers=headers)
), headers=headers, json=True)

data = json.loads(html)
return data
11 changes: 8 additions & 3 deletions uoftscrapers/scrapers/exams/utsg.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,15 @@ def scrape(location='.', year=None):
'Referer': UTSGExams.host
}
html = Scraper.get('%s%s' % (UTSGExams.host, p),
headers=headers)
soup = BeautifulSoup(html, 'html.parser')
headers=headers,
max_attempts=3)

if not soup.find('table', class_='vertical listing'):
try:
soup = BeautifulSoup(html, 'html.parser')
except TypeError:
soup = None

if not (html and soup and soup.find(class_='vertical listing')):
# no exam data available
Scraper.logger.info('No %s exams.' % p.upper())
continue
Expand Down
10 changes: 3 additions & 7 deletions uoftscrapers/scrapers/utils/layers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,20 @@ class LayersScraper:
"""

host = 'http://map.utoronto.ca/'
s = requests.Session()

@staticmethod
def get_layers_json(campus):
"""Retrieve the JSON structure from host."""

Scraper.logger.info('Retrieving map layers for %s.' % campus.upper())

headers = {
'Referer': LayersScraper.host
}
html = LayersScraper.s.get('%s%s%s' % (
headers = {'Referer': LayersScraper.host}
data = Scraper.get('%s%s%s' % (
LayersScraper.host,
'data/map/',
campus
), headers=headers).text
), headers=headers, json=True)

data = json.loads(html)
return data['layers']

@staticmethod
Expand Down
1 change: 1 addition & 0 deletions uoftscrapers/scrapers/utils/scraper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ def get(url, params=None, cookies=None, headers=None, json=False, max_attempts=1
doc = r
else:
sleep(0.5)
attempts += 1
Copy link
Member Author

@kashav kashav Apr 20, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this was by design, but whenever the request 404'd, it'd go on forever.

Try it with UTSGExams (any period past the current one).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops, yeah. we should better handle those errors anyway. I doubt there's any reason to try again if it got a 404.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CourseFinder often 404's, 401's, or 500's. It's a real pain.

except (requests.exceptions.Timeout,
requests.exceptions.ConnectionError):
attempts += 1
Expand Down