Django 2.2 #7196

SmileyChris · 2019-05-15T23:33:18Z

Upgrade to Django 2.2 and Python 3

I rebased against the Django 1.11 merged branch, then merged with master - had to deal with a few conflicts but was mostly trivial.

Tests for bedrock and lib are passing locally. @jgmize Let's do a more thorough test and see how things are looking...

jgmize · 2019-05-16T03:42:36Z

@SmileyChris based on my reading of https://circleci.com/gh/mozilla/bedrock/18651 we're getting a NameError: name 'get_page_content_cards' is not defined in bedrock/firefox/views.py from ./manage.py update_sitemaps. This same error also breaks make clean build-ci during our jenkins build pipeline.

jgmize · 2019-05-16T13:39:45Z

0a2211d fixed #7196 (comment) but now there's a new error in the same command. Once CirlcleCI is passing, I'll push this to the run-integration-tests branch to trigger that pipeline on our infra again.

jgmize · 2019-05-17T07:42:28Z

Now that 917efbd is passing all the CircleCI checks, I've pushed it to the run-integration-tests branch, triggering https://ci.vpn1.moz.works/blue/organizations/jenkins/bedrock_multibranch_pipeline/detail/run-integration-tests/309/pipeline -- for security reasons, that link is only accessible from a specific team-only vpn, but I will copy any relevant logs here in the event of any failures.

jgmize · 2019-05-17T07:50:25Z

Got the following error from the CI job above:

py.test -r a --verbose -n 5 --base-url https://bedrock-integration-tests.oregon-b.moz.works --reruns 2 --html /app/results/index.html --junitxml /app/results/junit.xml --driver SauceLabs --capability browserName chrome --capability platform 'Windows 10' --capability selenium-version 3.5.0 --capability build jenkins-bedrock_multibranch_pipeline-run-integration-tests-309 --capability public 'public restricted' -m 'not headless and not download' tests
============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /venv/bin/python
cachedir: .pytest_cache
driver: SauceLabs
sensitiveurl: (mozilla\.(com|org)|bedrock-prod)
metadata: {'Python': '3.7.3', 'Platform': 'Linux-4.4.0-148-generic-x86_64-with-debian-9.9', 'Packages': {'pytest': '4.4.1', 'py': '1.8.0', 'pluggy': '0.9.0'}, 'Plugins': {'xdist': '1.28.0', 'variables': '1.7.1', 'selenium': '1.16.0', 'rerunfailures': '7.0', 'metadata': '1.8.0', 'html': '1.20.0', 'forked': '1.0.2', 'django': '3.4.8', 'base-url': '1.4.1'}, 'PLATFORM': 'Windows 10', 'BUILD_TAG': 'jenkins-bedrock_multibranch_pipeline-run-integration-tests-309', 'Base URL': 'https://bedrock-integration-tests.oregon-b.moz.works', 'Driver': 'SauceLabs', 'Capabilities': {'browserName': 'chrome', 'platform': 'Windows 10', 'selenium-version': '3.5.0', 'build': 'jenkins-bedrock_multibranch_pipeline-run-integration-tests-309', 'public': 'public restricted'}}
Django settings: bedrock.settings.test (from ini file)
baseurl: https://bedrock-integration-tests.oregon-b.moz.works
rootdir: /app, inifile: setup.cfg
plugins: xdist-1.28.0, variables-1.7.1, selenium-1.16.0, rerunfailures-7.0, metadata-1.8.0, html-1.20.0, forked-1.0.2, django-3.4.8, base-url-1.4.1
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 3, in <module>
  File "/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 18, in <module>
    import traceback
  File "/usr/local/lib/python3.7/traceback.py", line 5, in <module>
    import linecache
  File "/venv/lib/python3.7/linecache.py", line 11, in <module>
    import tokenize
  File "/venv/lib/python3.7/tokenize.py", line 33, in <module>
    import re
  File "/venv/lib/python3.7/re.py", line 143, in <module>
    class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'
INFO 2019-05-17 00:41:52,374 base Raven is not configured (logging is disabled). Please see the documentation for more information.
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/_pytest/main.py", line 207, in wrap_session
INTERNALERROR>     config.hook.pytest_sessionstart(session=session)
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/pluggy/manager.py", line 68, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/pluggy/manager.py", line 62, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/xdist/dsession.py", line 81, in pytest_sessionstart
INTERNALERROR>     nodes = self.nodemanager.setup_nodes(putevent=self.queue.put)
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/xdist/workermanage.py", line 64, in setup_nodes
INTERNALERROR>     nodes.append(self.setup_node(spec, putevent))
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/xdist/workermanage.py", line 68, in setup_node
INTERNALERROR>     gw = self.group.makegateway(spec)
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/execnet/multi.py", line 128, in makegateway
INTERNALERROR>     gw = gateway_bootstrap.bootstrap(io, spec)
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/execnet/gateway_bootstrap.py", line 92, in bootstrap
INTERNALERROR>     bootstrap_import(io, spec)
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/execnet/gateway_bootstrap.py", line 27, in bootstrap_import
INTERNALERROR>     s = io.read(1)
INTERNALERROR>   File "/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 389, in read
INTERNALERROR>     "expected %d bytes, got %d" % (numbytes, len(buf)))
INTERNALERROR> EOFError: expected 1 bytes, got 0

jgmize · 2019-05-17T07:52:49Z

I've deployed 917efbd to https://bedrock-demo-pr7196.oregon-b.moz.works for additional testing.

jgmize · 2019-05-17T17:31:09Z

It looks like the enum issue may be due to the enum34 requirement in requirements/base.txt based on https://stackoverflow.com/a/45716067

jgmize · 2019-05-22T20:44:59Z

@SmileyChris as I mentioned in our meeting earlier, please feel free to ping me on slack once you've got enum34 removed from requirements/base.txt and rebased against master, and I'll push this to the run-integration-tests branch again.

jgmize · 2019-05-24T19:21:20Z

Status update

the enum issues have been resolved in the latest commit
integration tests were failing earlier due to an unrelated issue
- https://bugzilla.mozilla.org/show_bug.cgi?id=1553396
  - Fix bug 1553396: Update Fennec Nightly direct download links to bouncer #7213 is related to this bug
  - this PR will get that fix when rebased against master
logs from latest test runs triggered by another push to the run-integration-tests branch below

Chrome tests

docker/bin/run_integration_tests.sh chrome
DRIVER=SauceLabs
MARK_EXPRESSION='not headless and not download'
BROWSER_NAME=chrome
PLATFORM='Windows 10'
similar issues seen on firefox tests

==================================== ERRORS ====================================

_____________ ERROR at setup of test_communities_mobile_navigation _____________

[gw4] linux -- Python 3.7.3 /venv/bin/python


request = <SubRequest 'selenium' for <Function test_communities_mobile_navigation>>

selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="ae54a9b22bd042b9a389b689aab2ecdb")>


    @pytest.fixture

    def selenium(request, selenium):

        viewport = VIEWPORT['desktop']

        if request.keywords.get('viewport') is not None:

>           viewport = VIEWPORT[request.keywords.get('viewport').args[0]]

E           AttributeError: 'bool' object has no attribute 'args'


request    = <SubRequest 'selenium' for <Function test_communities_mobile_navigation>>

selenium   = <selenium.webdriver.remote.webdriver.WebDriver (session="ae54a9b22bd042b9a389b689aab2ecdb")>

viewport   = {'height': 1024, 'width': 1280}


tests/functional/conftest.py:65: AttributeError

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 11:58:55,657 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 11:59:03,672 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 11:59:12,201 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

------------------------------- pytest-selenium --------------------------------

URL: data:,

SauceLabs Job: https://saucelabs.com/jobs/ae54a9b22bd042b9a389b689aab2ecdb

_______________ ERROR at setup of test_spaces_mobile_navigation ________________

[gw3] linux -- Python 3.7.3 /venv/bin/python


request = <SubRequest 'selenium' for <Function test_spaces_mobile_navigation>>

selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="b74770b844a04f87880c2dde1b7af574")>


    @pytest.fixture

    def selenium(request, selenium):

        viewport = VIEWPORT['desktop']

        if request.keywords.get('viewport') is not None:

>           viewport = VIEWPORT[request.keywords.get('viewport').args[0]]

E           AttributeError: 'bool' object has no attribute 'args'


request    = <SubRequest 'selenium' for <Function test_spaces_mobile_navigation>>

selenium   = <selenium.webdriver.remote.webdriver.WebDriver (session="b74770b844a04f87880c2dde1b7af574")>

viewport   = {'height': 1024, 'width': 1280}


tests/functional/conftest.py:65: AttributeError

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 11:58:52,106 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 11:59:01,037 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 11:59:12,183 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

------------------------------- pytest-selenium --------------------------------

URL: data:,

SauceLabs Job: https://saucelabs.com/jobs/b74770b844a04f87880c2dde1b7af574

____________________ ERROR at setup of test_list_displayed _____________________

[gw1] linux -- Python 3.7.3 /venv/bin/python


request = <SubRequest 'selenium' for <Function test_list_displayed>>

selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="bbeb55ffec1f435faf7b926b7ee0054c")>


    @pytest.fixture

    def selenium(request, selenium):

        viewport = VIEWPORT['desktop']

        if request.keywords.get('viewport') is not None:

>           viewport = VIEWPORT[request.keywords.get('viewport').args[0]]

E           AttributeError: 'bool' object has no attribute 'args'


request    = <SubRequest 'selenium' for <Function test_list_displayed>>

selenium   = <selenium.webdriver.remote.webdriver.WebDriver (session="bbeb55ffec1f435faf7b926b7ee0054c")>

viewport   = {'height': 1024, 'width': 1280}


tests/functional/conftest.py:65: AttributeError

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 11:59:23,411 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 11:59:33,136 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 11:59:42,152 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

------------------------------- pytest-selenium --------------------------------

URL: data:,

SauceLabs Job: https://saucelabs.com/jobs/bbeb55ffec1f435faf7b926b7ee0054c

___________________ ERROR at setup of test_mobile_navigation ___________________

[gw3] linux -- Python 3.7.3 /venv/bin/python


request = <SubRequest 'selenium' for <Function test_mobile_navigation>>

selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="a466f971bffd4aad9329e5bb86cc462f")>


    @pytest.fixture

    def selenium(request, selenium):

        viewport = VIEWPORT['desktop']

        if request.keywords.get('viewport') is not None:

>           viewport = VIEWPORT[request.keywords.get('viewport').args[0]]

E           AttributeError: 'bool' object has no attribute 'args'


request    = <SubRequest 'selenium' for <Function test_mobile_navigation>>

selenium   = <selenium.webdriver.remote.webdriver.WebDriver (session="a466f971bffd4aad9329e5bb86cc462f")>

viewport   = {'height': 1024, 'width': 1280}


tests/functional/conftest.py:65: AttributeError

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:00:47,624 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:00:56,505 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:01:08,156 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

------------------------------- pytest-selenium --------------------------------

URL: data:,

SauceLabs Job: https://saucelabs.com/jobs/a466f971bffd4aad9329e5bb86cc462f

------------------ generated xml file: /app/results/junit.xml ------------------

----------------- generated html file: /app/results/index.html -----------------

=========================== short test summary info ============================

SKIPPED [2] /app/tests/functional/conftest.py:59: Firefox Accounts CTA is displayed only to Firefox users

SKIPPED [1] /app/tests/functional/conftest.py:59: Non-Firefox user does not see Firefox state

SKIPPED [4] /app/tests/functional/conftest.py:59: Whatsnew pages are shown to Firefox only.

SKIPPED [1] /app/tests/functional/conftest.py:59: Create Account form is only displayed to Firefox users

ERROR tests/functional/test_contact.py::test_communities_mobile_navigation

ERROR tests/functional/test_contact.py::test_spaces_mobile_navigation

ERROR tests/functional/test_history.py::test_list_displayed

ERROR tests/functional/test_navigation.py::test_mobile_navigation

========== 102 passed, 8 skipped, 4 error, 8 rerun in 543.84 seconds ===========

INFO 2019-05-24 11:57:41,460 base Raven is not configured (logging is disabled). Please see the documentation for more information.

script returned exit code 1

1.13.2  · Core 2.178  · fa27496  · 6th March 2019 06:07 PM

headless tests

docker/bin/run_integration_tests.sh headless


=================================== FAILURES ===================================
_ test_generated_pages[https://bedrock-integration-tests.oregon-b.moz.works/about/forums/] _
[gw0] linux -- Python 3.7.3 /venv/bin/python

url = 'https://bedrock-integration-tests.oregon-b.moz.works/about/forums/'

    @pytest.mark.headless
    @pytest.mark.nondestructive
    def test_generated_pages(url):
        r = requests.head(url, allow_redirects=True, timeout=TIMEOUT)
>       assert requests.codes.ok == r.status_code
E       assert 200 == 500
E         -200
E         +500

r          = <Response [500]>
url        = 'https://bedrock-integration-tests.oregon-b.moz.works/about/forums/'

tests/functional/test_generated_pages.py:69: AssertionError
_________________ test_link_hreflang_tags[/firefox/-locales1] __________________
[gw1] linux -- Python 3.7.3 /venv/bin/python

url = '/firefox/', locales = ('en-US', 'de', 'id')
base_url = 'https://bedrock-integration-tests.oregon-b.moz.works'

    @pytest.mark.headless
    @pytest.mark.nondestructive
    @pytest.mark.parametrize('url,locales', [
        ('/firefox/new/', ('en-US', 'de', 'id')),
        ('/firefox/', ('en-US', 'de', 'id')),
        ('/', ('en-US', 'de', 'id')),
    ])
    def test_link_hreflang_tags(url, locales, base_url):
        for locale in locales:
            full_url = '{}/{}{}'.format(base_url, locale, url)
            link_url = '{}/{}{}'.format('https://www.mozilla.org', locale, url)
            resp = requests.get(full_url, timeout=5)
>           assert LINK_TEMPLATE.format(url=link_url) in resp.content
E           TypeError: a bytes-like object is required, not 'str'

base_url   = 'https://bedrock-integration-tests.oregon-b.moz.works'
full_url   = 'https://bedrock-integration-tests.oregon-b.moz.works/en-US/firefox/'
link_url   = 'https://www.mozilla.org/en-US/firefox/'
locale     = 'en-US'
locales    = ('en-US', 'de', 'id')
resp       = <Response [200]>
url        = '/firefox/'

tests/functional/test_link_hreflang_tags.py:24: TypeError
_______________ test_link_hreflang_tags[/firefox/new/-locales0] ________________
[gw0] linux -- Python 3.7.3 /venv/bin/python

url = '/firefox/new/', locales = ('en-US', 'de', 'id')
base_url = 'https://bedrock-integration-tests.oregon-b.moz.works'

    @pytest.mark.headless
    @pytest.mark.nondestructive
    @pytest.mark.parametrize('url,locales', [
        ('/firefox/new/', ('en-US', 'de', 'id')),
        ('/firefox/', ('en-US', 'de', 'id')),
        ('/', ('en-US', 'de', 'id')),
    ])
    def test_link_hreflang_tags(url, locales, base_url):
        for locale in locales:
            full_url = '{}/{}{}'.format(base_url, locale, url)
            link_url = '{}/{}{}'.format('https://www.mozilla.org', locale, url)
            resp = requests.get(full_url, timeout=5)
>           assert LINK_TEMPLATE.format(url=link_url) in resp.content
E           TypeError: a bytes-like object is required, not 'str'

base_url   = 'https://bedrock-integration-tests.oregon-b.moz.works'
full_url   = 'https://bedrock-integration-tests.oregon-b.moz.works/en-US/firefox/new/'
link_url   = 'https://www.mozilla.org/en-US/firefox/new/'
locale     = 'en-US'
locales    = ('en-US', 'de', 'id')
resp       = <Response [200]>
url        = '/firefox/new/'

tests/functional/test_link_hreflang_tags.py:24: TypeError
_ test_generated_pages[https://bedrock-integration-tests.oregon-b.moz.works/credits/] _
[gw2] linux -- Python 3.7.3 /venv/bin/python

url = 'https://bedrock-integration-tests.oregon-b.moz.works/credits/'

    @pytest.mark.headless
    @pytest.mark.nondestructive
    def test_generated_pages(url):
>       r = requests.head(url, allow_redirects=True, timeout=TIMEOUT)

url        = 'https://bedrock-integration-tests.oregon-b.moz.works/credits/'

tests/functional/test_generated_pages.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/venv/lib/python3.7/site-packages/requests/api.py:101: in head
    return request('head', url, **kwargs)
/venv/lib/python3.7/site-packages/requests/api.py:60: in request
    return session.request(method=method, url=url, **kwargs)
/venv/lib/python3.7/site-packages/requests/sessions.py:533: in request
    resp = self.send(prep, **send_kwargs)
/venv/lib/python3.7/site-packages/raven/breadcrumbs.py:341: in send
    resp = real_send(self, request, *args, **kwargs)
/venv/lib/python3.7/site-packages/requests/sessions.py:668: in send
    history = [resp for resp in gen] if allow_redirects else []
/venv/lib/python3.7/site-packages/requests/sessions.py:668: in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <requests.sessions.Session object at 0x7fb25bfa30b8>
resp = <Response [302]>, req = <PreparedRequest [HEAD]>, stream = False
timeout = 60, verify = True, cert = None, proxies = OrderedDict()
yield_requests = False, adapter_kwargs = {}
hist = [<Response [302]>, <Response [301]>, <Response [302]>, <Response [301]>, <Response [302]>, <Response [301]>, ...]
url = '/en-US/credits/', previous_fragment = ''
prepared_request = <PreparedRequest [HEAD]>
parsed = ParseResult(scheme='', netloc='', path='/credits/', params='', query='', fragment='')
purged_headers = ('Content-Length', 'Content-Type', 'Transfer-Encoding')

    def resolve_redirects(self, resp, req, stream=False, timeout=None,
                          verify=True, cert=None, proxies=None, yield_requests=False, **adapter_kwargs):
        """Receives a Response. Returns a generator of Responses or Requests."""
    
        hist = []  # keep track of history
    
        url = self.get_redirect_target(resp)
        previous_fragment = urlparse(req.url).fragment
        while url:
            prepared_request = req.copy()
    
            # Update history and keep track of redirects.
            # resp.history must ignore the original request in this loop
            hist.append(resp)
            resp.history = hist[1:]
    
            try:
                resp.content  # Consume socket so it can be released
            except (ChunkedEncodingError, ContentDecodingError, RuntimeError):
                resp.raw.read(decode_content=False)
    
            if len(resp.history) >= self.max_redirects:
>               raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects, response=resp)
E               requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

adapter_kwargs = {}
cert       = None
header     = 'Transfer-Encoding'
headers    = {'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
hist       = [<Response [302]>, <Response [301]>, <Response [302]>, <Response [301]>, <Response [302]>, <Response [301]>, ...]
parsed     = ParseResult(scheme='', netloc='', path='/credits/', params='', query='', fragment='')
prepared_request = <PreparedRequest [HEAD]>
previous_fragment = ''
proxies    = OrderedDict()
purged_headers = ('Content-Length', 'Content-Type', 'Transfer-Encoding')
req        = <PreparedRequest [HEAD]>
resp       = <Response [302]>
rewindable = False
self       = <requests.sessions.Session object at 0x7fb25bfa30b8>
stream     = False
timeout    = 60
url        = '/en-US/credits/'
verify     = True
yield_requests = False

/venv/lib/python3.7/site-packages/requests/sessions.py:165: TooManyRedirects
_____________________ test_link_hreflang_tags[/-locales2] ______________________
[gw2] linux -- Python 3.7.3 /venv/bin/python

url = '/', locales = ('en-US', 'de', 'id')
base_url = 'https://bedrock-integration-tests.oregon-b.moz.works'

    @pytest.mark.headless
    @pytest.mark.nondestructive
    @pytest.mark.parametrize('url,locales', [
        ('/firefox/new/', ('en-US', 'de', 'id')),
        ('/firefox/', ('en-US', 'de', 'id')),
        ('/', ('en-US', 'de', 'id')),
    ])
    def test_link_hreflang_tags(url, locales, base_url):
        for locale in locales:
            full_url = '{}/{}{}'.format(base_url, locale, url)
            link_url = '{}/{}{}'.format('https://www.mozilla.org', locale, url)
            resp = requests.get(full_url, timeout=5)
>           assert LINK_TEMPLATE.format(url=link_url) in resp.content
E           TypeError: a bytes-like object is required, not 'str'

base_url   = 'https://bedrock-integration-tests.oregon-b.moz.works'
full_url   = 'https://bedrock-integration-tests.oregon-b.moz.works/en-US/'
link_url   = 'https://www.mozilla.org/en-US/'
locale     = 'en-US'
locales    = ('en-US', 'de', 'id')
resp       = <Response [200]>
url        = '/'

tests/functional/test_link_hreflang_tags.py:24: TypeError
------------------ generated xml file: /app/results/junit.xml ------------------
----------------- generated html file: /app/results/index.html -----------------
=========================== short test summary info ============================
FAILED tests/functional/test_generated_pages.py::test_generated_pages[https://bedrock-integration-tests.oregon-b.moz.works/about/forums/]
FAILED tests/functional/test_link_hreflang_tags.py::test_link_hreflang_tags[/firefox/-locales1]
FAILED tests/functional/test_link_hreflang_tags.py::test_link_hreflang_tags[/firefox/new/-locales0]
FAILED tests/functional/test_generated_pages.py::test_generated_pages[https://bedrock-integration-tests.oregon-b.moz.works/credits/]
FAILED tests/functional/test_link_hreflang_tags.py::test_link_hreflang_tags[/-locales2]
=============== 5 failed, 2263 passed, 10 rerun in 29.39 seconds ===============
INFO 2019-05-24 11:58:24,981 base Raven is not configured (logging is disabled). Please see the documentation for more information.
script returned exit code 1

jgmize · 2019-05-24T19:57:59Z

Sentry traceback from https://bedrock-integration-tests.oregon-b.moz.works/en-US/about/forums/
(See Response [500] in first headless test failure above):

UndefinedError: 'collections.OrderedDict object' has no attribute 'iteritems'
(16 additional frame(s) were not displayed)
...
  File "bedrock/mozorg/views.py", line 81, in forums_view
    return l10n_utils.render(request, 'mozorg/about/forums/forums.html', ctx)
  File "lib/l10n_utils/__init__.py", line 103, in render
    return django_render(request, template, context, **kwargs)
  File "django_jinja/backend.py", line 106, in render
    return mark_safe(self.template.render(context))

jgmize · 2019-05-24T20:04:27Z

The second headless test failure is due to a redirect loop:

$ curl -Is https://bedrock-demo-pr7196.oregon-b.moz.works/en-US/credits/ | egrep 'HTTP|Location'
HTTP/1.1 301 Moved Permanently
Location: /credits/
$ curl -Is https://bedrock-demo-pr7196.oregon-b.moz.works/credits/ | egrep 'HTTP|Location'
HTTP/1.1 302 Found
Location: /en-US/credits/

jgmize · 2019-05-24T20:44:34Z

Looking at the video for the selenium failures in the chrome tests for above, the AttributeError: 'bool' object has no attribute 'args' appears to be due to the pages not loading. However, https://bedrock-demo-pr7196.oregon-b.moz.works/en-US/contact/ and https://bedrock-demo-pr7196.oregon-b.moz.works/en-US/about/history/ (which are the equivalents of the URLs that were being tested) are working for me.

SmileyChris · 2019-05-31T04:45:00Z

@jgmize the two obvious fixes are done, try again and see if selenium is still a problem. I'm out until mid next week, so if it's an issue and you want to poke the selenium bear yourself then feel free 🐻

You can push to the LL branch to keep it as part of this PR.

jgmize · 2019-06-10T18:54:15Z

Unfortunately we are still getting some selenium errors:

==================================== ERRORS ====================================

_______________ ERROR at setup of test_spaces_mobile_navigation ________________

[gw3] linux -- Python 3.7.3 /venv/bin/python


request = <SubRequest 'selenium' for <Function test_spaces_mobile_navigation>>

selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="7b3f0702d06d4e0081802c45d1b69b98")>


    @pytest.fixture

    def selenium(request, selenium):

        viewport = VIEWPORT['desktop']

        if request.keywords.get('viewport') is not None:

>           viewport = VIEWPORT[request.keywords.get('viewport').args[0]]

E           AttributeError: 'bool' object has no attribute 'args'


request    = <SubRequest 'selenium' for <Function test_spaces_mobile_navigation>>

selenium   = <selenium.webdriver.remote.webdriver.WebDriver (session="7b3f0702d06d4e0081802c45d1b69b98")>

viewport   = {'height': 1024, 'width': 1280}


tests/functional/conftest.py:65: AttributeError

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:07:45,246 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:07:54,998 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:08:06,231 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

------------------------------- pytest-selenium --------------------------------

URL: data:,

SauceLabs Job: https://saucelabs.com/jobs/7b3f0702d06d4e0081802c45d1b69b98

_____________ ERROR at setup of test_communities_mobile_navigation _____________

[gw4] linux -- Python 3.7.3 /venv/bin/python


request = <SubRequest 'selenium' for <Function test_communities_mobile_navigation>>

selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="5f26a484c97a4dfbba29db77572afeb4")>


    @pytest.fixture

    def selenium(request, selenium):

        viewport = VIEWPORT['desktop']

        if request.keywords.get('viewport') is not None:

>           viewport = VIEWPORT[request.keywords.get('viewport').args[0]]

E           AttributeError: 'bool' object has no attribute 'args'


request    = <SubRequest 'selenium' for <Function test_communities_mobile_navigation>>

selenium   = <selenium.webdriver.remote.webdriver.WebDriver (session="5f26a484c97a4dfbba29db77572afeb4")>

viewport   = {'height': 1024, 'width': 1280}


tests/functional/conftest.py:65: AttributeError

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:07:52,599 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:08:02,124 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:08:12,305 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

------------------------------- pytest-selenium --------------------------------

URL: data:,

SauceLabs Job: https://saucelabs.com/jobs/5f26a484c97a4dfbba29db77572afeb4

____________________ ERROR at setup of test_list_displayed _____________________

[gw0] linux -- Python 3.7.3 /venv/bin/python


request = <SubRequest 'selenium' for <Function test_list_displayed>>

selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="9a8d6cd16cb74c11ac6d8cbcb7e78aea")>


    @pytest.fixture

    def selenium(request, selenium):

        viewport = VIEWPORT['desktop']

        if request.keywords.get('viewport') is not None:

>           viewport = VIEWPORT[request.keywords.get('viewport').args[0]]

E           AttributeError: 'bool' object has no attribute 'args'


request    = <SubRequest 'selenium' for <Function test_list_displayed>>

selenium   = <selenium.webdriver.remote.webdriver.WebDriver (session="9a8d6cd16cb74c11ac6d8cbcb7e78aea")>

viewport   = {'height': 1024, 'width': 1280}


tests/functional/conftest.py:65: AttributeError

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:08:18,050 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:08:30,152 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:08:40,820 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

------------------------------- pytest-selenium --------------------------------

URL: data:,

SauceLabs Job: https://saucelabs.com/jobs/9a8d6cd16cb74c11ac6d8cbcb7e78aea

___________________ ERROR at setup of test_mobile_navigation ___________________

[gw4] linux -- Python 3.7.3 /venv/bin/python


request = <SubRequest 'selenium' for <Function test_mobile_navigation>>

selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="bbe5b1028d6a4c4e859f506309ab2c78")>


    @pytest.fixture

    def selenium(request, selenium):

        viewport = VIEWPORT['desktop']

        if request.keywords.get('viewport') is not None:

>           viewport = VIEWPORT[request.keywords.get('viewport').args[0]]

E           AttributeError: 'bool' object has no attribute 'args'


request    = <SubRequest 'selenium' for <Function test_mobile_navigation>>

selenium   = <selenium.webdriver.remote.webdriver.WebDriver (session="bbe5b1028d6a4c4e859f506309ab2c78")>

viewport   = {'height': 1024, 'width': 1280}


tests/functional/conftest.py:65: AttributeError

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:09:26,788 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:09:37,182 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

---------------------------- Captured stdout setup -----------------------------

INFO 2019-05-24 12:09:46,563 pytest_selenium Capabilities from markers: {}

------------------------------ Captured log setup ------------------------------

pytest_selenium.py         115 INFO     Capabilities from markers: {}

------------------------------- pytest-selenium --------------------------------

URL: data:,

SauceLabs Job: https://saucelabs.com/jobs/bbe5b1028d6a4c4e859f506309ab2c78

------------------ generated xml file: /app/results/junit.xml ------------------

----------------- generated html file: /app/results/index.html -----------------

=========================== short test summary info ============================

SKIPPED [2] /app/tests/functional/conftest.py:59: Firefox Accounts CTA is displayed only to Firefox users

SKIPPED [1] /app/tests/functional/conftest.py:59: Non-Firefox user does not see Firefox state

SKIPPED [4] /app/tests/functional/conftest.py:59: Whatsnew pages are shown to Firefox only.

SKIPPED [1] /app/tests/functional/conftest.py:59: Create Account form is only displayed to Firefox users

ERROR tests/functional/test_contact.py::test_spaces_mobile_navigation

ERROR tests/functional/test_contact.py::test_communities_mobile_navigation

ERROR tests/functional/test_history.py::test_list_displayed

ERROR tests/functional/test_navigation.py::test_mobile_navigation

========== 102 passed, 8 skipped, 4 error, 8 rerun in 508.69 seconds ===========

INFO 2019-05-24 12:06:48,619 base Raven is not configured (logging is disabled). Please see the documentation for more information.

script returned exit code 1

jgmize · 2019-06-29T01:58:30Z

I rebased this branch on current master, resolved the merge conflicts, and force pushed to lincolnloop:django-2.2, triggering circleci #19657, and to mozilla:run-integration-tests, triggering jenkins build #329.

jgmize · 2019-06-29T23:25:14Z

After iterating over a bunch of rebase errors, replacing the broken usage of pytest.mark.viewport('mobile') in the selenium fixture with a separate selenium_mobile fixture, and some other miscellaneous cleanup, all of the CircleCI and Jenkins integration tests are now passing.

jgmize · 2019-07-01T21:30:04Z

I've rebased this branch again, resolved the merge conflicts, and pushed the results to the run-integration-tests branch triggering Jenkins #337. Assuming that all the tests are still passing after the latest rebase, I believe this will be ready to merge, but I would greatly appreciate a final once-over from @alexgibson or @pmac before we take this to production.

Edit: there was a failing CircleCI test after the last rebase, fixed in cd256f9, but Jenkins #337 passed.

…ally

alexgibson

I only took a look at the functional test changes, which all look good to me barring one small nit. Nice work, excited to see this land!

tests/functional/conftest.py

tests/functional/newsletter/test_newsletter_embed.py

pmac

I've got a few comments but nothing major to fix I don't think. Mostly some artifacts of automated 2to3 conversion with odd imports and from __future__ additions. It's worth getting this in and then cleaning up later IMHO. I'm going to pull this and run it locally, but the code changes are looking good.

pmac · 2019-07-01T13:46:11Z

Dockerfile

@@ -23,7 +23,7 @@ RUN gulp build --production
 ########
 # Python dependencies builder
 #
-FROM python:2-stretch AS python-builder
+FROM python:3-slim AS python-builder


I think we should pin a Python minor version just to be sure we have consistent builds. python:3.7-slim e.g..

pmac · 2019-07-01T13:47:37Z

Dockerfile

@@ -34,6 +34,7 @@ ENV PATH="/venv/bin:$PATH"
 COPY docker/bin/apt-install /usr/local/bin/
 RUN apt-install gettext build-essential libxml2-dev libxslt1-dev libxslt1.1

+RUN pip install virtualenv


With Python 3 venv is built in. You change these 2 lines to just RUN python -m venv /venv

bedrock/base/log_settings.py

bedrock/base/models.py

bedrock/base/tests/test_simple_dict_cache.py

bedrock/etc/urls.py

bedrock/events/models.py

bedrock/externalfiles/management/commands/update_externalfiles.py

pmac · 2019-07-02T19:04:22Z

bedrock/firefox/views.py

@@ -491,8 +489,6 @@ def get_template_names(self):
                template = 'firefox/developer/firstrun.html'
            else:
                template = 'firefox/dev-firstrun.html'
-        elif show_62_firstrun(version):


Was this meant to be included in this PR?

I removed this along the way because it appears to be redundant: the template matches the else clause below.

Ah. Yeah. Looks right. Just looked out of place in this PR. 👍

bedrock/mozorg/context_processors.py

pmac · 2019-07-02T21:06:15Z

Running locally seems to work well. Just did make build && make run and it's working for me. Nice work all 👍

SmileyChris · 2019-07-03T03:14:59Z

Since this is such a sweeping change, I thought it is probably a good time to fix the flake8 exceptions that had been chucked in ignore to deal with later :P

SmileyChris · 2019-07-03T03:58:55Z

Ok, I've made all the changes pertaining to actual code rather than ops or decisions over whether code should have been included (looks like that one might be something missed in the recent rebase?).

Let's chat tomorrow but I think we're pretty close to done now!

jgmize

I pushed the latest changes to the run-integration-tests branch one last time, giving us another green pipeline. Assuming @alexgibson agrees, I'd like to merge this and deploy to production in the morning.

alexgibson · 2019-07-03T06:36:52Z

If all seems well no objection here 👍

jgmize force-pushed the django-2.2 branch from 33e85cf to 9affad1 Compare June 29, 2019 01:51

jgmize force-pushed the django-2.2 branch from 65770c2 to 0520586 Compare June 29, 2019 21:53

jgmize requested review from pmac and alexgibson June 29, 2019 23:26

SmileyChris added 11 commits July 1, 2019 16:17

Complete removal of nose test artifacts

25da29d

Update base requirements to Django 2.2 and python 3 compatibile packages

c61c220

Fix print statements and django urlresolvers path

f2200e5

Remove old futures

f17f6e4

Futurize stage 1

a19a17c

Futurize stage 2

2514896

Fix renamed RegexURLResolver

9daf338

Update beautifulsoup4 for Py3 compatibility

a9ae672

Fix project name

818c375

Fix invalid unicode

da913cf

Fix simple cache backend

1333046

jgmize added 3 commits July 1, 2019 16:19

Fix more rebase errors

2756b0f

Remove unused FxVersionRedirectsMixin

1a76e70

Delete duplicate test with incorrect assertion

d6c5112

jgmize force-pushed the django-2.2 branch from 5253388 to d6c5112 Compare July 1, 2019 21:19

jgmize and others added 3 commits July 1, 2019 16:31

Fix another rebase error: prepend 'self.' to firefox_desktop

cd256f9

Fix issue with cached settings on a few tests when running pytest loc…

986c026

…ally

Fix a few more unescaped periods in regex strings

a099f0a

alexgibson requested changes Jul 2, 2019

View reviewed changes

tests/functional/conftest.py Show resolved Hide resolved

tests/functional/newsletter/test_newsletter_embed.py Outdated Show resolved Hide resolved

jgmize added 2 commits July 2, 2019 15:21

Remove deprecated smoke mark

f0a3501

Remove unnecessary pytest.param

eaaddd7

jgmize requested a review from alexgibson July 2, 2019 20:29

pmac approved these changes Jul 2, 2019

View reviewed changes

SmileyChris added 4 commits July 3, 2019 14:39

Tidy up some futurize cruft

f39dca0

Revert a test import change

9457926

Fix all previously ignored flake8 exceptions

fb12e4f

Fix a deprecated pytest yield test

7040330

SmileyChris added 4 commits July 3, 2019 15:17

Simplify raw object classes

0ba4ed3

Fix the futurize list(dict.values()) unnecessary changes

43e6e20

Fix flake8 formatting exception

1e9d391

Use force_text rather than manual isinstance

15737ac

jgmize approved these changes Jul 3, 2019

View reviewed changes

jgmize merged commit bf67e8b into mozilla:master Jul 3, 2019

alexgibson mentioned this pull request Jul 3, 2019

Bedrock local install fails for Django 2.2 #7368

Closed

alexgibson added the Roadmap label Jul 4, 2019

alexgibson added this to the Technical Roadmap Q3 2019 milestone Jul 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Django 2.2 #7196

Django 2.2 #7196

SmileyChris commented May 15, 2019

jgmize commented May 16, 2019 •

edited

Loading

jgmize commented May 16, 2019

jgmize commented May 17, 2019

jgmize commented May 17, 2019

jgmize commented May 17, 2019 •

edited

Loading

jgmize commented May 17, 2019

jgmize commented May 22, 2019

jgmize commented May 24, 2019 •

edited

Loading

jgmize commented May 24, 2019

jgmize commented May 24, 2019 •

edited

Loading

jgmize commented May 24, 2019 •

edited

Loading

SmileyChris commented May 31, 2019

jgmize commented Jun 10, 2019

jgmize commented Jun 29, 2019 •

edited

Loading

jgmize commented Jun 29, 2019 •

edited

Loading

jgmize commented Jul 1, 2019 •

edited

Loading

alexgibson left a comment

pmac left a comment

pmac Jul 1, 2019

pmac Jul 1, 2019

pmac Jul 2, 2019

jgmize Jul 3, 2019

pmac Jul 3, 2019

pmac commented Jul 2, 2019

SmileyChris commented Jul 3, 2019

SmileyChris commented Jul 3, 2019

jgmize left a comment

alexgibson commented Jul 3, 2019

Django 2.2 #7196

Django 2.2 #7196

Conversation

SmileyChris commented May 15, 2019

jgmize commented May 16, 2019 • edited Loading

jgmize commented May 16, 2019

jgmize commented May 17, 2019

jgmize commented May 17, 2019

jgmize commented May 17, 2019 • edited Loading

jgmize commented May 17, 2019

jgmize commented May 22, 2019

jgmize commented May 24, 2019 • edited Loading

Status update

Chrome tests

headless tests

jgmize commented May 24, 2019

jgmize commented May 24, 2019 • edited Loading

jgmize commented May 24, 2019 • edited Loading

SmileyChris commented May 31, 2019

jgmize commented Jun 10, 2019

jgmize commented Jun 29, 2019 • edited Loading

jgmize commented Jun 29, 2019 • edited Loading

jgmize commented Jul 1, 2019 • edited Loading

alexgibson left a comment

Choose a reason for hiding this comment

pmac left a comment

Choose a reason for hiding this comment

pmac Jul 1, 2019

Choose a reason for hiding this comment

pmac Jul 1, 2019

Choose a reason for hiding this comment

pmac Jul 2, 2019

Choose a reason for hiding this comment

jgmize Jul 3, 2019

Choose a reason for hiding this comment

pmac Jul 3, 2019

Choose a reason for hiding this comment

pmac commented Jul 2, 2019

SmileyChris commented Jul 3, 2019

SmileyChris commented Jul 3, 2019

jgmize left a comment

Choose a reason for hiding this comment

alexgibson commented Jul 3, 2019

jgmize commented May 16, 2019 •

edited

Loading

jgmize commented May 17, 2019 •

edited

Loading

jgmize commented May 24, 2019 •

edited

Loading

jgmize commented May 24, 2019 •

edited

Loading

jgmize commented May 24, 2019 •

edited

Loading

jgmize commented Jun 29, 2019 •

edited

Loading

jgmize commented Jun 29, 2019 •

edited

Loading

jgmize commented Jul 1, 2019 •

edited

Loading