Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CKAN 2.9 / Python 3 #55

Merged
merged 33 commits into from
Jul 28, 2022
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
3f19f96
Remove Travis conf files
amercader Sep 9, 2021
3d5d935
Enable github actions for tests
amercader Sep 9, 2021
a9190d9
Fix flake8 tests
amercader Sep 9, 2021
30e90a7
Some pytest changes
amercader Sep 10, 2021
9989928
First py3 fixes, blueprint
amercader Sep 14, 2021
0c9d039
Fix logic tests
amercader Sep 14, 2021
874352e
Fanstatic to webassets
amercader Sep 16, 2021
1a0fb27
Multiple fixes and updates for py3
amercader Sep 16, 2021
0f1d4ec
Update interface tests
amercader Sep 16, 2021
24c09c8
lint
amercader Sep 16, 2021
171ae27
Load plugins in logic tests
amercader Sep 16, 2021
1f09a82
Remove asserts
amercader Sep 17, 2021
c464350
Fix blueprint
amercader Sep 17, 2021
79cf7d3
db init cli
amercader Sep 17, 2021
9196a62
bump
amercader Jul 25, 2022
5bf7c55
[QOL-7740] convert test dates to strings
ThrawnCA Dec 8, 2021
1d91832
Fix pytest fixtures in plugin tests
amercader Jul 26, 2022
5c5a656
More pytest fixes
amercader Jul 26, 2022
ef74d77
More pytest fixes
amercader Jul 27, 2022
201ce1b
More verbosity
amercader Jul 27, 2022
91c72e1
py2/py3 pyfakefs requirement
amercader Jul 27, 2022
bf27702
Install proper requeriments depending on py version
amercader Jul 27, 2022
e0502b0
Fix upload tests for 2.9-py2
amercader Jul 27, 2022
1a9af61
Drop CKAN 2.7 and 2.8 support
amercader Jul 27, 2022
45ea928
Ignore session param in mock call
amercader Jul 28, 2022
175568f
Assert individual call args in mock call
amercader Jul 28, 2022
219d570
Cross-version post calls
amercader Jul 28, 2022
c087ce4
Cross-version upload handling
amercader Jul 28, 2022
f7e8ed8
Reenable 2.8 tests
amercader Jul 28, 2022
bc46967
Bump six requirement
amercader Jul 28, 2022
77576e6
Merge branch 'py3-28-tests' into py3
amercader Jul 28, 2022
9425f0e
Update supported versions in README
amercader Jul 28, 2022
9b44ee9
Update supported py versions in setup.py
amercader Jul 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
name: Tests
on: [push, pull_request]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: '3.6'
- name: Install requirements
run: pip install flake8 pycodestyle
- name: Check syntax
run: flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics --exclude ckan

test:
needs: lint
strategy:
matrix:
ckan-version: [2.9, 2.9-py2, 2.8, 2.7]
fail-fast: false

name: CKAN ${{ matrix.ckan-version }}
runs-on: ubuntu-latest
container:
image: openknowledge/ckan-dev:${{ matrix.ckan-version }}
services:
solr:
image: ckan/ckan-solr-dev:${{ matrix.ckan-version }}
postgres:
image: ckan/ckan-postgres-dev:${{ matrix.ckan-version }}
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
redis:
image: redis:3
env:
CKAN_SQLALCHEMY_URL: postgresql://ckan_default:pass@postgres/ckan_test
CKAN_DATASTORE_WRITE_URL: postgresql://datastore_write:pass@postgres/datastore_test
CKAN_DATASTORE_READ_URL: postgresql://datastore_read:pass@postgres/datastore_test
CKAN_SOLR_URL: http://solr:8983/solr/ckan
CKAN_REDIS_URL: redis://redis:6379/1

steps:
- uses: actions/checkout@v2
- name: Install requirements
run: |
pip install -r requirements.txt
pip install -r dev-requirements.txt
pip install -e .
# Replace default path to CKAN core config file with the one on the container
sed -i -e 's/use = config:.*/use = config:\/srv\/app\/src\/ckan\/test-core.ini/' test.ini
- name: Setup extension (CKAN >= 2.9)
if: ${{ matrix.ckan-version != '2.7' && matrix.ckan-version != '2.8' }}
run: |
ckan -c test.ini db init
- name: Setup extension (CKAN < 2.9)
if: ${{ matrix.ckan-version == '2.7' || matrix.ckan-version == '2.8' }}
run: |
paster --plugin=ckan db init -c test.ini
- name: Run tests
run: pytest --ckan-ini=test.ini --cov=ckanext.validation --cov-report=xml --cov-append --disable-warnings ckanext/validation/tests

- name: Upload coverage report to codecov
uses: codecov/codecov-action@v1
with:
file: ./coverage.xml
70 changes: 0 additions & 70 deletions bin/travis-build.bash

This file was deleted.

14 changes: 0 additions & 14 deletions bin/travis-run.sh

This file was deleted.

46 changes: 46 additions & 0 deletions ckanext/validation/blueprints.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# encoding: utf-8

from flask import Blueprint

from ckantoolkit import c, NotAuthorized, ObjectNotFound, abort, _, render, get_action

validation = Blueprint("validation", __name__)


def read(id, resource_id):

try:
validation = get_action(u"resource_validation_show")(
{u"user": c.user}, {u"resource_id": resource_id}
)

resource = get_action(u"resource_show")({u"user": c.user}, {u"id": resource_id})

dataset = get_action(u"package_show")(
{u"user": c.user}, {u"id": resource[u"package_id"]}
)

# Needed for core resource templates
c.package = c.pkg_dict = dataset
c.resource = resource

return render(
u"validation/validation_read.html",
extra_vars={
u"validation": validation,
u"resource": resource,
u"dataset": dataset,
u"pkg_dict": dataset,
},
)

except NotAuthorized:
abort(403, _(u"Unauthorized to read this validation report"))
except ObjectNotFound:

abort(404, _(u"No validation report exists for this resource"))


validation.add_url_rule(
"/dataset/<id>/resource/<resource_id>/validation", view_func=read
)
22 changes: 22 additions & 0 deletions ckanext/validation/cli.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import sys

import click

from ckanext.validation.model import create_tables, tables_exist


@click.group()
def validation():
"""Harvests remotely mastered metadata."""
pass


@validation.command()
def init_db():
"""Creates the necessary tables in the database."""
if tables_exist():
print(u"Validation tables already exist")
sys.exit(0)

create_tables()
print(u"Validation tables created")
6 changes: 5 additions & 1 deletion ckanext/validation/helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import json

from ckan.lib.helpers import url_for_static
from ckantoolkit import url_for, _, config, asbool, literal
from ckantoolkit import url_for, _, config, asbool, literal, h


def get_validation_badge(resource, in_listing=False):
Expand Down Expand Up @@ -89,3 +89,7 @@ def bootstrap_version():
return '3'
else:
return '2'


def use_webassets():
return int(h.ckan_version().split('.')[1]) >= 9
5 changes: 3 additions & 2 deletions ckanext/validation/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import json
import re

import six
import requests
from sqlalchemy.orm.exc import NoResultFound
from goodtables import validate
Expand Down Expand Up @@ -45,7 +46,7 @@ def run_validation_job(resource):
options = {}

resource_options = resource.get(u'validation_options')
if resource_options and isinstance(resource_options, basestring):
if resource_options and isinstance(resource_options, six.string_types):
resource_options = json.loads(resource_options)
if resource_options:
options.update(resource_options)
Expand Down Expand Up @@ -77,7 +78,7 @@ def run_validation_job(resource):
source = resource[u'url']

schema = resource.get(u'schema')
if schema and isinstance(schema, basestring):
if schema and isinstance(schema, six.string_types):
if schema.startswith('http'):
r = requests.get(schema)
schema = r.json()
Expand Down
10 changes: 5 additions & 5 deletions ckanext/validation/logic.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import logging
import json

import six
from sqlalchemy.orm.exc import NoResultFound

import ckan.plugins as plugins
Expand Down Expand Up @@ -262,14 +263,14 @@ def resource_validation_run_batch(context, data_dict):
count_resources = 0

dataset_ids = data_dict.get('dataset_ids')
if isinstance(dataset_ids, basestring):
if isinstance(dataset_ids, six.string_types):
try:
dataset_ids = json.loads(dataset_ids)
except ValueError as e:
dataset_ids = [dataset_ids]

search_params = data_dict.get('query')
if isinstance(search_params, basestring):
if isinstance(search_params, six.string_types):
try:
search_params = json.loads(search_params)
except ValueError as e:
Expand Down Expand Up @@ -467,7 +468,7 @@ def resource_create(context, data_dict):
context['use_cache'] = False
t.get_action('package_update')(context, pkg_dict)
context.pop('defer_commit')
except t.ValidationError, e:
except t.ValidationError as e:
try:
raise t.ValidationError(e.error_dict['resources'][-1])
except (KeyError, IndexError):
Expand All @@ -480,7 +481,6 @@ def resource_create(context, data_dict):
uploader.get_max_resource_size())

# Custom code starts

if get_create_mode_from_config() == u'sync':

run_validation = True
Expand Down Expand Up @@ -588,7 +588,7 @@ def resource_update(context, data_dict):
context['use_cache'] = False
updated_pkg_dict = t.get_action('package_update')(context, pkg_dict)
context.pop('defer_commit')
except t.ValidationError, e:
except t.ValidationError as e:
try:
raise t.ValidationError(e.error_dict['resources'][-1])
except (KeyError, IndexError):
Expand Down
3 changes: 2 additions & 1 deletion ckanext/validation/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import uuid
import logging

import six
from sqlalchemy import Column, Unicode, DateTime
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.dialects.postgresql import JSON
Expand All @@ -14,7 +15,7 @@


def make_uuid():
return unicode(uuid.uuid4())
return six.text_type(uuid.uuid4())


Base = declarative_base(metadata=metadata)
Expand Down
Loading