Skip to content

Latest commit

 

History

History
614 lines (505 loc) · 29.5 KB

7.6.0.md

File metadata and controls

614 lines (505 loc) · 29.5 KB
Raw

Arches 7.6.0 Release Notes

Major enhancements

  • Introduces an interface for developing front-end features in Vue is now provided via the createVueApplication() function. A minimal example is available in the Arches Vue Integration Styleguide. The createVueApplication() function is experimental in 7.6. It currently registers all available PrimeVue services and directives, such as toast (error messaging) plugins, tooltip animations, and more. Over time, if some of these features do not see significant use in core Arches (or if registering them leads to a performance drag), they may be dropped from the createVueApplication() wrapper. Implementers may always register any needed plugins/services in their own Vue applications. (Note: The vast majority of PrimeVue's functionality does not require these services or directives.)

  • Deprecates yarn, replacing it with npm

  • Adds Vue internationalization (i18n)

  • Adds TypeScript support

  • Adds frontend testing and Python testing for projects

  • Adds Github actions to build applications, run tests, and compare coverage between branches for projects.

  • Re-architects search to be totally component-based, now governed by search-view components. See docs for more details. #10804

  • Improves term search when selecting a Term Match from the term search dropdown, the ensuing query will match on both the Term Value and the Nodegroup corresponding to the one in the context label. String search still works as expected, annotated explicitly with the translation "Contains Term: "

  • Adds a JSON-LD bulk import module #10862

  • Adds Bulk Concept Editor to bulk data manager, #11117

  • Adds subdomain routing, which allows projects to handle different subdomains separately within the same application (eg. api.my_arches_project.com, admin.my_arches_project.com, and www.my_arches_project.com)

  • Adds base-root.htm template, which can be inherited by plugin templates for display without the Arches header and side navigation.

  • Adds permission frameworks - including "standard" (current behavior and default) and "default deny" (future (8.x) default - all users are denied resource permissions until they are explicitly granted). This also gives developers the ability to implement their own custom permission framework.

  • Adds unlocalized string datatype #9191

Performance Improvements

  • Reduce queries for related objects when indexing resources #10453
  • Add index to the edit_log table's resourceinstanceid column to improve performance #11499

Additional improvements and bug fixes

  • Add frontend testing framework #10774
  • Move node_modules and webpack directories to root #10757
  • Combine templates for Arches project and applications and move several dotfiles to root #10558
  • Fix an issue where webpack receives multiple build calls when running in a container #10490
  • Handle missing ontologies in JSON-LD api response
  • Delete from custom indexes when bulk deleting resources #10872
  • Invalid API routes now respond with JSON instead of HTML #11166
  • Workflow history API: return 400 (instead of 401) for attempts to update completed workflows #10710
  • Fix whatisthis command #10083
  • Forbid malicious keys in node configs
  • Update Google Analytics syntax supporting Google Analytics 4 #11132
  • Filter out tiles created during resource creation from activity stream API #9768
  • Ensure resource creation edit log timestamps precede resource update timestamps #9769
  • Support Shapefile import in map editor widget #11339
  • Fix nuisance failing requests to help template view #11478
  • Adds Github action for comparing test coverage between branches and rejecting branches that lower test coverage #10738
  • Ensure search facets have widget label on advanced search restore #11046
  • Persist unsaved value in string widget if a user switches between cards in the card tree #11092
  • Fix workbench sidepanel header misalignment occuring during provisional edit review #11323
  • Add project-level testing and GitHub test runners #10842
  • Lenient file-type checking mode #10862
  • Allow overriding search_results view #10699
  • Styling fix in resource model manage menu #10911
  • Harden SystemSettings model against too-early access #11118 [#11118]
  • Upgrade openpyxl package to 3.1.2 and fixes ETL modules #10726
  • Implement arches version compatibility check as a Django system check #11114
  • Fix internationalized string/json field entry problems in the Django admin #10597
  • Fix text widget dirty state calculation for newly created nodes #10873
  • Search Export: data exportable as "system values" (e.g. concept valueids) instead of "display values" (e.g. string preflabel) #10787
  • Prevent language code duplication during concept import; avoid creating Language objects other than English in new projects #10121
  • Add pre-commit hooks for formatting #10575
  • Graph.delete_instances() now uses BulkDataDeletion #10781
  • Return resourceid in ActivityStream rather than resource url #10665
  • Move arches application declaration to AppConfig #11009
  • Move the i18n/ url outside of the i18n_patterns function #10754
  • Fix hard coded Polygon geometry type in geo_utils.convert_multipart_to_singlepart #11218
  • Fix geocoder placeholder #11173
  • Fix error preventing user from changing a node's widget #10899
  • Raise limit of editable installs of arches applications from 3 to 9 #11274
  • Allow minzoom and maxzoom to be applied to a resource map source from a geojson node config #10929
  • Avoid retrieving full user information in the bulk data manager task status page.
  • Allow arbitrarily large geometries with > 1500 points and an ES document > 32k bytes #11080
  • Allow custom ES search documents to be added to a resource instance #10987
  • Ensure Saved Searches sortorder is used when called from system settings #11044
  • Fix child cards to use own card component instead of parents in report #11431
  • Add language support to spatial views and improved testing. #10705
  • Add an API for managing spatial views. #10704
  • Fix the issue with a geojson geometry not being refreshed when the node value is updated, #11029
  • Fix importing custom datatypes from Arches Applications #11233
  • Allow proper discovery of frontend applications #11253
  • Add the ability to search if files do/do not exist, search on file size #11179
  • Add the ability to search if a geometry exists/does not exist. Search on geometry type #11180
  • Add capability to use search-url in bulk exporters, #11015
  • Add capability to match the csv column names to the nodes in single csv importer, #10780
  • Add ability to execute bulk data manager modules from the command line, #10476

Dependency changes

System:
    Added:
        GNU gettext == 0.22.4

    Upgraded:
        NodeJS == 20.14.0
        Postgres: 14+

    Removed:
        yarn

Python:
    Upgraded:
        Django == 4.2.16 (or <5.0.0)
        openpyxl == 3.0.10
        filetype == 1.2.0

    Added:
        django-hosts
        tomli (on Python 3.10 only) == 2.0.1
        (dev dependencies):
            pre-commit
            black

    Removed:
        django_compressor
        django-libsass
        mapbox-vector-tile

        (dev dependencies):
            sauceclient

JavaScript:
    Upgraded:
        babel-loader == 9.1.3
        ckeditor4 == full/4.24.x
        copy-webpack-plugin == 12.0.2
        cross-fetch == 4.0.0
        css-loader == 7.1.1
        d3 == ^7.9.0
        datatables.net-bs == 1.13.11
        datatables.net-buttons == 2.4.3
        datatables.net-buttons-bs == 2.4.3
        datatables.net-responsive == 2.5.1
        datatables.net-responsive-bs == 2.5.1
        eslint == 9.0.0
        html-loader == 5.0.0
        @mapbox/geojsonhint == 3.3.0
        @mapbox/mapbox-gl-draw == 1.4.3
        moment-timezone == 0.5.45
        normalize.css ^8.0.1
        postcss-loader == 8.1.1
        sass-loader == 14.2.0
        style-loader == 4.0.0
        stylelint == 16.3.1
        stylelint-config-standard == 36.0.0
        stylelint-webpack-plugin == 5.0.0
        @turf/turf == 6.5.0
        vue == 3.5.8
        webpack-bundle-tracker == 3.1.0
        webpack-cli == 5.1.4
        webpack-dev-server == 5.0.4

    Added:
        @babel/plugin-transform-class-properties == 7.23.3
        @primevue/themes == 4.0.7
        @typescript-eslint/eslint-plugin == 6.18.1
        @typescript-eslint/parser == 6.18.1
        eslint-config-prettier == 9.1.0
        eslint-plugin-vue == 9.20.0
        prettier == 3.3.0
        primevue == 4.0.7
        nodemon == 3.0.2
        sass == 1.70.0
        ts-loader == 9.5.1
        typescript == 5.4.5
        typescript-eslint == 7.7.0
        vitest == 1.5.0
        vue3-gettext == 3.0.0-beta.6
        vue-tsc == 2.0.13

    Removed:
        @babel/plugin-proposal-class-properties
        eslint-webpack-plugin
        node-sass
        postcss-preset-env
        moment-timezone

Breaking changes

  • The minimum supported version of Python is now 3.10. Python 3.11 is encouraged, as it is significantly faster.
  • The minimum supported version of Node.js is now 20.14.0.
  • ESLint has been updated to 9.0.0, and requires configuration via eslint.config.mjs rather than .eslintrc
  • As part of the changes to allow for new permissions frameworks, the method signature for get_nodegroups_by_perm has changed - instead of returning a set of NodeGroups, it now returns a list of UUIDs.

Minor incompatibilities:

  • The signature of Graph.delete_instances changed from: 
        def delete_instances(self, verbose=False):
    to: 
        def delete_instances(self, userid=None, verbose=False):
  • The signature of arches.app.utils.module_importer.get_class_from_modulename changed from: 
        def get_class_from_modulename(modulename, classname, directory_list)
    to: 
        def get_class_from_modulename(modulename, classname, extension_type: ExtensionType)
  • BaseImportModule.cumulative_excel_files_size became BaseImportModule.cumulative_files_size (and a similar change was made to the JSON return value of BaseImportModule.read.)
  • BaseImportModule.stage_files is now abstract. (It used to call stage_excel_files(), which was only defined on certain subclasses.) stage_files() is now a more attractive target for overriding than run_load_task().
  • GeojsonFeatureCollectionDataType moved from arches.app.datatypes.datatypes to arches.app.datatypes.core.
  • unzip_file() moved from arches.setup to arches.app.utils.zip
  • Version-related utils moved from arches.setup to arches.version
  • add_to_update_fields() moved from arches.app.models.models to arches.app.models.utils.
  • arches.app.utils.compatibility was removed in favor of a Django system check.

Deprecations

  • Boolean values for the FILE_TYPE_CHECKING setting are deprecated. Starting with Arches 8.0, the allowed values will be:

    • None
    • "lenient" (new in 7.6)
    • "strict"

    For more, see the documentation for this setting.

  • As a consequence of migrating from setup.py to pyproject.toml, the -o install argument to manage.py packages (previously used to execute a project's setup.py) does nothing and has been deprecated. In Arches 8.0, it will raise an exception.

  • arches-project create is deprecated and will be removed in a future version. The replacement is arches-admin startproject, which more closely follows the Django syntax.

Upgrading Arches

  1. You must be upgraded to at least version 7.5.0 before proceeding. If you are on an earlier version, please refer to the upgrade process in the Version 7.5.0 release notes

  2. Be sure to backup your database before proceeding.

  3. Upgrade to Arches 7.6.0

    pip install --upgrade arches==7.6.0

  4. Uninstall removed Python dependencies:

    pip uninstall django_compressor
pip uninstall django-libsass
pip uninstall mapbox-vector-tile

  5. Uninstall removed system dependencies (optional):

    • yarn
      • This will differ depending on how yarn was installed.
        • brew: brew uninstall yarn
        • tarball: rm -rf "$HOME/.yarn"
        • npm: npm uninstall -g yarn
        • ubuntu: sudo apt-get remove yarn && sudo apt-get purge yarn
        • centos: yum remove yarn
        • windows: choco uninstall yarn

Upgrading an Arches project

  1. Follow instructions to upgrade each Arches project database to Postgres 14 or higher. https://www.postgresql.org/docs/current/upgrading.html

  2. If you have made customizations to files in your webpack directory, backup that directory as those files will be overwritten in the following steps. Read this for more information.

  3. Add GNU gettext to your system:

    1. If using Ubuntu/Linux:

      sudo apt-get update
sudo apt-get install gettext

    2. If using macOS:

      brew install gettext
brew link --force gettext

    3. If using Windows:

      You can run the scripts and install gettext under WSL2 like you would with regular Ubuntu (recommended) or install gettext via mingw64 or cygwin. You may also find precompiled binaries here

  4. Update Node.js to v20.14.0 -- the latest LTS version.

  5. Move package.json up a directory level, it should be in the same directory as manage.py.

  6. Update package.json:

    1. Add a license field 
      "license": "AGPL-3.0-only",
    2. Update the value of scripts: 
      "scripts": {
    "build_development": "npm run eslint:check && npm run ts:check && cross-env NODE_OPTIONS=--max-old-space-size=2048 webpack --config ./webpack/webpack.config.dev.js",
    "build_production": "npm run eslint:check && npm run ts:check && cross-env NODE_OPTIONS=--max-old-space-size=2048 NODE_ENV=production webpack --config ./webpack/webpack.config.prod.js",
    "build_test": "npm run eslint:check && npm run ts:check && cross-env NODE_OPTIONS=--max-old-space-size=2048 webpack --config ./webpack/webpack.config.dev.js --env test=true",
    "eslint:check": "eslint **/src",
    "eslint:fix": "eslint **/src --fix",
    "eslint:watch": "nodemon --watch . --ext ts,vue --exec npm run --silent eslint:check",
    "gettext:extract": "vue-gettext-extract",
    "gettext:compile": "vue-gettext-compile",
    "prettier:check": "prettier {{ project_name }}/src --check",
    "prettier:fix": "prettier {{ project_name }}/src --write",
    "ts:check": "vue-tsc --noEmit",
    "ts:watch": "vue-tsc --watch --noEmit",
    "start": "cross-env NODE_OPTIONS=--max-old-space-size=2048 webpack serve --config ./webpack/webpack.config.dev.js",
    "vitest": "vitest --run --coverage"
},
    3. Add and populate an overrides section: 
      "overrides": {
    "moment-timezone": "^0.5.45",
    "nomnom": "npm:@gerhobbelt/nomnom",
    "rimraf": "^5.0.7",
    "underscore": "^1.13.6"
},
    4. Update the dependency references to stable/7.6.0: 
      "dependencies": {
    "arches": "archesproject/arches#stable/7.6.0"
},
"devDependencies": {
    "arches-dev-dependencies": "archesproject/arches-dev-dependencies#stable/7.6.0"
},
    5. If you are using any relative dependency pathing ( eg. file:./path/to/local/dependency ) it must be removed because of the hoisting differences between yarn and npm. It should instead point towards a repository or npm package.

  7. Any yarn references will need to be updated to use npm instead. For instance, if you have any scripts calling yarn install or yarn build_development, they will need to be updated to npm install and npm run build_development.

  8. If your project has any CSS files with import statements, update the pattern. For example:

    @import '../node_modules/my-frontend-dependency/style.min.css';

    would be changed to:

    @import url(node_modules/my-frontend-dependency/style.min.css);

  9. Update .gitignore:

    1. Change {project_name}/media/node_modules to node_modules
    2. Change {project_name}/webpack/webpack-stats.json to webpack/webpack-stats.json
    3. If it exists, remove {project_name}/webpack/webpack-user-config.js
    4. Add the following: 
      *.egg-info
.DS_STORE
.tsconfig-paths.json
.frontend-configuration-settings.json
coverage/

  10. In settings.py:

    1. In INSTALLED_APPS:

      • remove "compressor"
      • add "django_hosts"
      • ensure your project is listed first (or at least before any other Arches Applications)

    2. After INSTALLED_APPS, add this line to ensure "arches.app" (provider of core arches templates) always follows any Arches Applications:

      INSTALLED_APPS += ("arches.app",)

    3. Add the following lines, replacing {{ project_name }} with the name of your project:

      ROOT_HOSTCONF = "{{ project_name }}.hosts"
DEFAULT_HOST = "{{ project_name }}"

    4. In MIDDLEWARE, uncomment django.middleware.clickjacking.XFrameOptionsMiddleware. (If you serve Arches content in a frame, configure the X_FRAME_OPTIONS setting.)

    5. Add the following lines after MIDDLEWARE:

      MIDDLEWARE.insert(  # this must resolve to first MIDDLEWARE entry
    0, 
    "django_hosts.middleware.HostsRequestMiddleware"
)

MIDDLEWARE.append(  # this must resolve last MIDDLEWARE entry
    "django_hosts.middleware.HostsResponseMiddleware"
)

    6. Update WEBPACK_LOADER to the following:

      WEBPACK_LOADER = {
    "DEFAULT": {
        "STATS_FILE": os.path.join(APP_ROOT, '..', 'webpack/webpack-stats.json'),
    },
}

    7. Update LOCALE_PATHS.append(...) to LOCALE_PATHS.insert(0, ...)

      LOCALE_PATHS.insert(0, os.path.join(APP_ROOT, 'locale'))

    8. Add "json" to FILE_TYPES if you wish to use the JSON-LD bulk import module.

    9. Remove the ARCHES_APPLICATIONS setting.

    10. Remove MIN_ARCHES_VERSION and MAX_ARCHES_VERSION. (If you had values other than None here, make a note of these values for use when pyproject.toml is discussed below.)

    11. Replace:

      STATICFILES_DIRS = build_staticfiles_dirs(
    root_dir=ROOT_DIR,
    app_root=APP_ROOT,
    arches_applications=ARCHES_APPLICATIONS,
)

      with:

      STATICFILES_DIRS = build_staticfiles_dirs(app_root=APP_ROOT)

    12. Replace:

      TEMPLATES = build_templates_config(
    root_dir=ROOT_DIR,
    debug=DEBUG,
    app_root=APP_ROOT,
    arches_applications=ARCHES_APPLICATIONS,
)

      with:

      TEMPLATES = build_templates_config(
    debug=DEBUG,
    app_root=APP_ROOT,
)

    13. Remove the following block:

      if __name__ == "__main__":
    transmit_webpack_django_config(
        root_dir=ROOT_DIR,
        app_root=APP_ROOT,
        arches_applications=ARCHES_APPLICATIONS,
        public_server_address=PUBLIC_SERVER_ADDRESS,
        static_url=STATIC_URL,
        webpack_development_server_port=WEBPACK_DEVELOPMENT_SERVER_PORT,
    )

  11. Update urls.py:

    from django.conf import settings
from django.conf.urls.static import static
from django.conf.urls.i18n import i18n_patterns
from django.urls import include, path

urlpatterns = [
    # project-level urls
]

# Ensure Arches core urls are superseded by project-level urls
urlpatterns.append(path('', include('arches.urls')))

# Adds URL pattern to serve media files during development
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

# Only handle i18n routing in active project. This will still handle the routes provided by Arches core and Arches applications,
# but handling i18n routes in multiple places causes application errors.
if settings.ROOT_URLCONF == __name__:
    if settings.SHOW_LANGUAGE_SWITCH is True:
        urlpatterns = i18n_patterns(*urlpatterns)

    urlpatterns.append(path("i18n/", include("django.conf.urls.i18n")))

  12. If you have updated templates/javascript.htm in your project with any project-level urls, they must be moved to templates/arches_urls.htm. All other updates to templates/javascript.htm should remain in-place. eg:

    <!-- my_project/templates/javascript.htm -->

{% extends "javascript.htm" %}

{% load static %}
{% load l10n %}
{% load i18n %}


{% block arches_urls %}
{{ block.super }}
<div class="arches-urls"
    my_project_url = "{% url 'my-project-url' %}"
></div>
{% endblock arches_urls %}

{% block arches_translations %}
{{ block.super }}
<div class='arches-translations'
    my-project-translation-string='{% trans "My Project Translation String" as myProjectTranslationString %}  "{{ myProjectTranslationString|escapejs }}"'
></div>
{% endblock arches_translations %}

    Should now be:

    <!-- my_project/templates/arches_urls.htm -->

{% extends "arches_urls.htm" %}

{% load static %}
{% load l10n %}
{% load i18n %}

{% block arches_urls %}
{{ block.super }}
<div class="arches-urls"
    my_project_url = "{% url 'my-project-url' %}"
></div>
{% endblock arches_urls %}

    With any arches_urls block taken out of javascript.htm:

    <!-- my_project/templates/javascript.htm -->

{% extends "javascript.htm" %}

{% load static %}
{% load l10n %}
{% load i18n %}

{% block arches_translations %}
{{ block.super }}
<div class='arches-translations'
    my-project-translation-string='{% trans "My Project Translation String" as myProjectTranslationString %}  "{{ myProjectTranslationString|escapejs }}"'
></div>
{% endblock arches_translations %}

  13. Within your project, with your Python 3 virtual environment activated run:

    python manage.py migrate

  14. Then run:

    python manage.py updateproject
    • Pay attention to notices about existing conflicting files that need manual reconciliation with the updated project template, if any. Some templates may have placeholders with {{ project_name }} or {{ project_name_title_case }} that need to be replaced with your actual project name.

  15. At the project root, you now have a pyproject.toml file containing your project's metadata. If you had any runtime dependencies other than arches in my_project/install/requirements.txt (or any development dependencies in my_project/install/requirements_dev.txt), move them to the corresponding dependency blocks in pyproject.toml. The core arches (runtime) dependency is already specified for you, as well as some development dependencies needed for the new GitHub Actions templates in this version.

    • If you had values other than None in MIN_ARCHES_VERSION and MAX_ARCHES_VERSION in settings.py, update the core arches dependency in the dependencies block of pyproject.toml.

  16. If you do any custom development, update and reinstall the python development dependencies, and install the pre-commit hooks. (You can always ignore these hooks with git commit --no-verify.)

    1. Navigate to the directory with your project's .pre-commit-config.yaml file. 
      pip install -e '.[dev]'
pre-commit install

  17. Navigate to the directory with your project's package.json file. Then run:

    npm install

  18. Start your application server in a separate terminal if it's not already running.

  19. In the same terminal window where you ran npm ( on the same level as package.json ):

    1. Run npm run gettext:extract followed by npm run gettext:compile. This will generate i18n files in your locale directory. Even if you're not planning on internationalizing your project, it's important to have these files for creating Vue components.

    2. Run npm start or npm run build_development. This will generate your media/build directory.

      • If running your project in development:
      • npm start will build the frontend of the application and then start a webpack development server
      • npm run build_development will build a development bundle for the frontend assests of the application -- this should complete in less than 2 minutes
      • If running your project in production:
      • npm run build_production This builds a production bundle. takes up to 2hrs depending on resources
      • Alternatively you can run python manage.py build_production. This will create a production bundle of frontend assets and also call collectstatic.

  20. Test your project with deprecation warnings enabled.

    python -Wall::DeprecationWarning manage.py test tests --settings=tests.test_settings

  21. Run system checks against your production settings.

    python manage.py check --deploy --settings=path.to.production.settings

  22. Be sure that your web server has write access to the following files:

    ${project_name}/.frontend-configuration-settings.json
${project_name}/.tsconfig-paths.json

  23. If you are running Arches on Apache, be sure to run:

    collectstatic

    and restart your server

    sudo service apache2 reload