Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combine synchronized storing and indexing, re-organize components, HAL support (no more DOI centric approach), dependency update, import format update #92

Merged
merged 36 commits into from
Apr 28, 2024
Merged

Combine synchronized storing and indexing, re-organize components, HAL support (no more DOI centric approach), dependency update, import format update #92

merged 36 commits into from
Apr 28, 2024

Commits on Nov 2, 2021

  1. Update Dockerfile

    Marcin Kardas committed Nov 2, 2021
    Configuration menu
    Copy the full SHA
    41b3f48 View commit details
    Browse the repository at this point in the history

  2. Prevent glutton from hanging for two minutes

    Marcin Kardas committed Nov 2, 2021
    Configuration menu
    Copy the full SHA
    fd74782 View commit details
    Browse the repository at this point in the history

  3. Add max number of connections to ES to config 

    By default the maximum number of connections per route is set to
[DEFAULT_MAX_CONN_PER_ROUTE](https://javadoc.io/static/org.elasticsearch.client/elasticsearch-rest-client/7.4.2/org/elasticsearch/client/RestClientBuilder.html#DEFAULT_MAX_CONN_PER_ROUTE) which is 10. I additionally set the maximum number
of connections for all routes in total to the same value (default is
[DEFAULT_MAX_CONN_TOTAL](https://javadoc.io/static/org.elasticsearch.client/elasticsearch-rest-client/7.4.2/org/elasticsearch/client/RestClientBuilder.html#DEFAULT_MAX_CONN_TOTAL) = 30). I don't think it makes sense to expose this parameter as well
in the config file as glutton uses only a single route.
    Marcin Kardas committed Nov 2, 2021
    Configuration menu
    Copy the full SHA
    6a8a2d1 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2022

  1. preparing working branch for version 0.3

    @kermitt2
    kermitt2 committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    e8d4ca5 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #68 from mkardas/incremental-update 

    Improve parallel processing
    @kermitt2
    kermitt2 authored Apr 12, 2022
    Configuration menu
    Copy the full SHA
    2c2b637 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2022

  1. fix dependency

    @kermitt2
    kermitt2 committed Apr 13, 2022
    Configuration menu
    Copy the full SHA
    89eba9c View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2022

  1. complete merge

    @kermitt2
    kermitt2 committed Apr 23, 2022
    Configuration menu
    Copy the full SHA
    9b88fcb View commit details
    Browse the repository at this point in the history

  2. avoid exploding the log size when something is going bad with the loa… 

    …ding
    @kermitt2
    kermitt2 committed Apr 23, 2022
    Configuration menu
    Copy the full SHA
    8e17030 View commit details
    Browse the repository at this point in the history

  3. increase the lmdb map size to be safe

    @kermitt2
    kermitt2 committed Apr 23, 2022
    Configuration menu
    Copy the full SHA
    55af8b6 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2022

  1. review Readme

    @kermitt2
    kermitt2 committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    626cca5 View commit details
    Browse the repository at this point in the history

  2. review Readme

    @kermitt2
    kermitt2 committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    a61eac6 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2023

  1. update crossref dump torrent version

    @kermitt2
    kermitt2 committed May 3, 2023
    Configuration menu
    Copy the full SHA
    a9a15ef View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2023

  1. some fix for parsing medline/pubmed recent xml

    @kermitt2
    kermitt2 committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    8ecb21e View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2023

  1. fix other medline import errors

    @kermitt2
    kermitt2 committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    35f7762 View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2023

  1. Big Bang change

    @kermitt2
    kermitt2 committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    e42a127 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2024

  1. extending pmid, various fixes

    @kermitt2
    kermitt2 committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    9641a59 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2024

  1. review resources

    @kermitt2
    kermitt2 committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    d2f6a99 View commit details
    Browse the repository at this point in the history

  2. complete oai-pmh and tei parser

    @kermitt2
    kermitt2 committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    9936337 View commit details
    Browse the repository at this point in the history

  3. use logback

    @kermitt2
    kermitt2 committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    eeec3d4 View commit details
    Browse the repository at this point in the history

  4. add clean gradle tasks, document

    @kermitt2
    kermitt2 committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    fcd4413 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. add indexer; update dependencies

    @kermitt2
    kermitt2 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    54345b0 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. add async indexing; fix some mapping; implement HAL API harvester ins… 

    …tead of OAI-PMH
    @kermitt2
    kermitt2 committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    f6446e9 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2024

  1. hal id injection via doi; some fix on server side and dependencies

    @kermitt2
    kermitt2 committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    46e702d View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2024

  1. review hal id injection

    @kermitt2
    kermitt2 committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    346e80d View commit details
    Browse the repository at this point in the history

  2. fix unused compression for one map

    @kermitt2
    kermitt2 committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    1d2b7b5 View commit details
    Browse the repository at this point in the history

  3. update es client

    @kermitt2
    kermitt2 committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    4a5fa5b View commit details
    Browse the repository at this point in the history

  4. review crossref combined storing+indexing

    @kermitt2
    kermitt2 committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    d1554a5 View commit details
    Browse the repository at this point in the history

  5. review doc, with proper readthedocs

    @kermitt2
    kermitt2 committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    96ae5a5 View commit details
    Browse the repository at this point in the history

  6. cleaning

    @kermitt2
    kermitt2 committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    d8d37ba View commit details
    Browse the repository at this point in the history

  7. fix conflict

    @kermitt2
    kermitt2 committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    43d7957 View commit details
    Browse the repository at this point in the history

  8. format variant

    @kermitt2
    kermitt2 committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    e9d0ef4 View commit details
    Browse the repository at this point in the history

  9. more variants

    @kermitt2
    kermitt2 committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    0d34a17 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. better logging

    @kermitt2
    kermitt2 committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    c58e836 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. add HAL audit and related process

    @kermitt2
    kermitt2 committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    860ff1c View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. fix hal id injection

    @kermitt2
    kermitt2 committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    da38e29 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2024

  1. update tests

    @kermitt2
    kermitt2 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    8efc84e View commit details
    Browse the repository at this point in the history