Merge branch 'IQSS:develop' into 9683_get_dataset_api_in_single_query

doc/release-notes/10015-RO-Crate-metadata-file.md

@@ -0,0 +1,10 @@
+Detection of mime-types based on a filename with extension and detection of the RO-Crate metadata files.
+
+From now on, filenames with extensions can be added into `MimeTypeDetectionByFileName.properties` file. Filenames added there will take precedence over simply recognizing files by extensions. For example, two new filenames are added into that file:
+```
+ro-crate-metadata.json=application/ld+json; profile="http://www.w3.org/ns/json-ld#flattened http://www.w3.org/ns/json-ld#compacted https://w3id.org/ro/crate"
+ro-crate-metadata.jsonld=application/ld+json; profile="http://www.w3.org/ns/json-ld#flattened http://www.w3.org/ns/json-ld#compacted https://w3id.org/ro/crate"
+```
+
+Therefore, files named `ro-crate-metadata.json` will be then detected as RO-Crated metadata files from now on, instead as generic `JSON` files.
+For more information on the RO-Crate specifications, see https://www.researchobject.org/ro-crate

doc/release-notes/10116-incomplete-metadata-label-setting.md

@@ -0,0 +1 @@
+Bug fixed for the ``incomplete metadata`` label being shown for published dataset with incomplete metadata in certain scenarios. This label will now be shown for draft versions of such datasets and published datasets that the user can edit. This label can also be made invisible for published datasets (regardless of edit rights) with the new option ``dataverse.ui.show-validity-label-when-published`` set to `false`.

doc/release-notes/10425-add-MIT-License.md

@@ -0,0 +1,3 @@
+A new file has been added to import the MIT License to Dataverse: licenseMIT.json.
+
+Documentation has been added to explain the procedure for adding new licenses to the guides.

doc/release-notes/10477-metadatablocks-api-extension-input-levels.md

@@ -0,0 +1,3 @@
+Changed ``api/dataverses/{id}/metadatablocks`` so that setting the query parameter ``onlyDisplayedOnCreate=true`` also returns metadata blocks with dataset field type input levels configured as required on the General Information page of the collection, in addition to the metadata blocks and their fields with the property ``displayOnCreate=true`` (which was the original behavior).
+
+A new endpoint ``api/dataverses/{id}/inputLevels`` has been created for updating the dataset field type input levels of a collection via API.

doc/release-notes/6.2-release-notes.md

@@ -417,12 +417,16 @@ In the following commands we assume that Payara 6 is installed in `/usr/local/pa
 
 As noted above, deployment of the war file might take several minutes due a database migration script required for the new storage quotas feature.
 
-6\. Restart Payara
+6\. For installations with internationalization:
+
+- Please remember to update translations via [Dataverse language packs](https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs).
+
+7\. Restart Payara
 
 - `service payara stop`
 - `service payara start`
 
-7\. Update the following Metadata Blocks to reflect the incremental improvements made to the handling of core metadata fields: 
+8\. Update the following Metadata Blocks to reflect the incremental improvements made to the handling of core metadata fields: 
 
  ```
  wget https://github.com/IQSS/dataverse/releases/download/v6.2/geospatial.tsv
@@ -442,7 +446,7 @@ wget https://github.com/IQSS/dataverse/releases/download/v6.2/biomedical.tsv
 curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/biomedical.tsv
 ```
 
-8\. For installations with custom or experimental metadata blocks:
+9\. For installations with custom or experimental metadata blocks:
 
 - Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.2/installation/prerequisites.html#solr-init-script))
 
@@ -455,7 +459,7 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta
      
 - Restart Solr instance (usually `service solr restart` depending on solr/OS)
 
-9\. Reindex Solr:
+10\. Reindex Solr:
 
    For details, see https://guides.dataverse.org/en/6.2/admin/solr-search-index.html but here is the reindex command:
 

doc/release-notes/8243-improve-language-controlled-vocab.md

@@ -0,0 +1,11 @@
+The Controlled Vocabuary Values list for the metadata field Language in the Citation block has been improved, with some missing two- and three-letter ISO 639 codes added, as well as more alternative names for some of the languages, making all these extra language identifiers importable.
+
+To be added to the 6.3 release instructions:
+
+Update the Citation block, to incorporate the improved controlled vocabulary for language [plus whatever other improvements may be made to the block in other PRs]:
+
+```
+wget https://raw.githubusercontent.com/IQSS/dataverse/v6.3/scripts/api/data/metadatablocks/citation.tsv
+curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv
+```
+

doc/release-notes/8655-re-add-cell-counting-biomedical-tsv.md

@@ -0,0 +1,12 @@
+## Release Highlights
+
+### Life Science Metadata
+
+Re-adding value `cell counting` to Life Science metadatablock's Measurement Type vocabularies accidentally removed in `v5.1`. 
+
+## Upgrade Instructions
+
+### Update the Life Science metadata block
+
+- `wget https://github.com/IQSS/dataverse/releases/download/v6.3/biomedical.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @biomedical.tsv -H "Content-type: text/tab-separated-values"`

doc/release-notes/8936-more-than-50000-entries-in-sitemap.md

@@ -0,0 +1,11 @@
+Dataverse can now handle more than 50,000 items when generating sitemap files, splitting the content across multiple files to comply with the Sitemap protocol.
+
+For details see https://dataverse-guide--10321.org.readthedocs.build/en/10321/installation/config.html#creating-a-sitemap-and-submitting-it-to-search-engines #8936 and #10321.
+
+## Upgrade instructions
+
+If your installation has more than 50,000 entries, you should re-submit your sitemap URL to Google or other search engines. The file in the URL will change from ``sitemap.xml`` to ``sitemap_index.xml``.
+
+As explained at https://dataverse-guide--10321.org.readthedocs.build/en/10321/installation/config.html#creating-a-sitemap-and-submitting-it-to-search-engines this is the command for regenerating your sitemap:
+
+`curl -X POST http://localhost:8080/api/admin/sitemap`

doc/release-notes/9276-allow-flexible-params-in-retrievaluri-cvoc.md

@@ -0,0 +1,14 @@
+## Release Highlights
+
+### Updates on Support for External Vocabulary Services
+
+#### HTTP Headers
+
+You are now able to add HTTP request headers required by the service you are implementing (#10331)
+
+#### Flexible params in retrievalUri
+
+You can now use `managed-fields` field names as well as the `term-uri-field` field name as parameters in the `retrieval-uri` when configuring an external vocabulary service. `{0}` as an alternative to using the `term-uri-field` name is still supported for backward compatibility.
+Also you can specify if the value must be url encoded with `encodeUrl:`. (#10404)
+
+For example : `"retrieval-uri": "https://data.agroportal.lirmm.fr/ontologies/{keywordVocabulary}/classes/{encodeUrl:keywordTermURL}"`

doc/release-notes/9739-url-validator.md

@@ -0,0 +1,7 @@
+## Release Highlights
+
+### URL validation is more permissive
+
+Url validation now allows two slashes in the path component of the URL. (#9750)
+Among other things, this allows metadata fields of `url` type to be filled with more complex url such as https://archive.softwareheritage.org/browse/directory/561bfe6698ca9e58b552b4eb4e56132cac41c6f9/?origin_url=https://github.com/gem-pasteur/macsyfinder&revision=868637fce184865d8e0436338af66a2648e8f6e1&snapshot=1bde3cb370766b10132c4e004c7cb377979928d1
+

doc/sphinx-guides/source/api/native-api.rst

@@ -898,7 +898,46 @@ The following attributes are supported:
 * ``filePIDsEnabled`` ("true" or "false") Restricted to use by superusers and only when the :ref:`:AllowEnablingFilePIDsPerCollection <:AllowEnablingFilePIDsPerCollection>` setting is true. Enables or disables registration of file-level PIDs in datasets within the collection (overriding the instance-wide setting).
 
 .. _collection-storage-quotas:
-
+
+Update Collection Input Levels
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Updates the dataset field type input levels in a collection.
+
+Please note that this endpoint overwrites all the input levels of the collection page, so if you want to keep the existing ones, you will need to add them to the JSON request body.
+
+If one of the input levels corresponds to a dataset field type belonging to a metadata block that does not exist in the collection, the metadata block will be added to the collection.
+
+This endpoint expects a JSON with the following format::
+
+  [
+    {
+      "datasetFieldTypeName": "datasetFieldTypeName1",
+      "required": true,
+      "include": true
+    },
+    {
+      "datasetFieldTypeName": "datasetFieldTypeName2",
+      "required": true,
+      "include": true
+    }
+  ]
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ID=root
+  export JSON='[{"datasetFieldTypeName":"geographicCoverage", "required":true, "include":true}, {"datasetFieldTypeName":"country", "required":true, "include":true}]'
+
+  curl -X PUT -H "X-Dataverse-key: $API_TOKEN" -H "Content-Type:application/json" "$SERVER_URL/api/dataverses/$ID/inputLevels" -d "$JSON"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -X PUT -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -H "Content-Type:application/json" "https://demo.dataverse.org/api/dataverses/root/inputLevels" -d '[{"datasetFieldTypeName":"geographicCoverage", "required":true, "include":false}, {"datasetFieldTypeName":"country", "required":true, "include":false}]'
+
 Collection Storage Quotas
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/sphinx-guides/source/container/running/production.rst

@@ -7,7 +7,32 @@ Production (Future)
 Status
 ------
 
-The images described in this guide are not yet recommended for production usage.
+The images described in this guide are not yet recommended for production usage, but we think we are close. We'd like to make the following improvements:
+
+- Tagged releases
+
+  - Currently, you have the choice between "alpha" images that change under your feet every time a new version of Dataverse is released or "unstable" images that track the "develop" branch, which is updated frequently. Instead, we'd like to offer images like 6.4, 6.5, etc. We are tracking this work at https://github.com/IQSS/dataverse/issues/10478 and there is some preliminary code at https://github.com/IQSS/dataverse/tree/10478-version-base-img . You are welcome to join the following discussions:
+
+    - https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/change.20version.20scheme.20base.20image.3F/near/405636949
+    - https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/tagging.20images.20with.20versions/near/366600747 
+
+- More docs on setting up additional features
+
+  - How to set up previewers. See https://github.com/IQSS/dataverse/issues/10506
+  - How to set up Rserve.
+
+- Go through all the features in docs and check what needs to be done differently with containers
+
+  - Check ports, for example.
+
+To join the discussion on what else might be needed before declaring images ready for production, please comment on https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/containers.20for.20production/near/434979159
+
+You are also very welcome to join our meetings. See "how to help" below.
+
+Limitations
+-----------
+
+- Multiple apps servers are not supported. See :ref:`multiple-app-servers` for more on this topic.
 
 How to Help
 -----------

doc/sphinx-guides/source/developers/coding-style.rst

@@ -18,6 +18,11 @@ Tabs vs. Spaces
 
 Don't use tabs. Use 4 spaces.
 
+Imports
+^^^^^^^
+
+Wildcard imports are neither encouraged nor discouraged. 
+
 Braces Placement
 ^^^^^^^^^^^^^^^^
 

doc/sphinx-guides/source/developers/tools.rst

@@ -137,6 +137,14 @@ For example...
 
 would be consistent with a file descriptor leak on the dataset page.
 
+JProfiler
++++++++++
+
+Tracking down resource drainage, bottlenecks etc gets easier using a profiler.
+
+We thank EJ Technologies for granting us a free open source project license for their Java profiler
+`JProfiler <https://www.ej-technologies.com/products/jprofiler/overview.html>`_.
+
 jmap and jstat
 ++++++++++++++