Skip to content

Commit

Permalink
Add example to docs how to get custom metadata inside the job. See gd…
Browse files Browse the repository at this point in the history
  • Loading branch information
poikilotherm authored and T-Haeussermann committed Apr 25, 2024
1 parent f4677c5 commit 74af823
Show file tree
Hide file tree
Showing 2 changed files with 75 additions and 1 deletion.
62 changes: 62 additions & 0 deletions docs/day2/examples/metadata-update-w-init.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
apiVersion: batch/v1
kind: Job
metadata:
generateName: metadata-update-dataverse-
labels:
app.kubernetes.io/name: metadata-update
app.kubernetes.io/version: "1.0"
app.kubernetes.io/component: job
app.kubernetes.io/part-of: dataverse
app.kubernetes.io/managed-by: kubectl
spec:
template:
metadata:
labels:
app.kubernetes.io/name: metadata-update
app.kubernetes.io/part-of: dataverse
spec:
containers:
- name: metadata-update
image: iqss/dataverse-k8s
command: ['scripts/metadata-update.sh']
envFrom:
- configMapRef:
name: dataverse
volumeMounts:
- name: api-secret
mountPath: "/opt/dataverse/secrets/api"
readOnly: true
- name: custom-metadata
mountPath: /metadata
readOnly: true
initContainers:
- name: check-solr-ready
image: giantswarm/tiny-tools
command: ['sh', '-c',
'until curl -sS -f "http://${SOLR_SERVICE_HOST}:${SOLR_SERVICE_PORT}/solr/collection1/admin/ping" 2>&1 | grep "status.:.OK";
do echo waiting for solr; sleep 3; done;']
- name: check-dataverse-ready
image: giantswarm/tiny-tools
command: ['sh', '-c',
'until curl -sS -f "http://${DATAVERSE_SERVICE_HOST}:${DATAVERSE_SERVICE_PORT_HTTP}/robots.txt" -m 2 2>&1 > /dev/null;
do echo waiting for dataverse; sleep 15; done;']
- name: get-metadata
image: giantswarm/tiny-tools
command:
- "curl"
args:
- "-sSo"
- "/metadata/test.tsv"
- "https://gist.githubusercontent.com/poikilotherm/e54660ab99a24b12e5179621c9c7efb5/raw/960085c8277ad33fa1e52f3c16a38ec6df3ef281/test.tsv"
volumeMounts:
- name: custom-metadata
mountPath: /metadata
volumes:
- name: api-secret
secret:
secretName: dataverse-api
- name: custom-metadata
emptyDir: {}
restartPolicy: Never
backoffLimit: 0
14 changes: 13 additions & 1 deletion docs/day2/job-metadata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ See also :guide_dv:`upstream admin guide <admin/metadataexport.html>` about
metadata exports.

How to get custom metadata blocks inside the job
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
------------------------------------------------

Deploying metadata is reusing the :doc:`/images/dataverse-k8s` by default.
You need to drop metadata TSV files to the ``/metadata`` directory of the jobs
Expand All @@ -75,3 +75,15 @@ This can happen via
1. ``ConfigMaps`` seem to be the easiest option, but in case you use large or large
amounts of custom metadata blocks, you might choose differently.
2. You could override upstream blocks this way. *You shouldn't do it.* Up to you.

Example with ``curl`` init container
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You could create a ``Job`` based on ``k8s/dataverse/jobs/metadata-update.yaml``,
which you extend like below.
(:download:`Download full example <examples/metadata-update-w-init.yaml>`)

.. literalinclude:: examples/metadata-update-w-init.yaml
:language: yaml
:lines: 19,20,26,30-32,33,44-54,55,59-60
:name: metadata-update-w-init

0 comments on commit 74af823

Please sign in to comment.