Copyright (C) 2016 A.Formica, R.Sipos
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Test project for the implementation of a generic purpose conditions database for physics experiment. This server was generated by the swagger-codegen project. By using the OpenAPI-Spec from a remote server, you can easily generate a server stub. This project is an example of building a swagger-enabled JAX-RS server. Some tests were also done to provide a Resteasy implementation.
The prototype uses Spring framework and the REST services are implemented via Jersey.
The prototype runs as a microservice using spring-boot. By default it uses an embedded undertow servlet container, but others like tomcat or jetty can be easily used instead of undertow.
You can directly download an html file from gitlab to visualize the API documentation on your browser:
https://drf-gitlab.cea.fr/api/v4/projects/523/repository/files/doc%2Findex.html/raw?ref=master
Download the project from gitlab (example below is using https):
git clone https://gitlab.cern.ch/formica/crest.git
or
git clone https://github.com/HSF/Crest.git
or
git clone https://drf-gitlab.cea.fr/svom/common/crest.git
git checkout cms-v0
if you are taking the github version.
This will create a directory crest in the location where you run the git command.
You need to have java >= 8 installed on your machine. If you have also gradle (version 5) you can build the project using the following command from the root project directory (crest):
gradle clean build
This command will generate a war (java web archive) file in : crestdb-web/build/libs/crest.war.
In case gradle is not installed on your machine, you can run the wrapper delivered with the project:
./gradlew clean build
If you want to select a specific JVM when you run gradle you can use a command like this:
gradle clean build -Dorg.gradle.java.home=/path to jvm/11.0/
Be careful to checkout the correct branch.
| ATLAS | CMS | SVOM |
|---|---|---|
| master | cms-v0 | cms-v0 |
| contains tag meta information | standard API | standard API |
This section is under maintenance.
The server will use by default an embedded undertow web server.
The server need by definition to have a database connection in order to store the conditions data. The database connections are defined in the files ./crestdb-web/src/main/resources/application-<profile>.yml. This file present different set of properties which are chosen by selecting a specific spring profile when running the server. The file should be edited if you are administering the conditions database in order to provide an appropriate set of parameters.
If you do not have any remote database available you should use the default spring profile.
The set of default properties to run the server is defined in config/application.properties which will be read by spring when starting the server. The file there will use the default spring profile and a local database instance h2database where to store the data (it is a sort of sqlite file).
It is essential to set appropriate parameters in this file. An example
is provided to start up a default profile with local h2 database.
spring.profiles.active=oracle
user.timezone=GMT
crest.api.name=/crestapi
crest.web.static=/tmp/data/web
crest.dump.dir=/tmp/data/dump
crest.log.dir=/tmp/data/dump
crest.db.password=somepassword
crest.db.user=ATLAS_PHYS_COND_01_W
crest.port=8090
crest.db.url=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=cman1-atlas.cern.ch)(PORT=10500))(LOAD_BALANCE=on)(ENABLE=BROKEN)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=int8r.cern.ch)))
crest.db.schema=ATLAS_PHYS_COND_01
spring.profiles.active=postgres
user.timezone=GMT
crest.web.static=/tmp/data/web
crest.dump.dir=/tmp/data/dump
crest.log.dir=/tmp/data/logs
management.endpoint.health.show-details=ALWAYS
crest.db.url=jdbc:postgresql://postgres-host:5432/crestdb
crest.db.user=someuser
crest.db.password=somepassword
To start the server you can simply run:
./entrypoint.sh
This script is the same that is used by the docker container (when packaging the server via the Dockerfile).
We provide the following commands as examples for alternative way (not maintained anymore):
cd crestdb-web
$ gradle bootRun "-Dspring.profiles.active=oracle" "-Dcrest.db.password=xxx"
or
$java -Dspring.profiles.active=oracle -Dcrest.db.password=xxx -jar crestdb-web/build/libs/crest.war
This part is obsolete. We are working on OAuth2 implementation.
To activate security you need to build the war file including the key-store. The file should go into /src/main/resources together with a complete ldap.properties file in which you need to set the manager password. These are not detailed instructions, it is more a reminder.
java -Dstore.password=xxx -Dkey.password=yyy -Dcrest.db.password=ddd -Dcrest.dump.dir=/data/data/dump -Dcrest.web.static=/data/data/web -Dspring.profiles.active=prod -jar crestdb-web/build/libs/crest.war
The prod profile is using CERN ldap. Here is an example of ldap properties.
USER_SEARCH_BASE="DC=cern,DC=ch"
USER_DN_PATTERNS="CN={0},OU=Users,DC=cern,DC=ch"
GROUP_SEARCH_BASE="OU=e-groups,OU=Workgroups,DC=cern,DC=ch"
GROUP_SEARCH_FILTER="member={0}"
GROUP_ROLE_ATTRIBUTE=cn
MANAGER_DN="CN=formica,OU=Users,OU=Organic Units,DC=cern,DC=ch"
MANAGER_PASSWORD=xxx
LDAP_AUTHENTICATOR_URL=ldaps://cerndc.cern.ch:636
ACCESS=hasRole('atlas-database')
In order to test security you can try to use curl:
curl -k -u user:password -X GET https://localhost:8443/crestapi/globaltags
The -k should skip verification on the certificate.
In order to connect to the ldap server we need to have the truststore correctly set and with an alias corresponding to the cerndc.cern.ch certificate. Some java properties need to be set for this:
-Djavax.net.ssl.trustStore=/ssl-crest-server.jks -Djavax.net.ssl.trustStorePassword=xxx -Djavax.net.debug=ssl
Be careful that the properties defined in the application.yml do not work for the truststore.
In order to add certificates to the truststore you can proceed in the following way:
echo -n | openssl s_client -connect cerndc.cern.ch:636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/examplecert.crt
openssl x509 -in /tmp/examplecert.crt -text
This will retrieve the server side certificate of the host you want to connect to for authentication.
keytool -import -trustcacerts -keystore ./crestdb-web/src/main/resources/ssl-crest-server.jks -storepass xxxxx -noprompt -alias cern -file /tmp/examplecert.crt
This instead will add the certificate to the truststore (which in our case is the same file). The truststore has been created using a command like:
keytool -genkey -alias crest_localhost_sslserver -keyalg RSA -keysize 2048 -validity 700 -keypass xxx -storepass xxxx -keystore ssl-crest-server.jks
You can view the swagger listing here (hopefully the server will be up!):
http://crest-undertow.web.cern.ch/crestapi/swagger.json
and if you want to play with the server using the swagger-ui you can access it here:
http://crest-undertow.web.cern.ch/ext/web/ui/index.html
Note that in principle you can get the same links working (a part from the hostname) if you run the server locally.
The same kind of visualisation is available directly in gitlab when accessing the specification file.
In order to regenerate the API we use the JSON schemas and templates which are store in the directories:
./swagger_schemas
./templates
To run code generation some scripts can be used as examples (./scripts).
The server stub generation is implemented as well as a gradle task:
./gradlew generateSwaggerCode
You can build a container using
docker build -t crest:1.0 .
You can run the container using
docker run --env-file .environment -p 8080:8080 -d crest:1.0
or
docker run --env-file .environment -p 8080:8080 -v /mnt/data/dump:/data/dump -v /mnt/data/web:/data/web --net=host -d crest:test
In the last example we have been mounting external volumes. These are useful for the swagger-ui and the possibility to dump a tag in a file system based structure. You can use the swagger-ui version that is provided within this project in the directory
./web/ui/
A special note about the file .environment . You need to have this file to set variables which are used at the startup of the server. Some of the variables are already provided in the version in git, but other are not. For example, to access Oracle at CERN (for the moment only integration cluster contains a crest schema) you need to have the variable crest.db.password=xxxxx correctly set for a writer account.
If you use spring.profiles.active=default you will have an h2 database created in jdbc:h2:/tmp/cresth2;DB_CLOSE_ON_EXIT=FALSE.
You can connect to a running container using commands like:
docker exec -i -t infallible_stonebraker /bin/bash
In order to push an image created into the CERN gitlab registry you need to login:
docker login -u formica gitlab-registry.cern.ch
You can choose another registry. The push command work only if the login is successful.
As an example for a deployment in a swarm look at ./swarm/docker-compose.yml.
In the same repository there is a script to help in generating the config maps needed and the secret.
An application.properties file should be created in order to run the server. Examples are shown above.
We gather here some notes on openshift deployment via gitlab-ci. These notes are for usage inside CERN.
For the moment in order for the deployment to work we need to have a public access to the gitlab project.
After committing a tag it seems that the deploy to openshift fails. TO BE DONE.
We have been merging our clients in this repository with the contribution of some colleagues from Juno collaboration. We have now work in progress in the following areas (available code can be seen in crestdb-client repository).
This is work in progress...documentation needs to be improved....
A small client is available in crestdb-client/python/cli that can be installed via pip.
One can also generate a python client via swagger.
Ask Juno colleagues.
Generated via swagger. Used for testing REST API.
Generated via swagger. This is just a demo.
In addition we have recently added a Web GUI in VueJS. The project can be found in web-ui/crest-ui. In order to run it one can simply follow the readme file. It can use for development purpose npm and node.