Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issues with docker-compose #41

Merged
merged 4 commits into from
Sep 24, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 5 additions & 7 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ FROM centos:centos8.3.2011 as osmt-base
LABEL Maintainer="WGU / OSN"
LABEL Version="1.0"

ENV JAVA_VERSION=11.0.9.11
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we removing the Java version?

Copy link
Contributor Author

@JohnKallies JohnKallies Sep 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We continue to pull in the latest OpenJDK 11. yum install is no longer able to find an openjdk package as the name is composed (using a specific version number, line 18, below).

For what it's worth, this isn't an automation issue or Docker networking. I built the image to that point and tried to yum install from the shell in a container. I see the package referenced online, but it's not available in this context.

We could possibly fix this by pointing to a different package repo?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we able to bump the Java version to the latest package?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can look at that. I thought that's what we would get by just saying java-11-openjdk on line 17.

Copy link
Contributor Author

@JohnKallies JohnKallies Sep 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@drey-bigney We were talking offline about pinning to a specific openjdk version. If we pin to specific versions of OS packages or software dependencies, then we absorb responsibility for doing security updates out-of-band from the build process. We trade security updates for the safety of known-good dependency trees. The world keeps turning, and things will drift... Better than pinning to a given version of OpenJDK, we should probably have a pipeline that runs all this and shows a build status badge.

I've added Issue #44 "Implement build badges for docker-compose". To me, this is a superior approach to pinning to a specific patch version of java-11-openjdk, and satisfies the open issues for this PR.

Re: omitting the integration tests from the Docker build, using a Maven profile was the cleanest, duct-tape-free self-documenting approach that I saw.

Copy link
Contributor

@drey-bigney drey-bigney Sep 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JohnKallies that makes sense. As long as we are sticking with the same major version of the JDK I doubt we'll see breaking changes and this'll free us from breaking the build system every time a minor JDK revision happens.
+1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. I thought that patch management was a key responsibility of any project's maintainers... Therefore, pinning is implied by the responsibility to maintain. If you don't pin, you don't know what you're getting. "Trusting the source" has led to countless problems for my team, when they don't follow semver and break compatibility, or the fixes have breaking side-effects that didn't affect the mainline users. I think pinning is the safe approach.

ENV JAVA_HOME=/etc/alternatives/jre
ENV USER=osmt
ENV BASE_DIR=/opt/${USER}
Expand All @@ -15,7 +14,7 @@ ENV BASE_DIR=/opt/${USER}
RUN /usr/bin/yum install -y epel-release \
&& /usr/bin/yum update -y \
&& /usr/bin/yum remove -y java-1.8.0-openjdk* \
&& /usr/bin/yum install -y curl java-11-openjdk-headless-${JAVA_VERSION} wget
&& /usr/bin/yum install -y curl java-11-openjdk wget

# Add in configuration files
ADD ./docker/etc /etc
Expand All @@ -34,15 +33,14 @@ RUN /usr/sbin/useradd -r -d ${BASE_DIR} -s /bin/bash ${USER} -k /etc/skel -m -U
FROM osmt-base as build

ENV JAVA_HOME=/etc/alternatives/jre
ENV JAVA_VERSION=11.0.9.11
ENV M2_VERSION=3.6.3
ENV M2_VERSION=3.8.1
ENV M2_HOME=/usr/local/maven
ENV PATH=${M2_HOME}/bin:${PATH}
ENV USER=osmt
ENV BASE_DIR=/opt/${USER}

# Install OpenJDK Development Packages
RUN /usr/bin/yum install -y java-11-openjdk-devel-${JAVA_VERSION}
RUN /usr/bin/yum install -y java-11-openjdk-devel

# Download / Install Maven
ADD https://www-eu.apache.org/dist/maven/maven-3/${M2_VERSION}/binaries/apache-maven-${M2_VERSION}-bin.tar.gz /usr/share/src/
Expand All @@ -59,15 +57,15 @@ WORKDIR ${BASE_DIR}/build

USER ${USER}

RUN mvn clean package
# The dockerfile-build Maven profile excludes certain api integration tests that require access to the Docker service.
RUN mvn clean package -P dockerfile-build

######################
### PACKAGING IMAGE ##
######################
FROM osmt-base

ENV JAVA_HOME=/etc/alternatives/jre
ENV JAVA_VERSION=11.0.9.11
drey-bigney marked this conversation as resolved.
Show resolved Hide resolved
ENV USER=osmt
ENV BASE_DIR=/opt/${USER}

Expand Down
55 changes: 54 additions & 1 deletion api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,11 @@
</properties>

<dependencies>
<dependency>
<groupId>edu.wgu.osmt</groupId>
<artifactId>osmt-ui</artifactId>
<version>1.0.5-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
Expand Down Expand Up @@ -259,7 +264,7 @@
<version>${flywaydb.version}</version>
<configuration>
<url>jdbc:mysql://localhost:3306/osmt_db</url>
<user>changeme</user>
<user>osmt</user>
<password>password</password>
<locations>
<location>classpath:db/migration</location>
Expand Down Expand Up @@ -291,7 +296,55 @@
<commitIdGenerationMode>full</commitIdGenerationMode>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-help-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<id>show-profiles</id>
<phase>compile</phase>
<goals>
<goal>active-profiles</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

<profiles>
<profile>
<id>dockerfile-build</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludes>
<!-- for Kotlin files, don't include the extension - https://stackoverflow.com/questions/39723763/kotlin-maven-not-executing-tests#39735811 -->
<exclude>AuditLogRepositoryTest</exclude>
<exclude>BatchImportRichSkillTest</exclude>
<exclude>BlsImportTest</exclude>
<exclude>CollectionEsRepoTest</exclude>
<exclude>CollectionRepositoryTest</exclude>
<exclude>ElasticSearchReindexerTest</exclude>
<exclude>JobCodeEsRepoTest</exclude>
<exclude>JobCodeRepositoryTest</exclude>
<exclude>KeywordEsRepoTest</exclude>
<exclude>KeywordRepositoryTest</exclude>
<exclude>OnetImportTest</exclude>
<exclude>RichSkillControllerTest</exclude>
<exclude>RichSkillEsRepoTest</exclude>
<exclude>RichSkillRepositoryTest</exclude>
<exclude>SearchControllerTest</exclude>
<exclude>TaskTest</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
37 changes: 37 additions & 0 deletions api/src/main/kotlin/edu/wgu/osmt/PropertyLogger.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package edu.wgu.osmt

import org.slf4j.LoggerFactory
import org.springframework.context.annotation.Profile
import org.springframework.context.event.ContextRefreshedEvent
import org.springframework.context.event.EventListener
import org.springframework.core.env.AbstractEnvironment
import org.springframework.core.env.EnumerablePropertySource
import org.springframework.core.env.PropertySource
import org.springframework.stereotype.Component
import java.util.*
import java.util.stream.StreamSupport


@Component
@Profile("dev")
class PropertyLogger {
@EventListener
fun handleContextRefresh(event: ContextRefreshedEvent) {
val env = event.applicationContext.environment
LOGGER.info("====== Environment and configuration ======")
LOGGER.info("Active profiles: {}", Arrays.toString(env.activeProfiles))
val sources = (env as AbstractEnvironment).propertySources
StreamSupport.stream(sources.spliterator(), false)
.filter { ps: PropertySource<*>? -> ps is EnumerablePropertySource<*> }
.map { ps: PropertySource<*> -> (ps as EnumerablePropertySource<*>).propertyNames }
.flatMap { array: Array<String>? -> Arrays.stream(array) }
.distinct()
.filter { prop: String -> !(prop.contains("credentials") || prop.contains("password")) }
.forEach { prop: String? -> LOGGER.info("{}: {}", prop, env.getProperty(prop!!)) }
LOGGER.info("===========================================")
}

companion object {
private val LOGGER = LoggerFactory.getLogger(PropertyLogger::class.java)
}
}
6 changes: 6 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,14 @@ services:
environment:
- "ENVIRONMENT=apiserver,dev"
- "MYSQL_DB_URI=osmt:password@db:3306"
- "DB_NAME=osmt_db"
- "ELASTICSEARCH_URI=elasticsearch:9200"
- "REDIS_URI=redis:6379"
- "MIGRATIONS_ENABLED=false"
- "OAUTH_ISSUER=${OAUTH_ISSUER}"
- "OAUTH_CLIENTID=${OAUTH_CLIENTID}"
- "OAUTH_CLIENTSECRET=${OAUTH_CLIENTSECRET}"
- "OAUTH_AUDIENCE=${OAUTH_AUDIENCE}"
depends_on:
- db
- elasticsearch
Expand Down
7 changes: 7 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,11 @@
<module>ui</module>
<module>api</module>
</modules>

<profiles>
<profile>
<id>dockerfile-build</id>
</profile>
</profiles>

</project>
2 changes: 1 addition & 1 deletion ui/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"start": "./node_modules/@angular/cli/bin/ng serve",
"start-hotreload": "./node_modules/@angular/cli/bin/ng serve --live-reload",
"build": "./node_modules/@angular/cli/bin/ng build",
"build-prod": "./node_modules/@angular/cli/bin/ng build --prod",
"build-prod": "./node_modules/@angular/cli/bin/ng build --prod --output-path dist/ui/META-INF/resources",
"test": "./node_modules/@angular/cli/bin/ng test",
"lint": "./node_modules/@angular/cli/bin/ng lint",
"e2e": "./node_modules/@angular/cli/bin/ng e2e",
Expand Down
14 changes: 12 additions & 2 deletions ui/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,11 @@
<artifactId>osmt-ui</artifactId>
<version>1.0.5-SNAPSHOT</version>

<packaging>pom</packaging>
<packaging>jar</packaging>

<properties>
<main.basedir>${project.parent.basedir}</main.basedir>
</properties>

<build>
<resources>
Expand All @@ -24,7 +28,7 @@
<plugin>
<groupId>com.github.eirslett</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<version>1.7.6</version>
<version>1.9.1</version>
<configuration>
<workingDirectory>./</workingDirectory>
<nodeVersion>v10.16.0</nodeVersion>
Expand All @@ -36,22 +40,28 @@
<goals>
<goal>install-node-and-npm</goal>
</goals>
<phase>generate-resources</phase>
</execution>
<execution>
<id>npm install</id>
<goals>
<goal>npm</goal>
</goals>
<configuration>
<arguments>install</arguments>
</configuration>
</execution>
<execution>
<id>npm run build-prod</id>
<goals>
<goal>npm</goal>
</goals>
<phase>generate-resources</phase>
<configuration>
<arguments>run build-prod</arguments>
</configuration>
</execution>

</executions>
</plugin>
</plugins>
Expand Down