Allow anonymous API acess to search and list endpoints for published …

…skills and collections (#16) * Support anonymous API usage of searching and listing for skills or collections. allow for rate limiting public requests with bucket4j * added a clarifying statement under attribution (#17) as per legal department, I have added the statement under attribution about the contribution covenant. * Update CONTRIBUTING.md (#22) Updated contribution guidelines * DMND-631 - Update code to include BLS JobCodes when the skills are im… (#26) * DMND-631 - Update code to include BLS JobCodes when the skills are imported using the UI batch import process. * DMND-631 - Addressed PR comments. * DMND-673 Update open source repo with osmt tests (#29) * DMND-674 Update open source repo with osmt-ui tests (#31) * Test improvements (#32) * * Added more tests * Upgraded testing dependencies to support code coverage. * Missed 2 files. * * Removed isAuthorEditable() in many cases because the intent was separate from using the defaultAuthorValue, but was not coded separately. i.e., if there is a defaultAuthorValue, then use it as the default. (#33) * Fixed the assumption that the same web server is used for both UI and Service. That's not always the case. * Temporary handling of the ElasticSearch limitation of 10000 RSDs. Until that is fixed by other means, we add a "+" to reduce confusion about the 10000 RSD count. * Add noauth profile config for local development (#34) * Add noauth profile config for local development SecurityConfigNoAuth.kt taken from ngp-aa-osmt * Add noauth profile config for local development * Update readme (#9) * added architectural diagram and BLS Onet import process * added Okta steps for setup * inserted PNG to assests folder to use in README file for the architectural diagram * fixed png so that it removed the box with the osmt description. * Closes #18 - Update README for community contributors (#39) * Closes #18 - Update README for community contributors also updated the API module's README * Address review feedback for project README and docs - also made a small tweak to .gitignore to support multiple env files. * Readme formatting * Add Testing Expections to CONTRIBUTING.md (#43) * Add Testing Expections to CONTRIBUTING.md - clarified the OSMT definition of unit test / integration test / e2e test * Update CONTRIBUTING.md Co-authored-by: drey-bigney <[email protected]> * Adding test for ElasticSearchReindexer (#45) * Fix issues with docker-compose (#41) * Fix issues with docker-compose - required changes in Dockerfile around yum install for certain openjdk version - required skipping tests when building app Docker image because unit tests are mixed with integration tests that rely on Dockerized services - fixed missinf fat jar functionality with UI as a dependency for API * Update pom.xml * Aadjust Dockerfile for review requests - excluded defacto integration tests from Dockerfile mvn package with "dockerfile-build" Maven profile * add new integration test to pom file for exclusions in docker build * Feature/add code coverage (#46) * * Added more tests * Upgraded testing dependencies to support code coverage. * Missed 2 files. * * Added more tests. * Added test for RichSkillSearchResultsComponent. Also added support for ActivatedRouteStub.setQueryParams. * Renamed ActivatedRouteStub.setParamsMap => setParams. * Support anonymous API usage of searching and listing for skills or collections. allow for rate limiting public requests with bucket4j * Rebase PR and update new test data - also reverted change to MockData for how RSDs relate to connections Co-authored-by: wgu-edwin <[email protected]> Co-authored-by: John Kallies <[email protected]> Co-authored-by: Roberto Meza <[email protected]> Co-authored-by: Devon Sumner <[email protected]> Co-authored-by: drey-bigney <[email protected]>
wgu-opensource · Oct 9, 2021 · 4e34f7c · 4e34f7c
1 parent 1d7291b
commit 4e34f7c
Show file tree

Hide file tree

Showing 17 changed files with 468 additions and 326 deletions.
diff --git a/api/pom.xml b/api/pom.xml
@@ -42,6 +42,8 @@
         <start-class>edu.wgu.osmt.ApplicationKt</start-class>
         <testcontainers.version>1.15.3</testcontainers.version>
         <spring-data-elasticsearch.version>4.1.0-RC2</spring-data-elasticsearch.version>
+        <ehcache.version>3.9.2</ehcache.version>
+        <bucket4j.version>4.10.0</bucket4j.version>
     </properties>
 
     <dependencies>
@@ -97,7 +99,6 @@
         <dependency>
             <groupId>com.okta.spring</groupId>
             <artifactId>okta-spring-boot-starter</artifactId>
-            <version>${okta.version}</version>
         </dependency>
         <dependency>
             <groupId>org.jetbrains.exposed</groupId>
@@ -170,6 +171,11 @@
             <artifactId>flyway-core</artifactId>
             <version>${flywaydb.version}</version>
         </dependency>
+        <dependency>
+            <groupId>com.okta.spring</groupId>
+            <artifactId>okta-spring-boot-starter</artifactId>
+            <version>${okta.version}</version>
+        </dependency>
         <dependency>
             <groupId>com.google.code.gson</groupId>
             <artifactId>gson</artifactId>
@@ -209,6 +215,56 @@
             <groupId>org.springframework.session</groupId>
             <artifactId>spring-session-data-redis</artifactId>
         </dependency>
+        <dependency>
+            <groupId>com.giffing.bucket4j.spring.boot.starter</groupId>
+            <artifactId>bucket4j-spring-boot-starter</artifactId>
+            <version>0.3.3</version>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-cache</artifactId>
+            <version>2.4.3</version>
+        </dependency>
+        <dependency>
+            <groupId>org.ehcache</groupId>
+            <artifactId>ehcache</artifactId>
+            <version>${ehcache.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>com.github.vladimir-bukhtoyarov</groupId>
+            <artifactId>bucket4j-core</artifactId>
+            <version>${bucket4j.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>com.github.vladimir-bukhtoyarov</groupId>
+            <artifactId>bucket4j-jcache</artifactId>
+            <version>${bucket4j.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>com.giffing.bucket4j.spring.boot.starter</groupId>
+            <artifactId>bucket4j-spring-boot-starter</artifactId>
+            <version>0.3.3</version>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-cache</artifactId>
+            <version>2.4.3</version>
+        </dependency>
+        <dependency>
+            <groupId>org.ehcache</groupId>
+            <artifactId>ehcache</artifactId>
+            <version>${ehcache.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>com.github.vladimir-bukhtoyarov</groupId>
+            <artifactId>bucket4j-core</artifactId>
+            <version>${bucket4j.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>com.github.vladimir-bukhtoyarov</groupId>
+            <artifactId>bucket4j-jcache</artifactId>
+            <version>${bucket4j.version}</version>
+        </dependency>
         <dependency>
             <groupId>com.fasterxml.jackson.core</groupId>
             <artifactId>jackson-core</artifactId>
@@ -264,7 +320,7 @@
                 <version>${flywaydb.version}</version>
                 <configuration>
                     <url>jdbc:mysql://localhost:3306/osmt_db</url>
-                    <user>osmt</user>
+                    <user>changeme</user>
                     <password>password</password>
                     <locations>
                         <location>classpath:db/migration</location>
@@ -296,55 +352,7 @@
                     <commitIdGenerationMode>full</commitIdGenerationMode>
                 </configuration>
             </plugin>
-            <plugin>
-                <groupId>org.apache.maven.plugins</groupId>
-                <artifactId>maven-help-plugin</artifactId>
-                <version>3.2.0</version>
-                <executions>
-                    <execution>
-                        <id>show-profiles</id>
-                        <phase>compile</phase>
-                        <goals>
-                            <goal>active-profiles</goal>
-                        </goals>
-                    </execution>
-                </executions>
-            </plugin>
         </plugins>
     </build>
 
-    <profiles>
-        <profile>
-            <id>dockerfile-build</id>
-            <build>
-                <plugins>
-                    <plugin>
-                        <groupId>org.apache.maven.plugins</groupId>
-                        <artifactId>maven-surefire-plugin</artifactId>
-                        <configuration>
-                            <excludes>
-                                <!-- for Kotlin files, don't include the extension - https://stackoverflow.com/questions/39723763/kotlin-maven-not-executing-tests#39735811 -->
-                                <exclude>AuditLogRepositoryTest</exclude>
-                                <exclude>BatchImportRichSkillTest</exclude>
-                                <exclude>BlsImportTest</exclude>
-                                <exclude>CollectionEsRepoTest</exclude>
-                                <exclude>CollectionRepositoryTest</exclude>
-                                <exclude>ElasticSearchReindexerTest</exclude>
-                                <exclude>JobCodeEsRepoTest</exclude>
-                                <exclude>JobCodeRepositoryTest</exclude>
-                                <exclude>KeywordEsRepoTest</exclude>
-                                <exclude>KeywordRepositoryTest</exclude>
-                                <exclude>OnetImportTest</exclude>
-                                <exclude>RichSkillControllerTest</exclude>
-                                <exclude>RichSkillEsRepoTest</exclude>
-                                <exclude>RichSkillRepositoryTest</exclude>
-                                <exclude>SearchControllerTest</exclude>
-                                <exclude>TaskTest</exclude>
-                            </excludes>
-                        </configuration>
-                    </plugin>
-                </plugins>
-            </build>
-        </profile>
-    </profiles>
 </project>
diff --git a/api/src/main/kotlin/edu/wgu/osmt/ApiServer.kt b/api/src/main/kotlin/edu/wgu/osmt/ApiServer.kt
@@ -19,6 +19,7 @@ import org.slf4j.Logger
 import org.slf4j.LoggerFactory
 import org.springframework.beans.factory.annotation.Autowired
 import org.springframework.boot.CommandLineRunner
+import org.springframework.cache.annotation.EnableCaching
 import org.springframework.context.annotation.Bean
 import org.springframework.context.annotation.Profile
 import org.springframework.stereotype.Component
@@ -27,6 +28,7 @@ import org.springframework.web.servlet.config.annotation.EnableWebMvc
 @Component
 @Profile("apiserver")
 @EnableWebMvc
+@EnableCaching
 class ApiServer {
     val logger: Logger = LoggerFactory.getLogger(ApiServer::class.java)
 

diff --git a/api/src/main/kotlin/edu/wgu/osmt/Application.kt b/api/src/main/kotlin/edu/wgu/osmt/Application.kt
@@ -6,8 +6,10 @@ import org.springframework.boot.autoconfigure.SpringBootApplication
 import org.springframework.boot.context.properties.ConfigurationPropertiesScan
 import org.springframework.boot.context.properties.EnableConfigurationProperties
 import org.springframework.boot.runApplication
+import org.springframework.cache.annotation.EnableCaching
 
 @SpringBootApplication
+@EnableCaching
 @ConfigurationPropertiesScan("edu.wgu.osmt.config")
 @EnableConfigurationProperties(DbConfig::class, EsConfig::class)
 class Application

diff --git a/api/src/main/kotlin/edu/wgu/osmt/HasAllPaginated.kt b/api/src/main/kotlin/edu/wgu/osmt/HasAllPaginated.kt
@@ -7,6 +7,8 @@ import edu.wgu.osmt.elasticsearch.*
 import org.springframework.http.HttpEntity
 import org.springframework.http.HttpHeaders
 import org.springframework.http.ResponseEntity
+import org.springframework.security.core.annotation.AuthenticationPrincipal
+import org.springframework.security.oauth2.jwt.Jwt
 import org.springframework.web.bind.annotation.RequestParam
 import org.springframework.web.util.UriComponentsBuilder
 
@@ -33,10 +35,14 @@ interface HasAllPaginated<T> {
             required = false,
             defaultValue = PublishStatus.DEFAULT_API_PUBLISH_STATUS_SET
         ) status: Array<String>,
-        @RequestParam(required = false) sort: String?
+        @RequestParam(required = false) sort: String?,
+        @AuthenticationPrincipal user: Jwt?
     ): HttpEntity<List<T>> {
 
-        val publishStatuses = status.mapNotNull { PublishStatus.forApiValue(it) }.toSet()
+        val publishStatuses = status.mapNotNull {
+            val status = PublishStatus.forApiValue(it)
+            if (user == null && (status == PublishStatus.Deleted  || status == PublishStatus.Draft)) null else status
+        }.toSet()
         val sortEnum: SortOrder = sortOrderCompanion.forValueOrDefault(sort)
         val pageable = OffsetPageable(from, size, sortEnum.sort)
 

diff --git a/api/src/main/kotlin/edu/wgu/osmt/RoutePaths.kt b/api/src/main/kotlin/edu/wgu/osmt/RoutePaths.kt
@@ -8,15 +8,17 @@ object RoutePaths {
     const val SEARCH_COLLECTIONS = "$SEARCH_PATH/collections"
 
     const val SKILLS_PATH = "/api/skills"
-    const val SKILL_LIST = SKILLS_PATH
+    const val SKILLS_LIST = SKILLS_PATH
+    const val SKILLS_CREATE = SKILLS_PATH
     const val SKILL_PUBLISH = "$SKILLS_PATH/publish"
     const val SKILL_DETAIL = "$SKILLS_PATH/{uuid}"
     const val SKILL_UPDATE = "$SKILL_DETAIL/update"
     const val SKILL_AUDIT_LOG = "${SKILL_DETAIL}/log"
 
 
     const val COLLECTIONS_PATH = "/api/collections"
-    const val COLLECTION_LIST = COLLECTIONS_PATH
+    const val COLLECTIONS_LIST = COLLECTIONS_PATH
+    const val COLLECTION_CREATE = COLLECTIONS_PATH
     const val COLLECTION_PUBLISH = "$COLLECTIONS_PATH/publish"
     const val COLLECTION_DETAIL = "${COLLECTIONS_PATH}/{uuid}"
     const val COLLECTION_UPDATE = "${COLLECTION_DETAIL}/update"
@@ -42,4 +44,8 @@ object RoutePaths {
         const val SORT = "sort"
         const val COLLECTION_ID = "collectionId"
     }
+
+    fun scrubForConfigure(routePath: String): String {
+        return routePath.replace("{uuid}", "*")
+    }
 }
diff --git a/api/src/main/kotlin/edu/wgu/osmt/collection/CollectionController.kt b/api/src/main/kotlin/edu/wgu/osmt/collection/CollectionController.kt
@@ -2,6 +2,7 @@ package edu.wgu.osmt.collection
 
 import edu.wgu.osmt.HasAllPaginated
 import edu.wgu.osmt.RoutePaths
+import edu.wgu.osmt.api.GeneralApiException
 import edu.wgu.osmt.api.model.*
 import edu.wgu.osmt.auditlog.AuditLog
 import edu.wgu.osmt.auditlog.AuditLogRepository
@@ -36,19 +37,23 @@ class CollectionController @Autowired constructor(
 
     override val elasticRepository = collectionEsRepo
 
-    override val allPaginatedPath: String = RoutePaths.COLLECTIONS_PATH
+    override val allPaginatedPath: String = RoutePaths.COLLECTIONS_LIST
     override val sortOrderCompanion = CollectionSortEnum.Companion
 
-    @GetMapping(RoutePaths.COLLECTIONS_PATH, produces = [MediaType.APPLICATION_JSON_VALUE])
+    @GetMapping(RoutePaths.COLLECTIONS_LIST, produces = [MediaType.APPLICATION_JSON_VALUE])
     @ResponseBody
     override fun allPaginated(
         uriComponentsBuilder: UriComponentsBuilder,
         size: Int,
         from: Int,
         status: Array<String>,
-        sort: String?
+        sort: String?,
+        @AuthenticationPrincipal user: Jwt?
     ): HttpEntity<List<CollectionDoc>> {
-        return super.allPaginated(uriComponentsBuilder, size, from, status, sort)
+        if (!appConfig.allowPublicLists && user === null) {
+            throw GeneralApiException("Unauthorized", HttpStatus.UNAUTHORIZED)
+        }
+        return super.allPaginated(uriComponentsBuilder, size, from, status, sort, user)
     }
 
     @GetMapping(RoutePaths.COLLECTION_DETAIL, produces = [MediaType.APPLICATION_JSON_VALUE])
@@ -64,7 +69,7 @@ class CollectionController @Autowired constructor(
         return "forward:/collections/$uuid"
     }
 
-    @PostMapping(RoutePaths.COLLECTIONS_PATH, produces = [MediaType.APPLICATION_JSON_VALUE])
+    @PostMapping(RoutePaths.COLLECTION_CREATE, produces = [MediaType.APPLICATION_JSON_VALUE])
     @ResponseBody
     fun createCollections(
         @RequestBody apiCollectionUpdates: List<ApiCollectionUpdate>,

diff --git a/api/src/main/kotlin/edu/wgu/osmt/config/AppConfig.kt b/api/src/main/kotlin/edu/wgu/osmt/config/AppConfig.kt
@@ -31,6 +31,12 @@ class AppConfig(
     @Value("\${app.loginSuccessRedirectUrl}")
     val loginSuccessRedirectUrl: String,
 
+    @Value("\${app.allowPublicSearching}")
+    val allowPublicSearching: Boolean = true,
+
+    @Value("\${app.allowPublicLists}")
+    val allowPublicLists: Boolean = true,
+
     @Value("\${app.baseLineAuditLogIfEmpty}")
     val baseLineAuditLogIfEmpty: Boolean
 ) {

diff --git a/api/src/main/kotlin/edu/wgu/osmt/elasticsearch/SearchController.kt b/api/src/main/kotlin/edu/wgu/osmt/elasticsearch/SearchController.kt
@@ -16,6 +16,8 @@ import edu.wgu.osmt.richskill.RichSkillEsRepo
 import edu.wgu.osmt.richskill.RichSkillDoc
 import org.springframework.beans.factory.annotation.Autowired
 import org.springframework.http.*
+import org.springframework.security.core.annotation.AuthenticationPrincipal
+import org.springframework.security.oauth2.jwt.Jwt
 import org.springframework.stereotype.Controller
 import org.springframework.transaction.annotation.Transactional
 import org.springframework.web.bind.annotation.*
@@ -44,9 +46,17 @@ class SearchController @Autowired constructor(
             defaultValue = PublishStatus.DEFAULT_API_PUBLISH_STATUS_SET
         ) status: Array<String>,
         @RequestParam(required = false) sort: String?,
-        @RequestBody apiSearch: ApiSearch
+        @RequestBody apiSearch: ApiSearch,
+        @AuthenticationPrincipal user: Jwt?
     ): HttpEntity<List<CollectionDoc>> {
-        val publishStatuses = status.mapNotNull { PublishStatus.forApiValue(it) }.toSet()
+        if (!appConfig.allowPublicSearching && user === null) {
+            throw GeneralApiException("Unauthorized", HttpStatus.UNAUTHORIZED)
+        }
+
+        val publishStatuses = status.mapNotNull {
+            val status = PublishStatus.forApiValue(it)
+            if (user == null && (status == PublishStatus.Deleted  || status == PublishStatus.Draft)) null else status
+        }.toSet()
         val sortEnum: CollectionSortEnum = CollectionSortEnum.forValueOrDefault(sort)
         val pageable = OffsetPageable(from, size, sortEnum.sort)
 
@@ -85,9 +95,17 @@ class SearchController @Autowired constructor(
         ) status: Array<String>,
         @RequestParam(required = false) sort: String?,
         @RequestParam(required = false) collectionId: String?,
-        @RequestBody apiSearch: ApiSearch
+        @RequestBody apiSearch: ApiSearch,
+        @AuthenticationPrincipal user: Jwt?
     ): HttpEntity<List<RichSkillDoc>> {
-        val publishStatuses = status.mapNotNull { PublishStatus.forApiValue(it) }.toSet()
+        if (!appConfig.allowPublicSearching && user === null) {
+            throw GeneralApiException("Unauthorized", HttpStatus.UNAUTHORIZED)
+        }
+
+        val publishStatuses = status.mapNotNull {
+            val status = PublishStatus.forApiValue(it)
+            if (user == null && (status == PublishStatus.Deleted  || status == PublishStatus.Draft)) null else status
+        }.toSet()
         val sortEnum = sort?.let{SkillSortEnum.forApiValue(it)}
         val pageable = OffsetPageable(offset = from, limit = size, sort = sortEnum?.sort)
 
@@ -132,9 +150,10 @@ class SearchController @Autowired constructor(
         ) status: Array<String>,
         @RequestParam(required = false) sort: String?,
         @PathVariable uuid: String,
-        @RequestBody apiSearch: ApiSearch
+        @RequestBody apiSearch: ApiSearch,
+        @AuthenticationPrincipal user: Jwt?
     ): HttpEntity<List<RichSkillDoc>> {
-        return searchSkills(uriComponentsBuilder, size, from, status, sort, uuid, apiSearch)
+        return searchSkills(uriComponentsBuilder, size, from, status, sort, uuid, apiSearch, user)
     }
 
     @GetMapping(RoutePaths.SEARCH_JOBCODES_PATH, produces = [MediaType.APPLICATION_JSON_VALUE])

diff --git a/api/src/main/kotlin/edu/wgu/osmt/richskill/RichSkillController.kt b/api/src/main/kotlin/edu/wgu/osmt/richskill/RichSkillController.kt
@@ -2,6 +2,7 @@ package edu.wgu.osmt.richskill
 
 import edu.wgu.osmt.HasAllPaginated
 import edu.wgu.osmt.RoutePaths
+import edu.wgu.osmt.api.GeneralApiException
 import edu.wgu.osmt.api.model.*
 import edu.wgu.osmt.auditlog.AuditLog
 import edu.wgu.osmt.auditlog.AuditLogRepository
@@ -36,22 +37,26 @@ class RichSkillController @Autowired constructor(
 
     val keywordDao = KeywordDao.Companion
 
-    override val allPaginatedPath: String = RoutePaths.SKILLS_PATH
+    override val allPaginatedPath: String = RoutePaths.SKILLS_LIST
     override val sortOrderCompanion = SkillSortEnum.Companion
 
-    @GetMapping(RoutePaths.SKILLS_PATH, produces = [MediaType.APPLICATION_JSON_VALUE])
+    @GetMapping(RoutePaths.SKILLS_LIST, produces = [MediaType.APPLICATION_JSON_VALUE])
     @ResponseBody
     override fun allPaginated(
         uriComponentsBuilder: UriComponentsBuilder,
         size: Int,
         from: Int,
         status: Array<String>,
-        sort: String?
+        sort: String?,
+        @AuthenticationPrincipal user: Jwt?
     ): HttpEntity<List<RichSkillDoc>> {
-        return super.allPaginated(uriComponentsBuilder, size, from, status, sort)
+        if (!appConfig.allowPublicLists && user === null) {
+            throw GeneralApiException("Unauthorized", HttpStatus.UNAUTHORIZED)
+        }
+        return super.allPaginated(uriComponentsBuilder, size, from, status, sort, user)
     }
 
-    @PostMapping(RoutePaths.SKILLS_PATH, produces = [MediaType.APPLICATION_JSON_VALUE])
+    @PostMapping(RoutePaths.SKILLS_CREATE, produces = [MediaType.APPLICATION_JSON_VALUE])
     @ResponseBody
     fun createSkills(
         @RequestBody apiSkillUpdates: List<ApiSkillUpdate>,