Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change Cache to Embedded H2 #113

Merged
merged 34 commits into from
May 30, 2022
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
83a8c90
Make clicking "Unknown" language do nothing.
Attacktive May 3, 2022
d57b05c
Merge branch 'tipsy:master' into master
Attacktive May 16, 2022
6bd15d4
Implement user cache using embedded H2.
Attacktive May 18, 2022
84f8914
Separate caching to its own service
Attacktive May 19, 2022
8f5889b
Change UserProfile to be not Serializable
Attacktive May 19, 2022
8347ea6
Fix copy-paste errors.
Attacktive May 19, 2022
e487b1c
Replace Jackson with Gson which reduced boilerplate code by a bit.
Attacktive May 20, 2022
a5c85ce
Introduce an extension function
Attacktive May 20, 2022
bf5cdd5
Remove unnecessary JsonDeserializer class.
Attacktive May 20, 2022
415d70c
Switch back to Jackson as deserializer.
Attacktive May 20, 2022
c97ffd7
Add warning suppression.
Attacktive May 20, 2022
5a929a9
Replace redundant creation of ObjectMapper.
Attacktive May 23, 2022
2f599bb
Replace LocalDateTime with Instant.
Attacktive May 23, 2022
8306d92
Prevent multiple invocations of generateUserProfile.
Attacktive May 23, 2022
0707188
Cache by lazy.
Attacktive May 23, 2022
caaa097
Remove unnecessary JSON deserialization from lookUpInCache.
Attacktive May 23, 2022
c06acf5
Prevent NPE.
Attacktive May 23, 2022
494ed41
Update src/main/kotlin/app/CacheService.kt
Attacktive May 23, 2022
a0f11e7
Revert "Prevent multiple invocations of generateUserProfile."
Attacktive May 23, 2022
34acc07
Remove unnecessary custom deserializer.
Attacktive May 23, 2022
899a28f
Rename function.
Attacktive May 23, 2022
1d04b35
Reduce invocations of CacheService#selectJsonFromDb.
Attacktive May 23, 2022
b072c87
Extract functions from duplicate code.
Attacktive May 23, 2022
80154a0
Update src/main/kotlin/app/CacheService.kt
Attacktive May 24, 2022
8e15906
Remove unnecessary class UserNotLoadableException.
Attacktive May 24, 2022
33d6994
Remove unnecessary lazy initialization.
Attacktive May 24, 2022
797fcfd
Change id values to lowercase in DB.
Attacktive May 24, 2022
849c6e4
Change data type of `timestamp` from 'TIMESTAMP WITH TIME ZONE' to 'T…
Attacktive May 24, 2022
290d234
Use HikariCP.
Attacktive May 24, 2022
afd1253
Update src/main/kotlin/app/util/HikariCpDataSource.kt
Attacktive May 24, 2022
aea8648
Replace string concatenation with multi-line string
Attacktive May 30, 2022
1162a9e
Replace uses of magic numbers
Attacktive May 30, 2022
1312286
Remove intermediate local variable
Attacktive May 30, 2022
e9abd19
Make CacheService have `HikariCpDataSource.connection` as its field
Attacktive May 30, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@
<version>${kotlin.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>2.1.212</version>
<scope>runtime</scope>
</dependency>
</dependencies>

<build>
Expand Down
58 changes: 0 additions & 58 deletions src/main/kotlin/app/Cache.kt

This file was deleted.

101 changes: 101 additions & 0 deletions src/main/kotlin/app/CacheService.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
@file:Suppress("SqlResolve")

package app

import com.fasterxml.jackson.core.JsonParser
import com.fasterxml.jackson.databind.DeserializationContext
import com.fasterxml.jackson.databind.deser.std.StdDeserializer
import com.fasterxml.jackson.databind.module.SimpleModule
import com.fasterxml.jackson.module.kotlin.jacksonObjectMapper
import com.fasterxml.jackson.module.kotlin.readValue
import org.slf4j.LoggerFactory
import java.sql.DriverManager
import java.time.Instant
import java.time.temporal.ChronoUnit
import java.util.Date

object CacheService {
private const val urlToDb = "jdbc:h2:mem:userinfo"
private val log = LoggerFactory.getLogger(CacheService.javaClass)
private val objectMapper = jacksonObjectMapper()
.registerModule(SimpleModule().addDeserializer(Date::class.java, DateDeserializer()))
Attacktive marked this conversation as resolved.
Show resolved Hide resolved

private fun createTableIfAbsent() {
val connection = DriverManager.getConnection(urlToDb)
val statement = connection.createStatement()

statement.execute(
tipsy marked this conversation as resolved.
Show resolved Hide resolved
"CREATE TABLE IF NOT EXISTS userinfo (" +
"id VARCHAR2 PRIMARY KEY," +
"timestamp TIMESTAMP WITH TIME ZONE, " +
Attacktive marked this conversation as resolved.
Show resolved Hide resolved
"data JSON" +
Attacktive marked this conversation as resolved.
Show resolved Hide resolved
")"
)
}

fun lookUpInCache(username: String): String? {
val connection = DriverManager.getConnection(urlToDb)
Attacktive marked this conversation as resolved.
Show resolved Hide resolved

createTableIfAbsent()

val preparedStatement = connection.prepareStatement(
tipsy marked this conversation as resolved.
Show resolved Hide resolved
"SELECT " +
"timestamp, " +
"data " +
"FROM userinfo " +
"WHERE id = ?"
)
preparedStatement.setString(1, username)
Attacktive marked this conversation as resolved.
Show resolved Hide resolved

val result = preparedStatement.executeQuery()
result.use {
// guaranteed to be at most one.
if (it.next()) {
val timestamp = it.getTimestamp(1).toInstant()
val diffInHours = ChronoUnit.HOURS.between(timestamp, Instant.now())
if (diffInHours <= 6) {
val json: String? = it.getString(2)
tipsy marked this conversation as resolved.
Show resolved Hide resolved
if (json != null) {
log.debug("cache hit: {}", json)
}

return json
}
}
}

log.debug("cache miss for username: {}", username)

return null
}

fun getUserFromCache(username: String): UserProfile? {
Attacktive marked this conversation as resolved.
Show resolved Hide resolved
val json = lookUpInCache(username) ?: return null

return objectMapper.readValue<UserProfile>(json)
}
Attacktive marked this conversation as resolved.
Show resolved Hide resolved

fun saveInCache(userProfile: UserProfile) {
val connection = DriverManager.getConnection(urlToDb)

createTableIfAbsent()

val json = objectMapper.writeValueAsString(userProfile)

val preparedStatement = connection.prepareStatement(
"MERGE INTO userinfo (id, timestamp, data) KEY (id) " +
"VALUES (?, CURRENT_TIMESTAMP(), ? FORMAT JSON)"
)

preparedStatement.setString(1, userProfile.user.login)
preparedStatement.setString(2, json)

preparedStatement.execute()
}

private class DateDeserializer: StdDeserializer<Date>(Date::class.java) {
override fun deserialize(jsonParser: JsonParser, context: DeserializationContext): Date {
return Date(jsonParser.readValueAs(Long::class.java))
}
}
}
15 changes: 15 additions & 0 deletions src/main/kotlin/app/UserProfile.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package app

import org.eclipse.egit.github.core.User

data class UserProfile(
val user: User,
val quarterCommitCount: Map<String, Int>,
val langRepoCount: Map<String, Int>,
val langStarCount: Map<String, Int>,
val langCommitCount: Map<String, Int>,
val repoCommitCount: Map<String, Int>,
val repoStarCount: Map<String, Int>,
val repoCommitCountDescriptions: Map<String, String?>,
val repoStarCountDescriptions: Map<String, String?>
Attacktive marked this conversation as resolved.
Show resolved Hide resolved
)
80 changes: 35 additions & 45 deletions src/main/kotlin/app/UserService.kt
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,7 @@ package app
import app.util.CommitCountUtil
import org.eclipse.egit.github.core.Repository
import org.eclipse.egit.github.core.RepositoryCommit
import org.eclipse.egit.github.core.User
import org.slf4j.LoggerFactory
import java.io.Serializable
import java.time.Instant
import java.util.concurrent.ConcurrentHashMap
import java.util.stream.IntStream
import kotlin.streams.toList
Expand All @@ -28,42 +25,17 @@ object UserService {
fun canLoadUser(user: String): Boolean {
val remainingRequests by lazy { GhService.remainingRequests }
val hasFreeRemainingRequests by lazy { remainingRequests > (freeRequestCutoff ?: remainingRequests) }
val userCache by lazy { CacheService.lookUpInCache(user) }
return Config.unrestricted()
|| Cache.contains(user)
|| (userCache != null)
|| hasFreeRemainingRequests
|| (remainingRequests > 0 && hasStarredRepo(user))
}

fun getUserProfile(username: String): UserProfile {
if (Cache.invalid(username)) {
val user = GhService.users.getUser(username)
val repos = GhService.repos.getRepositories(username).filter { !it.isFork && it.size != 0 }
val repoCommits = repos.parallelStream().map { it to commitsForRepo(it).filter { it.author?.login.equals(username, ignoreCase = true) } }.toList().toMap()
val langRepoGrouping = repos.groupingBy { (it.language ?: "Unknown") }

val quarterCommitCount = CommitCountUtil.getCommitsForQuarters(user, repoCommits)
val langRepoCount = langRepoGrouping.eachCount().toList().sortedBy { (_, v) -> -v }.toMap()
val langStarCount = langRepoGrouping.fold(0) { acc, repo -> acc + repo.watchers }.toList().sortedBy { (_, v) -> -v }.toMap()
val langCommitCount = langRepoGrouping.fold(0) { acc, repo -> acc + repoCommits[repo]!!.size }.toList().sortedBy { (_, v) -> -v }.toMap()
val repoCommitCount = repoCommits.map { it.key.name to it.value.size }.toList().sortedBy { (_, v) -> -v }.take(10).toMap()
val repoStarCount = repos.filter { it.watchers > 0 }.map { it.name to it.watchers }.sortedBy { (_, v) -> -v }.take(10).toMap()

val repoCommitCountDescriptions = repoCommitCount.map { it.key to repos.find { r -> r.name == it.key }?.description }.toMap()
val repoStarCountDescriptions = repoStarCount.map { it.key to repos.find { r -> r.name == it.key }?.description }.toMap()

Cache.putUserProfile(UserProfile(
user,
quarterCommitCount,
langRepoCount,
langStarCount,
langCommitCount,
repoCommitCount,
repoStarCount,
repoCommitCountDescriptions,
repoStarCountDescriptions
))
synchronized(username) {
return (CacheService.getUserFromCache(username) ?: generateUserProfile(username))
}
return Cache.getUserProfile(username)!!
}

private fun hasStarredRepo(username: String): Boolean {
Expand Down Expand Up @@ -97,18 +69,36 @@ object UserService {
listOf()
}

}
private fun generateUserProfile(username: String): UserProfile {
Attacktive marked this conversation as resolved.
Show resolved Hide resolved
val user = GhService.users.getUser(username)
val repos = GhService.repos.getRepositories(username).filter { !it.isFork && it.size != 0 }
val repoCommits = repos.parallelStream().map { it to commitsForRepo(it).filter { it.author?.login.equals(username, ignoreCase = true) } }.toList().toMap()
val langRepoGrouping = repos.groupingBy { (it.language ?: "Unknown") }

val quarterCommitCount = CommitCountUtil.getCommitsForQuarters(user, repoCommits)
val langRepoCount = langRepoGrouping.eachCount().toList().sortedBy { (_, v) -> -v }.toMap()
val langStarCount = langRepoGrouping.fold(0) { acc, repo -> acc + repo.watchers }.toList().sortedBy { (_, v) -> -v }.toMap()
val langCommitCount = langRepoGrouping.fold(0) { acc, repo -> acc + repoCommits[repo]!!.size }.toList().sortedBy { (_, v) -> -v }.toMap()
val repoCommitCount = repoCommits.map { it.key.name to it.value.size }.toList().sortedBy { (_, v) -> -v }.take(10).toMap()
val repoStarCount = repos.filter { it.watchers > 0 }.map { it.name to it.watchers }.sortedBy { (_, v) -> -v }.take(10).toMap()

val repoCommitCountDescriptions = repoCommitCount.map { it.key to repos.find { r -> r.name == it.key }?.description }.toMap()
val repoStarCountDescriptions = repoStarCount.map { it.key to repos.find { r -> r.name == it.key }?.description }.toMap()

data class UserProfile(
val user: User,
val quarterCommitCount: Map<String, Int>,
val langRepoCount: Map<String, Int>,
val langStarCount: Map<String, Int>,
val langCommitCount: Map<String, Int>,
val repoCommitCount: Map<String, Int>,
val repoStarCount: Map<String, Int>,
val repoCommitCountDescriptions: Map<String, String?>,
val repoStarCountDescriptions: Map<String, String?>
) : Serializable {
val timeStamp = Instant.now().toEpochMilli()
val userProfile = UserProfile(
user,
quarterCommitCount,
langRepoCount,
langStarCount,
langCommitCount,
repoCommitCount,
repoStarCount,
repoCommitCountDescriptions,
repoStarCountDescriptions
)

CacheService.saveInCache(userProfile)
Attacktive marked this conversation as resolved.
Show resolved Hide resolved

return userProfile;
}
}