Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix AWS JSON protocol errors by harmonising relevant AWS libs #52

Merged
merged 1 commit into from
Jan 8, 2025
Merged

Fix AWS JSON protocol errors by harmonising relevant AWS libs #52

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,25 +35,23 @@ releaseProcess := Seq[ReleaseStep](
resolvers ++= Resolver.sonatypeOssRepos("releases")

libraryDependencies ++= Seq(
"com.gu" %% "content-api-models-scala" % "25.0.0",
"com.gu" %% "content-api-models-scala" % "26.0.0",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found in apple-news that this would need to be upgraded to fit in with the upgrade of the FAPI client.

[info] Resolved apple-news_2.13 dependencies
[info] Fetching artifacts of apple-news_2.13
[info] Fetched artifacts of apple-news_2.13
[error] stack trace is suppressed; run last update for the full output
[error] (update) found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[error]
[error] 	* com.gu:content-api-models-scala_2.13:26.0.0 (early-semver) is selected over 25.0.0
[error] 	    +- com.gu:content-api-client_2.13:33.0.0              (depends on 26.0.0)
[error] 	    +- com.gu:content-api-client-default_2.13:33.0.0      (depends on 26.0.0)
[error] 	    +- com.gu:content-api-firehose-client_2.13:1.0.26-PREVIEW.upgrade-aws-sdk-version-to-avoid-json-protocol-conflicts.2025-01-08T1159.b870d366 (depends on 25.0.0)

"com.gu" %% "thrift-serializer" % "5.0.5",
"software.amazon.kinesis" % "amazon-kinesis-client" % "2.6.0",
"software.amazon.kinesis" % "amazon-kinesis-client" % "3.0.1",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not read this document about upgrading from KCL v2 to v3:

https://docs.aws.amazon.com/streams/latest/dev/kcl-migration-from-2-3.html

...this is unfortunate as it probably means that the release that included this PR (v1.0.26) is probably missing at least one crucial setting, and so should be avoided.

At the very least, the increased IAM permissions required by KCL v3 means that https://github.com/guardian/apple-news/pull/398 failed:

"com.typesafe.scala-logging" %% "scala-logging" % "3.9.5",
"com.twitter" %% "scrooge-core" % "21.12.0",
"org.scalatest" %% "scalatest" % "3.2.19" % Test
)
) ++ Seq("aws-json-protocol", "kinesis").map(artifact => "software.amazon.awssdk" % artifact % "2.29.23")

val jacksonVersion = "2.17.2"
dependencyOverrides ++= Seq(
"com.charleskorn.kaml" % "kaml" % "0.53.0",
"com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion,
"com.fasterxml.jackson.core" % "jackson-annotations" % jacksonVersion,
"com.fasterxml.jackson.core" % "jackson-core" % jacksonVersion,
"software.amazon.awssdk" % "netty-nio-client" % "2.26.25",
Copy link
Member Author

@rtyley rtyley Dec 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"software.amazon.awssdk" % "netty-nio-client" had a dependency override added first with #26, then to version 2.26.25 with #49 in October 2024.

However, as in this PR we're updating all relevant software.amazon.awssdk libraries to 2.29.23, past 2.26.25, the manual upgrade is no longer necessary.

"org.json" % "json" % "20231013",
"org.xerial.snappy" % "snappy-java" % "1.1.10.4",
"org.apache.commons" % "commons-compress" % "1.26.0",
"com.amazon.ion" % "ion-java" % "1.10.5",//overriding until a version of amazon-kinesis-client is available that removes the ion-java vulnerability
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ion-java had a dependency override added with #42 in March 2024. ion-java was a dependency of AWS SDK v1 (com.amazonaws:aws-java-sdk-core:1.12.151). However, release 3.0.0 of the Amazon Kinesis Client Library for Java removed the last transitory dependency of the KCL on AWS SDK v1, and so content-api-firehose-client no longer depends upon it.

"software.amazon.glue" % "schema-registry-serde" % "1.1.19", //overriding until a version of amazon-kinesis-client is available that removes the ion-java vulnerability
"org.apache.avro" % "avro" % "1.11.4", //overriding until a version of amazon-kinesis-client / glue-schema-registry-common is available that removes the avro vuln
"com.google.protobuf" % "protobuf-java" % "3.25.5" //overriding until a version of amazon-kinesis-client is available that removes the protobuf vuln
Expand Down
2 changes: 2 additions & 0 deletions project/plugins.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,5 @@ addSbtPlugin("org.scalariform" % "sbt-scalariform" % "1.8.3")



addDependencyTreePlugin

Loading