Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Volume annotation download: zip with BEST_SPEED #6036

Merged
merged 9 commits into from
Feb 14, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
- Added a button next to the histogram which adapts the contrast and brightness to the currently visible data. [#5961](https://github.com/scalableminds/webknossos/pull/5961)
- Running uploads can now be cancelled. [#5958](https://github.com/scalableminds/webknossos/pull/5958)
- Annotations with multiple volume layers can now be uploaded. (Note that merging multiple annotations with multiple volume layers each is not supported.) [#6028](https://github.com/scalableminds/webknossos/pull/6028)
- Decrease volume annotation download latency by using a different compression level. [#6036](https://github.com/scalableminds/webknossos/pull/6036)

### Changed
- Upgraded webpack build tool to v5 and all other webpack related dependencies to their latest version. Enabled persistent caching which speeds up server restarts during development as well as production builds. [#5969](https://github.com/scalableminds/webknossos/pull/5969)
Expand Down
2 changes: 2 additions & 0 deletions app/controllers/AnnotationIOController.scala
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package controllers

import java.io.{BufferedOutputStream, File, FileOutputStream}
import java.util.zip.Deflater

import akka.actor.ActorSystem
import akka.stream.Materializer
Expand Down Expand Up @@ -377,6 +378,7 @@ Expects:
case (volumeLayer, index) =>
volumeLayer.volumeDataOpt.foreach { volumeData =>
val dataZipName = volumeLayer.volumeDataZipName(index, fetchedVolumeLayers.length == 1)
zipper.stream.setLevel(Deflater.BEST_SPEED)
zipper.addFileFromBytes(dataZipName, volumeData)
}
}
Expand Down
6 changes: 5 additions & 1 deletion util/src/main/scala/com/scalableminds/util/io/ZipIO.scala
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,12 @@ object ZipIO extends LazyLogging {
def zip(sources: List[NamedStream], out: OutputStream)(implicit ec: ExecutionContext): Future[Unit] =
zip(sources.toIterator, out)

def zip(sources: Iterator[NamedStream], out: OutputStream)(implicit ec: ExecutionContext): Future[Unit] = {
def zip(sources: Iterator[NamedStream], out: OutputStream, level: Int = -1)(
implicit ec: ExecutionContext): Future[Unit] = {
val zip = startZip(out)
if (level != -1) {
zip.stream.setLevel(level)
}
if (sources.nonEmpty) {
for {
_ <- zipIterator(sources, zip)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,10 @@ import play.api.libs.Files
import play.api.libs.Files.TemporaryFileCreator
import play.api.libs.iteratee.Enumerator
import play.api.libs.json.{JsObject, JsValue, Json}

import java.io._
import java.nio.file.Paths
import java.util.zip.Deflater

import scala.collection.mutable
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future
Expand Down Expand Up @@ -220,13 +221,16 @@ class VolumeTracingService @Inject()(
val buckets: Iterator[NamedStream] =
new WKWBucketStreamSink(dataLayer)(dataLayer.bucketProvider.bucketStream(Some(tracing.version)))

val zipResult = ZipIO.zip(buckets, os)
val before = System.currentTimeMillis()
val zipResult = ZipIO.zip(buckets, os, level = Deflater.BEST_SPEED)

zipResult.onComplete {
case failure: scala.util.Failure[Unit] =>
logger.debug(
s"Failed to send zipped volume data for $tracingId: ${TextUtils.stackTraceAsString(failure.exception)}")
case _: scala.util.Success[Unit] => logger.debug(s"Successfully sent zipped volume data for $tracingId")
case _: scala.util.Success[Unit] =>
val after = System.currentTimeMillis()
logger.info(s"Zipping volume data for $tracingId took ${after - before} ms")
}
zipResult
}
Expand Down