Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce TaskType Resolution Restrictions for Multi-Res Volume Tasks #4891

Merged
merged 47 commits into from
Nov 5, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
21f009f
[WIP] enforce task type magnification settings in tasks
fm3 Oct 23, 2020
a377520
pass allowedMagnifications to rpc methods
fm3 Oct 23, 2020
fa6613c
pass magnifications as query string
fm3 Oct 23, 2020
ecc3035
respect resolution restrictions for volume tracings.
fm3 Oct 26, 2020
6855901
add option to downsample existing volume tracing
fm3 Oct 27, 2020
4094f32
remove outdated assertion, update resolution list after downsampling
fm3 Oct 27, 2020
d4890eb
sleep 10s
fm3 Oct 27, 2020
2a65db2
remove sleep and bump dev-proxy-timeout to 5 min
philippotto Oct 27, 2020
3c0205a
fix task creation with resolution restrictions
fm3 Oct 28, 2020
219a47d
Merge branch 'multires-volumes' into multires-volumes-restricted
fm3 Oct 28, 2020
3282035
use inclusive check
fm3 Oct 28, 2020
0fcad72
Merge branch 'multires-volumes-restricted' of github.com:scalablemind…
philippotto Oct 28, 2020
6766c92
when uploading zips with differing resolution sets, fail
fm3 Oct 28, 2020
89c622a
Merge branch 'multires-volumes-restricted' of github.com:scalablemind…
fm3 Oct 28, 2020
09004c5
remove unused import
fm3 Oct 28, 2020
a2561e0
refresh snapshots (tracings contain organization name)
fm3 Oct 28, 2020
2643726
write header.wkw to every mag in volume download, fix anisotropic dir…
fm3 Oct 29, 2020
aca011d
add resolution-samenss assertion in volume dnd import. fix flow types
fm3 Oct 29, 2020
bc7178c
fix restriction passing in initial data
fm3 Oct 29, 2020
695d101
Add button to trigger downsampling of volume annotation.
philippotto Oct 28, 2020
d0f673b
Merge branch 'multires-volumes-restricted' of github.com:scalablemind…
philippotto Oct 29, 2020
022e0f8
fix parameter passing in duplicate rpc
fm3 Nov 2, 2020
2dbf12e
avoid undefined behavior of bucket iterator by checking hasNext. do n…
fm3 Nov 2, 2020
e92bf9a
make tasktype resolution restrictions immutable. sort resolution list
fm3 Nov 2, 2020
5ab1743
update migration guide
fm3 Nov 3, 2020
9f15bd2
remove debug output
fm3 Nov 3, 2020
09fb073
improve initiate-volume-downsampling modal and move its button to the…
philippotto Nov 3, 2020
2171f5d
Merge branch 'multires-volumes-restricted' of github.com:scalablemind…
philippotto Nov 3, 2020
0c7c41b
show toast when mag-restriction is violated instead of prohibiting th…
philippotto Nov 3, 2020
a95d9a8
Merge branch 'multires-volumes' into multires-volumes-restricted
philippotto Nov 3, 2020
d8d622a
Merge branch 'multires-volumes' into multires-volumes-restricted
philippotto Nov 3, 2020
10143cd
don't allow editing mag-restrictions in already created tasktype
philippotto Nov 3, 2020
6ea304d
remove some more debug output, unify variable names
fm3 Nov 4, 2020
5535ba4
forbid creating new nodes or using volume tools when the current mag …
philippotto Nov 4, 2020
1a681cb
Merge branch 'multires-volumes-restricted' of github.com:scalablemind…
philippotto Nov 4, 2020
a5edb7f
clean up
philippotto Nov 4, 2020
a5152b3
Update MIGRATIONS.unreleased.md
fm3 Nov 4, 2020
8b4e07b
fix styling of button-link
philippotto Nov 4, 2020
0722933
Merge branch 'multires-volumes-restricted' of github.com:scalablemind…
philippotto Nov 4, 2020
3888a91
use ListBuffer instead of HashSet as per pr feedback
fm3 Nov 4, 2020
66c43ea
integrate some feedback
philippotto Nov 4, 2020
4a818db
Merge branch 'multires-volumes-restricted' of github.com:scalablemind…
philippotto Nov 4, 2020
bf48913
Apply suggestions from code review
philippotto Nov 4, 2020
fa84163
remove unnecessary SET_TOOL dependence
philippotto Nov 4, 2020
e3bea0e
Merge branch 'multires-volumes-restricted' of github.com:scalablemind…
philippotto Nov 4, 2020
7e34b0d
do map ids if initialLargestSegmentId != 0
fm3 Nov 5, 2020
1368ae2
Merge branch 'multires-volumes' into multires-volumes-restricted
fm3 Nov 5, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion MIGRATIONS.unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,13 @@ This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
User-facing changes are documented in the [changelog](CHANGELOG.released.md).

## Unreleased
- To convert individual legacy volume annotations to multi-resolution volume annotations, download and re-upload them (upload may take a while).
- As volume annotations in arbitrary magnifications are now supported and the behavior of magnification restrictions of tasks has changed (allow full zoom, but disable tools unless in correct magnification), you may want to restrict all volume and hybrid task types to mag 1 to achieve the old behavior (mag1-only):
```
update webknossos.tasktypes
set settings_allowedmagnifications = '{"min":1,"max":1,"shouldRestrict":true}'
where (tracingtype = 'volume' or tracingtype = 'hybrid')
and (settings_allowedmagnifications is null or settings_allowedmagnifications::json->>'shouldRestrict'='false');
```

### Postgres Evolutions:
- [057-add-layer-specific-view-configs.sql](conf/evolutions/056-add-layer-specific-view-configs.sql)
Expand Down
15 changes: 14 additions & 1 deletion app/controllers/AnnotationController.scala
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,20 @@ class AnnotationController @Inject()(
for {
_ <- bool2Fox(AnnotationType.Explorational.toString == typ) ?~> "annotation.makeHybrid.explorationalsOnly"
annotation <- provider.provideAnnotation(typ, id, request.identity)
_ <- annotationService.makeAnnotationHybrid(annotation) ?~> "annotation.makeHybrid.failed"
organization <- organizationDAO.findOne(request.identity._organization)
_ <- annotationService.makeAnnotationHybrid(annotation, organization.name) ?~> "annotation.makeHybrid.failed"
updated <- provider.provideAnnotation(typ, id, request.identity)
json <- annotationService.publicWrites(updated, Some(request.identity)) ?~> "annotation.write.failed"
} yield {
JsonOk(json)
}
}

def downsample(typ: String, id: String) = sil.SecuredAction.async { implicit request =>
for {
_ <- bool2Fox(AnnotationType.Explorational.toString == typ) ?~> "annotation.downsample.explorationalsOnly"
annotation <- provider.provideAnnotation(typ, id, request.identity)
_ <- annotationService.downsampleAnnotation(annotation) ?~> "annotation.downsample.failed"
updated <- provider.provideAnnotation(typ, id, request.identity)
json <- annotationService.publicWrites(updated, Some(request.identity)) ?~> "annotation.write.failed"
} yield {
Expand Down
75 changes: 49 additions & 26 deletions app/controllers/TaskController.scala
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ import com.scalableminds.util.geometry.{BoundingBox, Point3D, Vector3D}
import com.scalableminds.util.mvc.ResultBox
import com.scalableminds.util.tools.{Fox, FoxImplicits, JsonHelper}
import com.scalableminds.webknossos.tracingstore.SkeletonTracing.{SkeletonTracing, SkeletonTracingOpt, SkeletonTracings}
import com.scalableminds.webknossos.tracingstore.VolumeTracing.{VolumeTracing, VolumeTracingOpt, VolumeTracings}
import com.scalableminds.webknossos.tracingstore.VolumeTracing.VolumeTracing
import com.scalableminds.webknossos.tracingstore.tracings.volume.ResolutionRestrictions
import com.scalableminds.webknossos.tracingstore.tracings.{ProtoGeometryImplicits, TracingType}
import javax.inject.Inject
import models.annotation.nml.NmlResults.NmlParseResult
import models.annotation.nml.NmlService
import models.annotation._
import models.binary.{DataSet, DataSetDAO, DataSetService}
import models.binary.{DataSet, DataSetDAO}
import models.project.{Project, ProjectDAO}
import models.task._
import models.team.{Team, TeamDAO}
Expand Down Expand Up @@ -74,7 +74,6 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
dataSetDAO: DataSetDAO,
userTeamRolesDAO: UserTeamRolesDAO,
userService: UserService,
dataSetService: DataSetService,
tracingStoreService: TracingStoreService,
teamDAO: TeamDAO,
taskDAO: TaskDAO,
Expand All @@ -88,7 +87,7 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
with ProtoGeometryImplicits
with FoxImplicits {

val MAX_OPEN_TASKS = conf.WebKnossos.Tasks.maxOpenPerUser
private val MAX_OPEN_TASKS: Int = conf.WebKnossos.Tasks.maxOpenPerUser

def read(taskId: String) = sil.SecuredAction.async { implicit request =>
for {
Expand All @@ -114,8 +113,9 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
} yield result
}

def duplicateAllBaseTracings(taskParametersList: List[TaskParameters],
organizationId: ObjectId)(implicit ctx: DBAccessContext, m: MessagesProvider) =
private def duplicateAllBaseTracings(taskParametersList: List[TaskParameters], organizationId: ObjectId)(
implicit ctx: DBAccessContext,
m: MessagesProvider): Fox[List[TaskParameters]] =
Fox.serialCombined(taskParametersList)(
params =>
Fox
Expand All @@ -141,9 +141,10 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
annotation: Annotation,
params: TaskParameters,
tracingStoreClient: TracingStoreRpcClient,
organizationId: ObjectId)(implicit ctx: DBAccessContext, m: MessagesProvider): Fox[String] =
organizationId: ObjectId,
resolutionRestrictions: ResolutionRestrictions)(implicit ctx: DBAccessContext, m: MessagesProvider): Fox[String] =
annotation.volumeTracingId
.map(id => tracingStoreClient.duplicateVolumeTracing(id))
.map(id => tracingStoreClient.duplicateVolumeTracing(id, resolutionRestrictions = resolutionRestrictions))
.getOrElse(
annotationService
.createVolumeTracingBase(
Expand All @@ -152,13 +153,15 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
params.boundingBox,
params.editPosition,
params.editRotation,
false
volumeShowFallbackLayer = false,
resolutionRestrictions = resolutionRestrictions
)
.flatMap(tracingStoreClient.saveVolumeTracing(_)))
.flatMap(tracingStoreClient.saveVolumeTracing(_, resolutionRestrictions = resolutionRestrictions)))

def duplicateBaseTracings(baseAnnotation: BaseAnnotation, taskParameters: TaskParameters, organizationId: ObjectId)(
implicit ctx: DBAccessContext,
m: MessagesProvider) = {
private def duplicateBaseTracings(
baseAnnotation: BaseAnnotation,
taskParameters: TaskParameters,
organizationId: ObjectId)(implicit ctx: DBAccessContext, m: MessagesProvider): Fox[BaseAnnotation] = {

@SuppressWarnings(Array("TraversableHead")) // We check if nonCancelledTaskAnnotations are empty before so head always works
def checkForTask(taskId: ObjectId): Fox[Annotation] =
Expand Down Expand Up @@ -196,13 +199,16 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
duplicateSkeletonTracingOrCreateSkeletonTracingBase(annotation, taskParameters, tracingStoreClient).map(Some(_))
else Fox.successful(None)
newVolumeId <- if (taskType.tracingType == TracingType.volume || taskType.tracingType == TracingType.hybrid)
duplicateVolumeTracingOrCreateVolumeTracingBase(annotation, taskParameters, tracingStoreClient, organizationId)
.map(Some(_))
duplicateVolumeTracingOrCreateVolumeTracingBase(annotation,
taskParameters,
tracingStoreClient,
organizationId,
taskType.settings.resolutionRestrictions).map(Some(_))
else Fox.successful(None)
} yield BaseAnnotation(baseAnnotationIdValidated.id, newSkeletonId, newVolumeId)
}

def createTaskSkeletonTracingBases(paramsList: List[TaskParameters])(
private def createTaskSkeletonTracingBases(paramsList: List[TaskParameters])(
implicit ctx: DBAccessContext,
m: MessagesProvider): Fox[List[Option[SkeletonTracing]]] =
Fox.serialCombined(paramsList) { params =>
Expand All @@ -221,7 +227,7 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
} yield skeletonTracingOpt
}

def createTaskVolumeTracingBases(paramsList: List[TaskParameters], organizationId: ObjectId)(
private def createTaskVolumeTracingBases(paramsList: List[TaskParameters], organizationId: ObjectId)(
implicit ctx: DBAccessContext,
m: MessagesProvider): Fox[List[Option[(VolumeTracing, Option[File])]]] =
Fox.serialCombined(paramsList) { params =>
Expand All @@ -236,7 +242,8 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
params.boundingBox,
params.editPosition,
params.editRotation,
volumeShowFallbackLayer = false
volumeShowFallbackLayer = false,
resolutionRestrictions = taskType.settings.resolutionRestrictions
)
.map(v => Some((v, None)))
} else Fox.successful(None)
Expand Down Expand Up @@ -296,7 +303,8 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
params.boundingBox,
params.editPosition,
params.editRotation,
volumeShowFallbackLayer = false
volumeShowFallbackLayer = false,
resolutionRestrictions = taskType.settings.resolutionRestrictions
)
.map(v => (v, None)))

Expand Down Expand Up @@ -449,6 +457,7 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
bool2Fox(fullTasks.forall(tuple => tuple._1.baseAnnotation.isDefined || tuple._2.isDefined || tuple._3.isDefined))

def assertAllOnSameDataset(firstDatasetName: String): Fox[String] = {
@scala.annotation.tailrec
def allOnSameDatasetIter(
requestedTasksRest: List[(TaskParameters, Option[SkeletonTracing], Option[(VolumeTracing, Option[File])])],
dataSetName: String): Boolean =
Expand Down Expand Up @@ -490,12 +499,8 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
case _ => savedId
}
}
volumeTracingIds: List[Box[Option[String]]] <- Fox.sequence(requestedTasks.map(_.map(_._3)).map {
case Full(Some((tracing, initialFile))) =>
tracingStoreClient.saveVolumeTracing(tracing, initialFile).map(Some(_))
case f: Failure => box2Fox(f)
case _ => Fox.successful(None)
})
volumeTracingIds: List[Box[Option[String]]] <- Fox.sequence(
requestedTasks.map(requestedTask => saveVolumeTracingIfPresent(requestedTask, tracingStoreClient)))
skeletonTracingsIdsMerged = mergeTracingIds((requestedTasks.map(_.map(_._1)), skeletonTracingIds).zipped.toList,
isSkeletonId = true)
volumeTracingsIdsMerged = mergeTracingIds((requestedTasks.map(_.map(_._1)), volumeTracingIds).zipped.toList,
Expand Down Expand Up @@ -526,6 +531,24 @@ class TaskController @Inject()(annotationDAO: AnnotationDAO,
} yield Ok(Json.toJson(result))
}

private def saveVolumeTracingIfPresent(
requestedTaskBox: Box[(TaskParameters, Option[SkeletonTracing], Option[(VolumeTracing, Option[File])])],
tracingStoreClient: TracingStoreRpcClient)(implicit ctx: DBAccessContext): Fox[Option[String]] =
requestedTaskBox.map { tuple =>
(tuple._1, tuple._3)
} match {
case Full((params: TaskParameters, Some((tracing, initialFile)))) =>
for {
taskTypeIdValidated <- ObjectId.parse(params.taskTypeId) ?~> "taskType.id.invalid"
taskType <- taskTypeDAO.findOne(taskTypeIdValidated) ?~> "taskType.notFound"
saveResult <- tracingStoreClient
.saveVolumeTracing(tracing, initialFile, resolutionRestrictions = taskType.settings.resolutionRestrictions)
.map(Some(_))
} yield saveResult
case f: Failure => box2Fox(f)
case _ => Fox.successful(None)
}

private def warnIfTeamHasNoAccess(requestedTasks: List[TaskParameters], dataSet: DataSet)(
implicit ctx: DBAccessContext): Fox[List[String]] = {
val projectNames = requestedTasks.map(_.projectName).distinct
Expand Down
1 change: 1 addition & 0 deletions app/controllers/TaskTypeController.scala
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ class TaskTypeController @Inject()(taskTypeDAO: TaskTypeDAO,
taskTypeIdValidated <- ObjectId.parse(taskTypeId) ?~> "taskType.id.invalid"
taskType <- taskTypeDAO.findOne(taskTypeIdValidated) ?~> "taskType.notFound" ~> NOT_FOUND
_ <- bool2Fox(taskTypeFromForm.tracingType == taskType.tracingType) ?~> "taskType.tracingTypeImmutable"
_ <- bool2Fox(taskTypeFromForm.settings.allowedMagnifications == taskType.settings.allowedMagnifications) ?~> "taskType.allowedMagnificationsImmutable"
updatedTaskType = taskTypeFromForm.copy(_id = taskType._id)
_ <- Fox.assertTrue(userService.isTeamManagerOrAdminOf(request.identity, taskType._team)) ?~> "notAllowed" ~> FORBIDDEN
_ <- Fox
Expand Down
11 changes: 5 additions & 6 deletions app/controllers/WKTracingStoreController.scala
Original file line number Diff line number Diff line change
Expand Up @@ -56,15 +56,14 @@ class WKTracingStoreController @Inject()(tracingStoreService: TracingStoreServic
if (annotation.state == Finished) Fox.failure("annotation already finshed")
else Fox.successful(())

def dataSource(name: String, organizationName: String, dataSetName: String): Action[AnyContent] = Action.async {
implicit request =>
def dataSource(name: String, organizationName: Option[String], dataSetName: String): Action[AnyContent] =
Action.async { implicit request =>
tracingStoreService.validateAccess(name) { _ =>
implicit val ctx: DBAccessContext = GlobalAccessContext
val organizationNameOpt = if (organizationName == "") None else Some(organizationName)
for {
organizationIdOpt <- Fox.runOptional(organizationNameOpt) {
organizationIdOpt <- Fox.runOptional(organizationName) {
organizationDAO.findOneByName(_)(GlobalAccessContext).map(_._id)
} ?~> Messages("organization.notFound", organizationNameOpt.getOrElse("")) ~> NOT_FOUND
} ?~> Messages("organization.notFound", organizationName.getOrElse("")) ~> NOT_FOUND
organizationId <- Fox.fillOption(organizationIdOpt) {
dataSetDAO.getOrganizationForDataSet(dataSetName)(GlobalAccessContext)
} ?~> Messages("dataSet.noAccess", dataSetName) ~> FORBIDDEN
Expand All @@ -74,5 +73,5 @@ class WKTracingStoreController @Inject()(tracingStoreService: TracingStoreServic
dataSource <- dataSetService.dataSourceFor(dataSet)
} yield Ok(Json.toJson(dataSource))
}
}
}
}
Loading