Reject dataset uploads if organization storage quota is exceeded

[fm3]

Steps to test:

• set reportUsedStorage.enabled = true for your datastore in application.conf (note that if your organization is super new, you may want to lower rescanInterval as well so that you don’t have to wait 24 hours. initial data sets lastScanTime to creationTime.)
• upload a dataset, should still work
• in the local db, limit your organization’s includedStorage field, retry, should be rejected

Issues:

• fixes Block uploading datasets when storage quota is exceeded #6808

---

• Updated changelog
• Considered common edge cases

[fm3]

@philippotto It is not simple for the backend to make the top-level error message very explanatory. Instead, the user currently gets this.

If I remember correctly, there are already some instances where the frontend does exists-checks in the error message to add another message to the user, do you think something like that could work here too?

Additionaly, I noticed that the state of the upload page is kind of stuck if the request fails like so. Maybe it would be possible to (somewhat?) reset it in this case, but not super important I guess.

[fm3]

@philippotto ping :) Could you estimate if this would be a lot of effort? If so, we can also merge this as is.

[philippotto]

Sorry, I forgot about that! Looking into it right now :)

[philippotto]

As discussed, I went with the early storage check so that the button is disabled (and an alert box is shown). Will look like this:

[daniel-wer]

LGTM! I couldn't get the storage reporter to run even after adapting both application.conf keys mentioned in the PR description. The organization page always reported 0.0 GB 🤔 Still, the normal dataset upload succeeded. Also, I investigated and fixed another issue which I encountered during testing.

[fm3]

Huh, this is weird. Could it be that the displayed number is rounded down to 0.0? what does api/organizations contain?

[daniel-wer]

Huh, this is weird. Could it be that the displayed number is rounded down to 0.0? what does api/organizations contain?

Good point! Indeed the response contains "usedStorageBytes": 10535441467 which is 10 GB. If the limit is displayed in TB with one digit after the decimal point this seems to be caused by rounding, as you suspected. Sorry for the false alarm, then :)

[daniel-wer]

I retested and everything works as expected 🎉

[philippotto]

The organization page always reported 0.0 GB

I suppose you mean 0.0 TB? Otherwise, the rounding error explained in the following comments doesn't make sense to me 😅

[daniel-wer]

I suppose you mean 0.0 TB? Otherwise, the rounding error explained in the following comments doesn't make sense to me

In fact, the organization page didn't show any unit, because the upper limit was infinity. So at that point I assumed the unit might be GB, but it turned out to be TB.

[fm3]

might be worth restructuring that display to avoid this kind of confusion for future users. Might make sense to separate the units for used vs allowed, but that does not play well with the current diagram style 🤷