Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Add support for uploading books #492

Merged
merged 35 commits into from
Nov 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
f0e4c61
Add janky test interface
JMicheli Oct 12, 2024
f65615a
Add shell of endpoint and configurable mounting
JMicheli Oct 12, 2024
1b9894d
Add test API call.
JMicheli Oct 14, 2024
1c5c5b7
Single-file upload working, permissions cause 500 error
JMicheli Oct 14, 2024
a637ec5
Fix permissions
JMicheli Oct 14, 2024
0f7db35
Adjust test code to fit new sdk.
JMicheli Oct 15, 2024
ea7494f
Add toolbar buttons (non-functional)
JMicheli Oct 18, 2024
b4361e0
wip: design upload modal
aaronleopold Oct 19, 2024
ac43e78
Add separately configurable max file upload size.
JMicheli Oct 19, 2024
23db5d2
wip: typed multipart
aaronleopold Oct 19, 2024
85a5a88
Doc comments for new config vars.
JMicheli Oct 20, 2024
2daf39e
Move config endpoints to base of api.
JMicheli Oct 21, 2024
597f204
Add note to get_server_config
JMicheli Oct 21, 2024
c940d0c
Working file placement test.
JMicheli Oct 21, 2024
6c6c871
Add place_at handling and utoipa annotations.
JMicheli Oct 21, 2024
7d8da9f
Get upload series path working
JMicheli Oct 21, 2024
9a3638c
wip: upload ui and progress tracking
aaronleopold Oct 22, 2024
118ef2a
Get series upload working with single zip.
JMicheli Oct 24, 2024
cc177a0
Correct series upload implementation
JMicheli Oct 24, 2024
8f44ca2
locale, small ui adjustments
aaronleopold Oct 25, 2024
f138e39
Merge remote-tracking branch 'origin/experimental' into jm/uploader
aaronleopold Oct 25, 2024
27014c4
fix autobump after merge, no time for that
aaronleopold Oct 25, 2024
3f8b102
small ui-focused changes
aaronleopold Oct 26, 2024
e19b927
Add series validation logic and clean up file
JMicheli Oct 27, 2024
94d2969
Add scanning after upload and support additional file types.
JMicheli Oct 28, 2024
7089771
Fix error causing checks to fail
JMicheli Oct 28, 2024
331d955
Now it's fixed.
JMicheli Oct 28, 2024
03027b3
Okay... now?
JMicheli Oct 28, 2024
d401fea
Fix typo in doc comment.
JMicheli Oct 28, 2024
418aaf1
fix fetch upload config without permission
aaronleopold Oct 28, 2024
d8002b2
fix unsorted imports lint
aaronleopold Oct 28, 2024
72b8fe1
Address review comments (incomplete)
JMicheli Oct 29, 2024
2d81100
Add validation logic.
JMicheli Oct 29, 2024
d5fa8d1
Address async comment.
JMicheli Oct 30, 2024
9bcb32e
fix context error and hide series upload for series files
aaronleopold Oct 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
242 changes: 161 additions & 81 deletions Cargo.lock

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ derive_builder = "0.20.0"
chrono = { version = "0.4.38", features = ["serde"] }
futures = "0.3.30"
futures-util = "0.3.30"
infer = "0.16.0"
itertools = "0.13.0"
keyring = { version = "3.4.0", features = ["apple-native", "windows-native", "sync-secret-service"] }
lettre = { version = "0.11.4", default-features = false, features = [
Expand Down Expand Up @@ -66,3 +67,5 @@ urlencoding = "2.1.3"
proc-macro2 = "1.0.87"
quote = "1.0.37"
syn = "2.0.79"
# Note: We need to keep this downgraded for the time being. See https://github.com/stumpapp/stump/issues/427#issuecomment-2332857700
zip = { version = "=1.1.3", features = ["deflate"], default-features = false }
4 changes: 4 additions & 0 deletions apps/server/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ axum = { version = "0.7.5", features = [
"multipart",
] }
axum-macros = "0.4.1"
axum_typed_multipart = "0.13.1"
axum-extra = { version = "0.9.3", features = [
"typed-header",
"query"
Expand All @@ -22,6 +23,7 @@ cli = { path = "../../crates/cli" }
chrono = { workspace = true }
futures-util = { workspace = true }
hyper = "0.14.27"
infer = { workspace = true }
jsonwebtoken = "9.3.0"
linemux = { git = "https://github.com/jmagnuson/linemux.git", rev = "acaafc602afac5d7a9cd3e087dafc937cac1e364" }
local-ip-address = "0.6.2"
Expand All @@ -36,6 +38,7 @@ serde-untagged = "0.1.2"
serde_with = { workspace = true }
specta = { workspace = true }
stump_core = { path = "../../core" }
tempfile = "3.13.0"
tower-http = { version = "0.5.2", features = [
"fs",
"cors",
Expand All @@ -52,6 +55,7 @@ tracing = { workspace = true }
urlencoding = { workspace = true }
utoipa = { version = "4.2.3", features = ["axum_extras"] }
utoipa-swagger-ui = { version = "7.1.0", features = ["axum"] }
zip = { workspace = true }

[dev-dependencies]
axum-test = "15.3.1"
Expand Down
7 changes: 7 additions & 0 deletions apps/server/src/routers/api/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
NamedType,
};

use stump_core::config::StumpConfig;

use crate::{
config::jwt::CreatedToken,
filter::{
Expand All @@ -35,6 +37,7 @@
CreateBookClubScheduleBookOption, GetBookClubsParams, UpdateBookClub,
UpdateBookClubMember,
},
config::UploadConfig,
emailer::{
CreateOrUpdateEmailDevice, CreateOrUpdateEmailer, EmailerIncludeParams,
EmailerSendRecordIncludeParams, PatchEmailDevice,
Expand Down Expand Up @@ -240,6 +243,10 @@
format!("{}\n\n", ts_export::<CreateOrUpdateSmartListView>()?).as_bytes(),
)?;

file.write_all(format!("{}\n\n", ts_export::<UploadConfig>()?).as_bytes())?;

Check warning on line 246 in apps/server/src/routers/api/mod.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/mod.rs#L246

Added line #L246 was not covered by tests

file.write_all(format!("{}\n\n", ts_export::<StumpConfig>()?).as_bytes())?;

Check warning on line 248 in apps/server/src/routers/api/mod.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/mod.rs#L248

Added line #L248 was not covered by tests

Ok(())
}
}
64 changes: 64 additions & 0 deletions apps/server/src/routers/api/v1/config.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
use axum::{extract::State, middleware, routing::get, Extension, Json, Router};
use serde::{Deserialize, Serialize};
use specta::Type;
use stump_core::{config::StumpConfig, db::entity::UserPermission};

use crate::{
config::state::AppState,
errors::APIResult,
middleware::auth::{auth_middleware, RequestContext},
};

pub(crate) fn mount(app_state: AppState) -> Router<AppState> {
Router::new()
.route("/config", get(get_server_config))
.route("/config/upload", get(get_upload_config))
.layer(middleware::from_fn_with_state(app_state, auth_middleware))
}

Check warning on line 17 in apps/server/src/routers/api/v1/config.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/config.rs#L12-L17

Added lines #L12 - L17 were not covered by tests

#[utoipa::path(
get,
path = "/api/v1/config",
tag = "config",
responses(
(status = 200, description = "Successfully retrieved server config"),
(status = 401, description = "Unauthorized"),
(status = 500, description = "Internal server error")
)
)]
async fn get_server_config(
State(ctx): State<AppState>,
Extension(req): Extension<RequestContext>,
) -> APIResult<Json<StumpConfig>> {
req.enforce_permissions(&[UserPermission::ManageServer])?;

Check warning on line 33 in apps/server/src/routers/api/v1/config.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/config.rs#L19-L33

Added lines #L19 - L33 were not covered by tests

Ok(Json(StumpConfig::clone(&ctx.config)))
}

Check warning on line 36 in apps/server/src/routers/api/v1/config.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/config.rs#L35-L36

Added lines #L35 - L36 were not covered by tests

#[derive(Debug, Deserialize, Serialize, Type)]

Check warning on line 38 in apps/server/src/routers/api/v1/config.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/config.rs#L38

Added line #L38 was not covered by tests
pub struct UploadConfig {
enabled: bool,
max_file_upload_size: usize,
}

#[utoipa::path(
get,
path = "/api/v1/config/upload",
tag = "config",
responses(
(status = 200, description = "Successfully retrieved upload config"),
(status = 401, description = "Unauthorized"),
(status = 500, description = "Internal server error")
)
)]
async fn get_upload_config(
State(ctx): State<AppState>,
Extension(req): Extension<RequestContext>,
) -> APIResult<Json<UploadConfig>> {
req.enforce_permissions(&[UserPermission::UploadFile])?;

Check warning on line 58 in apps/server/src/routers/api/v1/config.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/config.rs#L44-L58

Added lines #L44 - L58 were not covered by tests

Ok(Json(UploadConfig {
enabled: ctx.config.enable_upload,
max_file_upload_size: ctx.config.max_file_upload_size,
}))
}

Check warning on line 64 in apps/server/src/routers/api/v1/config.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/config.rs#L60-L64

Added lines #L60 - L64 were not covered by tests
9 changes: 5 additions & 4 deletions apps/server/src/routers/api/v1/library.rs
Original file line number Diff line number Diff line change
Expand Up @@ -728,11 +728,11 @@
.await?
.ok_or(APIError::NotFound(String::from("Library not found")))?;

let (content_type, bytes) =
let upload_data =

Check warning on line 731 in apps/server/src/routers/api/v1/library.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/library.rs#L731

Added line #L731 was not covered by tests
validate_and_load_image(&mut upload, Some(ctx.config.max_image_upload_size))
.await?;

let ext = content_type.extension();
let ext = upload_data.content_type.extension();

Check warning on line 735 in apps/server/src/routers/api/v1/library.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/library.rs#L735

Added line #L735 was not covered by tests
let library_id = library.id;

// Note: I chose to *safely* attempt the removal as to not block the upload, however after some
Expand All @@ -745,10 +745,11 @@
),
}

let path_buf = place_thumbnail(&library_id, ext, &bytes, &ctx.config).await?;
let path_buf =
place_thumbnail(&library_id, ext, &upload_data.bytes, &ctx.config).await?;

Check warning on line 749 in apps/server/src/routers/api/v1/library.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/library.rs#L748-L749

Added lines #L748 - L749 were not covered by tests

Ok(ImageResponse::from((
content_type,
upload_data.content_type,

Check warning on line 752 in apps/server/src/routers/api/v1/library.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/library.rs#L752

Added line #L752 was not covered by tests
fs::read(path_buf).await?,
)))
}
Expand Down
9 changes: 5 additions & 4 deletions apps/server/src/routers/api/v1/media/thumbnails.rs
Original file line number Diff line number Diff line change
Expand Up @@ -266,10 +266,10 @@
.await?
.ok_or(APIError::NotFound(String::from("Media not found")))?;

let (content_type, bytes) =
let upload_data =

Check warning on line 269 in apps/server/src/routers/api/v1/media/thumbnails.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/media/thumbnails.rs#L269

Added line #L269 was not covered by tests
validate_and_load_image(&mut upload, Some(ctx.config.max_image_upload_size))
.await?;
let ext = content_type.extension();
let ext = upload_data.content_type.extension();

Check warning on line 272 in apps/server/src/routers/api/v1/media/thumbnails.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/media/thumbnails.rs#L272

Added line #L272 was not covered by tests
let book_id = media.id;

// Note: I chose to *safely* attempt the removal as to not block the upload, however after some
Expand All @@ -283,10 +283,11 @@
);
}

let path_buf = place_thumbnail(&book_id, ext, &bytes, &ctx.config).await?;
let path_buf =
place_thumbnail(&book_id, ext, &upload_data.bytes, &ctx.config).await?;

Check warning on line 287 in apps/server/src/routers/api/v1/media/thumbnails.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/media/thumbnails.rs#L286-L287

Added lines #L286 - L287 were not covered by tests

Ok(ImageResponse::from((
content_type,
upload_data.content_type,

Check warning on line 290 in apps/server/src/routers/api/v1/media/thumbnails.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/media/thumbnails.rs#L290

Added line #L290 was not covered by tests
fs::read(path_buf).await?,
)))
}
16 changes: 13 additions & 3 deletions apps/server/src/routers/api/v1/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

pub(crate) mod auth;
pub(crate) mod book_club;
pub(crate) mod config;
pub(crate) mod emailer;
pub(crate) mod epub;
pub(crate) mod filesystem;
Expand All @@ -33,10 +34,11 @@
pub(crate) mod series;
pub(crate) mod smart_list;
pub(crate) mod tag;
pub(crate) mod upload;
pub(crate) mod user;

pub(crate) fn mount(app_state: AppState) -> Router<AppState> {
Router::new()
let mut router = Router::new()

Check warning on line 41 in apps/server/src/routers/api/v1/mod.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/mod.rs#L41

Added line #L41 was not covered by tests
.merge(auth::mount(app_state.clone()))
.merge(epub::mount(app_state.clone()))
.merge(emailer::mount(app_state.clone()))
Expand All @@ -52,12 +54,20 @@
.merge(user::mount(app_state.clone()))
.merge(reading_list::mount(app_state.clone()))
.merge(smart_list::mount(app_state.clone()))
.merge(book_club::mount(app_state))
.merge(book_club::mount(app_state.clone()))
.merge(config::mount(app_state.clone()))

Check warning on line 58 in apps/server/src/routers/api/v1/mod.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/mod.rs#L57-L58

Added lines #L57 - L58 were not covered by tests
.route("/claim", get(claim))
.route("/ping", get(ping))
// TODO: should /version or /check-for-updates be behind any auth reqs?
.route("/version", post(version))
.route("/check-for-update", get(check_for_updates))
.route("/check-for-update", get(check_for_updates));

// Conditionally attach upload routes based on settings.
if app_state.config.enable_upload {
router = router.merge(upload::mount(app_state.clone()));
}

Check warning on line 68 in apps/server/src/routers/api/v1/mod.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/mod.rs#L63-L68

Added lines #L63 - L68 were not covered by tests

router

Check warning on line 70 in apps/server/src/routers/api/v1/mod.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/mod.rs#L70

Added line #L70 was not covered by tests
}

#[derive(Serialize, Type, ToSchema)]
Expand Down
9 changes: 5 additions & 4 deletions apps/server/src/routers/api/v1/series.rs
Original file line number Diff line number Diff line change
Expand Up @@ -612,10 +612,10 @@
.await?
.ok_or(APIError::NotFound(String::from("Series not found")))?;

let (content_type, bytes) =
let upload_data =

Check warning on line 615 in apps/server/src/routers/api/v1/series.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/series.rs#L615

Added line #L615 was not covered by tests
validate_and_load_image(&mut upload, Some(ctx.config.max_image_upload_size))
.await?;
let ext = content_type.extension();
let ext = upload_data.content_type.extension();

Check warning on line 618 in apps/server/src/routers/api/v1/series.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/series.rs#L618

Added line #L618 was not covered by tests
let series_id = series.id;

// Note: I chose to *safely* attempt the removal as to not block the upload, however after some
Expand All @@ -628,10 +628,11 @@
),
}

let path_buf = place_thumbnail(&series_id, ext, &bytes, &ctx.config).await?;
let path_buf =
place_thumbnail(&series_id, ext, &upload_data.bytes, &ctx.config).await?;

Check warning on line 632 in apps/server/src/routers/api/v1/series.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/series.rs#L631-L632

Added lines #L631 - L632 were not covered by tests

Ok(ImageResponse::from((
content_type,
upload_data.content_type,

Check warning on line 635 in apps/server/src/routers/api/v1/series.rs

View check run for this annotation

Codecov / codecov/patch

apps/server/src/routers/api/v1/series.rs#L635

Added line #L635 was not covered by tests
fs::read(path_buf).await?,
)))
}
Expand Down
Loading
Loading