Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting QSV_FREEMEMORY_HEADROOM_PCT to 0 disables memory availability check #2353

Merged
merged 2 commits into from
Dec 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/ENVIRONMENT_VARIABLES.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
| `QSV_RDR_BUFFER_CAPACITY` | reader buffer size (default - 128k (bytes): 131072) |
| `QSV_SKIP_FORMAT_CHECK` | if set, skips mime-type checking of input files. Set this when optimizing for performance and when encountering false positives as a format check involves scanning the input file to infer the mime-type/format. |
| `QSV_WTR_BUFFER_CAPACITY` | writer buffer size (default - 512k (bytes): 524288) |
| `QSV_FREEMEMORY_HEADROOM_PCT` | the percentage of free available memory required when running qsv in "non-streaming" mode (i.e. the entire file needs to be loaded into memory). If the incoming file is greater than the available memory after the headroom is subtracted, qsv will not proceed. See [Memory Management](#memory-management) for more info. (default: (percent) 20 ) |
| `QSV_FREEMEMORY_HEADROOM_PCT` | the percentage of free available memory required when running qsv in "non-streaming" mode (i.e. the entire file needs to be loaded into memory). If the incoming file is greater than the available memory after the headroom is subtracted, qsv will not proceed. Set to 0 to skip memory check. See [Memory Management](#memory-management) for more info. (default: (percent) 20 ) |
| `QSV_MEMORY_CHECK` | if set, check if input file size < AVAILABLE memory - HEADROOM (CONSERVATIVE mode) when running in "non-streaming" mode. Otherwise, qsv will only check if the input file size < TOTAL memory - HEADROOM (NORMAL mode). This is done to prevent Out-of-Memory errors. See [Memory Management](#memory-management) for more info. |
| `QSV_LOG_LEVEL` | desired level (default - off; `error`, `warn`, `info`, `trace`, `debug`). |
| `QSV_LOG_DIR` | when logging is enabled, the directory where the log files will be stored. If the specified directory does not exist, qsv will attempt to create it. If not set, the log files are created in the directory where qsv was started. See [Logging](docs/Logging.md#logging) for more info. |
Expand Down
3 changes: 2 additions & 1 deletion dotenv.template
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,8 @@ QSV_WTR_BUFFER_CAPACITY = 524288
# the percentage of free available memory required when running qsv in
# "non-streaming" mode (i.e. the entire file needs to be loaded into memory).
# If the incoming file is greater than the available memory after the headroom
# is subtracted, qsv will not proceed. See Memory Management for more info.
# is subtracted, qsv will not proceed. Set to 0 to skip memory check.
# See Memory Management for more info.
# (default: (percent) 20 )
QSV_FREEMEMORY_HEADROOM_PCT = 20

Expand Down
14 changes: 10 additions & 4 deletions src/util.rs
Original file line number Diff line number Diff line change
Expand Up @@ -623,15 +623,21 @@ pub fn mem_file_check(

let conservative_memcheck_work = get_envvar_flag("QSV_MEMORY_CHECK") || conservative_memcheck;

let mut mem_pct = env::var("QSV_FREEMEMORY_HEADROOM_PCT")
.unwrap_or_else(|_| DEFAULT_FREEMEMORY_HEADROOM_PCT.to_string())
.parse::<u8>()
.unwrap_or(DEFAULT_FREEMEMORY_HEADROOM_PCT);

// if QSV_FREEMEMORY_HEADROOM_PCT is 0, we skip the memory check
if mem_pct == 0 {
return Ok(i64::MAX);
}

let mut sys = sysinfo::System::new();
sys.refresh_memory();
let avail_mem = sys.available_memory();
let free_swap = sys.free_swap();
let total_mem = sys.total_memory();
let mut mem_pct = env::var("QSV_FREEMEMORY_HEADROOM_PCT")
.unwrap_or_else(|_| DEFAULT_FREEMEMORY_HEADROOM_PCT.to_string())
.parse::<u8>()
.unwrap_or(DEFAULT_FREEMEMORY_HEADROOM_PCT);

// for safety, we don't want to go below 10% memory headroom
// nor above 90% memory headroom as its too memory-restrictive
Expand Down
Loading