Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn if parsed memory/scratchsize may be too small #510

Merged
merged 1 commit into from
Feb 18, 2025

Conversation

obbardc
Copy link
Member

@obbardc obbardc commented Jul 1, 2024

Since the --memory and --scratchsize arguments are parsed with human-readable suffixes into bytes (e.g 2048MB or 2GB), if a user does not put a suffix the arguments are parsed as bytes which can cause issues later when a user sets the memory to be 2048 assuming it will be parsed as MB.

Change the documentation to match reality and also add a warning if the user has set either parameter less than the recommended default.

Closes: #509

@obbardc obbardc force-pushed the wip/obbardc/debos-warn-low-size branch from 4ba7ea3 to 13f6822 Compare July 1, 2024 10:54
@obbardc obbardc requested a review from sjoerdsimons July 1, 2024 10:54
@daissi
Copy link
Contributor

daissi commented Nov 14, 2024

Something is wrong with --scratchsize:

./debos ../debos-recipes/rpi3/debimage-rpi3.yaml --scratchsize 100MB -m 100MB
2024/11/14 18:35:32 WARNING: Memory size of 100MB is less than recommended 2048MB
2024/11/14 18:35:32 WARNING: Scratch size of 95MB is less than recommended 2048MB

While is works for --memory, the conversion for scratchsize doesn't give us the same value 100MB vs 95MB

@obbardc
Copy link
Member Author

obbardc commented Nov 14, 2024

Something is wrong with --scratchsize:

./debos ../debos-recipes/rpi3/debimage-rpi3.yaml --scratchsize 100MB -m 100MB
2024/11/14 18:35:32 WARNING: Memory size of 100MB is less than recommended 2048MB
2024/11/14 18:35:32 WARNING: Scratch size of 95MB is less than recommended 2048MB

While is works for --memory, the conversion for scratchsize doesn't give us the same value 100MB vs 95MB

I guess that is because for scratchsize we convert to bytes with units.FromHumanSize but for memory we do not :-).

Perhaps we should unify this by:

  1. using units.FromHumanSize to convert for both scratchsize & memory.
  2. convert to MB using forumula / 1000 / 1000 for both variables instead of / 1024 / 1024. That should make the printed number the same.

To be honest, I threw this together quickly just for some feedback. Happy to rework it.

@daissi
Copy link
Contributor

daissi commented Nov 15, 2024

I would say let's use for both units.FromHumanSize for consistency :) This should avoid confusion.

@obbardc obbardc changed the title debos: Warn if parsed memory/scratchsize may be too small Warn if parsed memory/scratchsize may be too small Nov 26, 2024
@sjoerdsimons
Copy link
Member

I would say let's use for both units.FromHumanSize for consistency :) This should avoid confusion.

go-units isn't great FromHumanSize is really more for disk sizes (decimal units) while RamInBytes follows binary units which is more suitable for memory (see https://en.wikipedia.org/wiki/Byte#Multiple-byte_units). However annoyingly it will accept both MiB and MB for either, but will parse them according to the predetermined table.... So if you do FromHumanSize("100Mib") you'll get 100 * 1000**2

For this PR assuming a MB for disk is 1000**2 while a MB for ram is 1024**2 would be the correct solution. Ideally FromHumanSize to recognize MiB as binary, but doesn't look like that will happen (go-units issue). However the expectation of what an MB is sadly just is different depending on the context 🤷‍♂️

Copy link
Member

@sjoerdsimons sjoerdsimons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comment :)

@obbardc obbardc force-pushed the wip/obbardc/debos-warn-low-size branch from 13f6822 to 88132f8 Compare February 16, 2025 16:14
@obbardc obbardc requested a review from sjoerdsimons February 16, 2025 16:14
@obbardc
Copy link
Member Author

obbardc commented Feb 16, 2025

@sjoerdsimons made teh changes according to your suggestions :-)

Copy link
Member

@sjoerdsimons sjoerdsimons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry i should have looked at the sizes more closely yesterday, i just ended up being in the 1024 vs 1000 rathole too much ;) TL;DR I think giving warnings is great, but the current threshold seems a bit low, if you run on a small VM (e.g. 1 or 2G memory instance) then having a smaller then 2G memory size is fine. Especially if you use a scratchdisk, where the VMs memory mostly just needs to be enough to run apt.

So lets be a bit less trigger-happy on these warnings, as you said it's mostly for people forgetting to add M. Which we'll still easily filter out even with a very conservative "accepted" minimum

Since the --memory and --scratchsize arguments are parsed with human-readable
suffixes into bytes (e.g 2048MB or 2GB), if a user does not put a suffix
the arguments are parsed as bytes which can cause issues later when a user
sets the memory to be 2048 assuming it will be parsed as MB.

Change the documentation to match reality and also add a warning if the user
has set either parameter less than a recommended minimum.

Closes: #509
Signed-off-by: Christopher Obbard <[email protected]>
@obbardc obbardc force-pushed the wip/obbardc/debos-warn-low-size branch from 88132f8 to fe5f7e9 Compare February 17, 2025 14:30
@obbardc obbardc requested a review from sjoerdsimons February 17, 2025 14:31
@sjoerdsimons sjoerdsimons added this pull request to the merge queue Feb 18, 2025
Merged via the queue into main with commit e510081 Feb 18, 2025
62 checks passed
@sjoerdsimons sjoerdsimons deleted the wip/obbardc/debos-warn-low-size branch February 18, 2025 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using "--memory" breaks fakemachine backend
3 participants