From 573c255d15d69cac2436017cd8d7a26867951fe9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bart=C5=82omiej=20Dach?= Date: Wed, 14 Dec 2022 22:52:55 +0100 Subject: [PATCH 1/6] Add description of osu!(lazer)'s file storage system --- .../Release_stream/Lazer/File_storage/en.md | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 wiki/Client/Release_stream/Lazer/File_storage/en.md diff --git a/wiki/Client/Release_stream/Lazer/File_storage/en.md b/wiki/Client/Release_stream/Lazer/File_storage/en.md new file mode 100644 index 000000000000..25d9a1461c6b --- /dev/null +++ b/wiki/Client/Release_stream/Lazer/File_storage/en.md @@ -0,0 +1,52 @@ +# File storage in osu!(lazer) + +By default, osu!(lazer) stores beatmaps, skins, and score replay files in the following directories: + +- `%appdata%/osu/files` (Windows), +- `~/.local/share/osu/files` (Linux & macOS), +- `Android/data/sh.ppy.osulazer/files/files` (Android). + +On desktop platforms, the file store can also be moved in its entirety to another location, using the `Change folder location...` button in client settings. + +## Storage structure + +osu!(stable)'s storage structure, where data was directly accessible to users as standard files, resulted in a lot of teething issues due to having to explicitly handle unwanted modifications of key files. To avoid this, osu!(lazer) employs a more stringent method of file storage. As such, there is no equivalent of the `Songs` and `Skins` folders. All files that are imported to lazer are stored under filenames that reflect their [SHA-256 hashes](https://en.wikipedia.org/wiki/SHA-2). Mappings to these files are held inside a client database. + +For example, a file with the SHA-256 hash of + +``` +1a47929b6056d34d25a95eeb2012395ceed66af6f40cc37c898a08482d6325d2 +``` + +shall be stored under the path of + +``` +files/1/1a/1a47929b6056d34d25a95eeb2012395ceed66af6f40cc37c898a08482d6325d2 +``` + +This allows for saving storage space by preventing duplicates of the same file from being persisted to disk, and prevents users (or other applications) from easily tampering with files that should not be tampered with. All inconveniences compared to stable that arise from this choice have been addressed by either allowing exports or adding new in-game management features, or will be addressed in due time by new features. + +At the time of writing, the correct procedure to manually modify skins or beatmaps outside of the facilities available in-game is to export the item, update it, and then reimport it again with the desired changes. + +## Migration from osu!(stable) + +### Via hard links + +On most Windows systems, osu!(lazer) will be able to import data from the stable version of the game without having to create a second copy of the data on disk. This is possible thanks to an operating system feature called *hard links*. + +A hard link is conceptually similar to a *shortcut* (also known as a *symbolic link*), in that it is a method that allows a user to have the same file available from multiple different places on their filesystem. However, while shortcuts are just plain files that point to a different file (and therefore require additional space), hard links work one level deeper, at the level of the filesystem itself. + +In simpler terms, two hard-linked files are just two different names for the same piece of physical space on the hard drive. The same data is available through either of those names, which has several other implications: + +- If a file is edited using one of those names, then the other name will immediately see the same changes applied to it. +- If a file is moved or deleted through one of those names, the other name is not affected at all. Compare this to shortcuts, where moving or deleting the original file breaks the shortcut, because there is now nothing to point to. + +After a migration has been completed using hard links, both the `Songs` folder in stable, and a `files` folder in lazer will contain the same beatmaps, but the files inside both will point to the same data on the disk. Deleting songs in one installation will not affect the other. Updating or editing beatmaps in one installation will also not affect the other, as long as osu! is used to make changes, rather than external applications. + +This method is the most efficient way of sharing the same files between lazer and stable installs, because it incurs no overhead on either end. However, because it is a filesystem-level feature, it requires that the hard-linked files reside on a common drive, and that the drive is formatted using NTFS. + +Note that if disk usage is checked using the `Properties` window in Explorer or any other similar method, it will appear that both the `Songs` directory and the `files` directory are consuming disk space, potentially misleading users to think that the data is stored twice. This is an artifact of file size accounting, and the actual underlying data is **not** duplicated. This can be verified by comparing the total amount of free space on the entire disk before and after the migration. + +### Via file copy + +On other operating systems and filesystems where hard links are not available (or the functionality to create them has not been implemented yet), the migration from stable to lazer works by copying all files across from the old installation to the new. Both installs are therefore fully separated, but also consume twice the disk space. From fe437f5843f58c2aa1da4e19759fb08da7214764 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bart=C5=82omiej=20Dach?= Date: Thu, 15 Dec 2022 19:43:07 +0100 Subject: [PATCH 2/6] Thin down description of hard linkage --- wiki/Client/Release_stream/Lazer/File_storage/en.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/wiki/Client/Release_stream/Lazer/File_storage/en.md b/wiki/Client/Release_stream/Lazer/File_storage/en.md index 25d9a1461c6b..8b405ad832dd 100644 --- a/wiki/Client/Release_stream/Lazer/File_storage/en.md +++ b/wiki/Client/Release_stream/Lazer/File_storage/en.md @@ -34,19 +34,14 @@ At the time of writing, the correct procedure to manually modify skins or beatma On most Windows systems, osu!(lazer) will be able to import data from the stable version of the game without having to create a second copy of the data on disk. This is possible thanks to an operating system feature called *hard links*. -A hard link is conceptually similar to a *shortcut* (also known as a *symbolic link*), in that it is a method that allows a user to have the same file available from multiple different places on their filesystem. However, while shortcuts are just plain files that point to a different file (and therefore require additional space), hard links work one level deeper, at the level of the filesystem itself. +A hard link is conceptually similar to a *shortcut* in that it is a method that allows a user to have the same file available from multiple different places on their filesystem. However, while shortcuts are just plain files that point to a different file (and therefore require additional space), hard links work one level deeper, at the level of the filesystem itself. -In simpler terms, two hard-linked files are just two different names for the same piece of physical space on the hard drive. The same data is available through either of those names, which has several other implications: - -- If a file is edited using one of those names, then the other name will immediately see the same changes applied to it. -- If a file is moved or deleted through one of those names, the other name is not affected at all. Compare this to shortcuts, where moving or deleting the original file breaks the shortcut, because there is now nothing to point to. - -After a migration has been completed using hard links, both the `Songs` folder in stable, and a `files` folder in lazer will contain the same beatmaps, but the files inside both will point to the same data on the disk. Deleting songs in one installation will not affect the other. Updating or editing beatmaps in one installation will also not affect the other, as long as osu! is used to make changes, rather than external applications. - -This method is the most efficient way of sharing the same files between lazer and stable installs, because it incurs no overhead on either end. However, because it is a filesystem-level feature, it requires that the hard-linked files reside on a common drive, and that the drive is formatted using NTFS. +In simple terms, two hard-linked files are just two different names for the same piece of physical space on the hard drive. The same data is available through either of those names. This means that once the migration completes using hard links, both the `Songs` folder in stable, and the `files` folder in lazer will contain files which will point to shared underlying data on the disk. Deleting files from one installation will not affect the other. Updating or editing files in one installation will also not affect the other if osu! is used to make changes, rather than external applications. Note that if disk usage is checked using the `Properties` window in Explorer or any other similar method, it will appear that both the `Songs` directory and the `files` directory are consuming disk space, potentially misleading users to think that the data is stored twice. This is an artifact of file size accounting, and the actual underlying data is **not** duplicated. This can be verified by comparing the total amount of free space on the entire disk before and after the migration. +As hard links are a filesystem-level feature, it is required that the hard-linked files reside on a common drive, and that the drive is formatted using NTFS. + ### Via file copy On other operating systems and filesystems where hard links are not available (or the functionality to create them has not been implemented yet), the migration from stable to lazer works by copying all files across from the old installation to the new. Both installs are therefore fully separated, but also consume twice the disk space. From 702bc83670a8793c557213e25ad800e1604548c2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bart=C5=82omiej=20Dach?= Date: Thu, 15 Dec 2022 21:07:11 +0100 Subject: [PATCH 3/6] Apply suggestions from code review Co-authored-by: Walavouchey <36758269+Walavouchey@users.noreply.github.com> --- wiki/Client/Release_stream/Lazer/File_storage/en.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/wiki/Client/Release_stream/Lazer/File_storage/en.md b/wiki/Client/Release_stream/Lazer/File_storage/en.md index 8b405ad832dd..fef7428fa83a 100644 --- a/wiki/Client/Release_stream/Lazer/File_storage/en.md +++ b/wiki/Client/Release_stream/Lazer/File_storage/en.md @@ -6,7 +6,7 @@ By default, osu!(lazer) stores beatmaps, skins, and score replay files in the fo - `~/.local/share/osu/files` (Linux & macOS), - `Android/data/sh.ppy.osulazer/files/files` (Android). -On desktop platforms, the file store can also be moved in its entirety to another location, using the `Change folder location...` button in client settings. +On desktop platforms, the file store can also be moved in its entirety to another location, using the `Change folder location...` button in the client settings. ## Storage structure @@ -18,7 +18,7 @@ For example, a file with the SHA-256 hash of 1a47929b6056d34d25a95eeb2012395ceed66af6f40cc37c898a08482d6325d2 ``` -shall be stored under the path of +would be stored under the path of ``` files/1/1a/1a47929b6056d34d25a95eeb2012395ceed66af6f40cc37c898a08482d6325d2 @@ -26,7 +26,7 @@ files/1/1a/1a47929b6056d34d25a95eeb2012395ceed66af6f40cc37c898a08482d6325d2 This allows for saving storage space by preventing duplicates of the same file from being persisted to disk, and prevents users (or other applications) from easily tampering with files that should not be tampered with. All inconveniences compared to stable that arise from this choice have been addressed by either allowing exports or adding new in-game management features, or will be addressed in due time by new features. -At the time of writing, the correct procedure to manually modify skins or beatmaps outside of the facilities available in-game is to export the item, update it, and then reimport it again with the desired changes. +For now, the correct procedure to manually modify skins or beatmaps outside of the facilities available in-game is to export the item, update it, and then reimport it again with the desired changes. ## Migration from osu!(stable) From dec9f47478207ea81bd10f7d26323ae6ae13d6fc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bart=C5=82omiej=20Dach?= Date: Thu, 15 Dec 2022 22:41:56 +0100 Subject: [PATCH 4/6] Apply more suggestions from code review Co-authored-by: Walavouchey <36758269+Walavouchey@users.noreply.github.com> --- wiki/Client/Release_stream/Lazer/File_storage/en.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/wiki/Client/Release_stream/Lazer/File_storage/en.md b/wiki/Client/Release_stream/Lazer/File_storage/en.md index fef7428fa83a..0096d8581c0b 100644 --- a/wiki/Client/Release_stream/Lazer/File_storage/en.md +++ b/wiki/Client/Release_stream/Lazer/File_storage/en.md @@ -36,7 +36,7 @@ On most Windows systems, osu!(lazer) will be able to import data from the stable A hard link is conceptually similar to a *shortcut* in that it is a method that allows a user to have the same file available from multiple different places on their filesystem. However, while shortcuts are just plain files that point to a different file (and therefore require additional space), hard links work one level deeper, at the level of the filesystem itself. -In simple terms, two hard-linked files are just two different names for the same piece of physical space on the hard drive. The same data is available through either of those names. This means that once the migration completes using hard links, both the `Songs` folder in stable, and the `files` folder in lazer will contain files which will point to shared underlying data on the disk. Deleting files from one installation will not affect the other. Updating or editing files in one installation will also not affect the other if osu! is used to make changes, rather than external applications. +In simple terms, two hard-linked files are just two different names for the same piece of physical space on the hard drive. This means that once the migration completes using hard links, both the `Songs` folder in stable, and the `files` folder in lazer will contain files which will point to shared underlying data on the disk. Deleting files from one installation will not affect the other. Updating or editing files causes the files to become separate, and will therefore also not affect the other installation, as long as osu! is used to make the changes. Note that if disk usage is checked using the `Properties` window in Explorer or any other similar method, it will appear that both the `Songs` directory and the `files` directory are consuming disk space, potentially misleading users to think that the data is stored twice. This is an artifact of file size accounting, and the actual underlying data is **not** duplicated. This can be verified by comparing the total amount of free space on the entire disk before and after the migration. @@ -44,4 +44,4 @@ As hard links are a filesystem-level feature, it is required that the hard-linke ### Via file copy -On other operating systems and filesystems where hard links are not available (or the functionality to create them has not been implemented yet), the migration from stable to lazer works by copying all files across from the old installation to the new. Both installs are therefore fully separated, but also consume twice the disk space. +On other operating systems and filesystems where hard links are not available (or the functionality to create them has not been implemented yet), the migration from stable to lazer works by copying all files across from the old installation to the new. Both installs are therefore fully separated, but also consume up to twice the disk space. From 98efd3854909153b76d201dd946a8069c0ce8127 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bart=C5=82omiej=20Dach?= Date: Fri, 16 Dec 2022 15:54:36 +0100 Subject: [PATCH 5/6] Fix bad path to file store on macOS --- wiki/Client/Release_stream/Lazer/File_storage/en.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/wiki/Client/Release_stream/Lazer/File_storage/en.md b/wiki/Client/Release_stream/Lazer/File_storage/en.md index 0096d8581c0b..560c718d1111 100644 --- a/wiki/Client/Release_stream/Lazer/File_storage/en.md +++ b/wiki/Client/Release_stream/Lazer/File_storage/en.md @@ -3,7 +3,8 @@ By default, osu!(lazer) stores beatmaps, skins, and score replay files in the following directories: - `%appdata%/osu/files` (Windows), -- `~/.local/share/osu/files` (Linux & macOS), +- `~/.local/share/osu/files` (Linux), +- `~/Library/Application Support/osu/files` (macOS), - `Android/data/sh.ppy.osulazer/files/files` (Android). On desktop platforms, the file store can also be moved in its entirety to another location, using the `Change folder location...` button in the client settings. From 7bd3de4fcd071596869de8f612d9e3f35632be92 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bart=C5=82omiej=20Dach?= Date: Fri, 16 Dec 2022 15:59:06 +0100 Subject: [PATCH 6/6] Link lazer upgrade guide with file storage description --- wiki/Help_centre/Upgrading_to_lazer/en.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/wiki/Help_centre/Upgrading_to_lazer/en.md b/wiki/Help_centre/Upgrading_to_lazer/en.md index 1f98194087b1..5916be907be7 100644 --- a/wiki/Help_centre/Upgrading_to_lazer/en.md +++ b/wiki/Help_centre/Upgrading_to_lazer/en.md @@ -110,7 +110,7 @@ Currently, beatmaps, skins, scores, replays and collections can be imported into #### If I import my beatmaps to lazer, will it use double the disk space? -If you have both lazer and stable on the same drive, [hard links](https://en.wikipedia.org/wiki/Hard_link) are used to avoid using extra disk space. +If you have both lazer and stable on the same drive, [hard links](/wiki/Client/Release_stream/Lazer/File_storage#via-hard-links) are used to avoid using extra disk space. In all other cases, importing beatmaps will use double the disk space. @@ -228,7 +228,7 @@ We have a huge backlog of user-requested features and improvements that we will #### How do I access my songs folder? -There is no songs folder in lazer! This allows us to do cool things like not require pressing `F5` at song select to refresh beatmaps (because beatmaps are always in a good state) and reduce the disk space used by beatmaps by 20–40%. You can read more about [the way lazer stores files](https://github.com/ppy/osu/wiki/User-file-storage). +There is no songs folder in lazer! This allows us to do cool things like not require pressing `F5` at song select to refresh beatmaps (because beatmaps are always in a good state) and reduce the disk space used by beatmaps by 20–40%. You can read more about [the way lazer stores files](/wiki/Client/Release_stream/Lazer/File_storage). If you need to make changes to a beatmap, please use the editor. Going forward we will introduce a mode in the editor which makes a beatmap's folder temporarily accessible for external editing. This will allow you to use external tools on a beatmap during the creation process.