Skip to content

Commit

Permalink
Update to latest CoW library and update READMEs with latest info (#589)
Browse files Browse the repository at this point in the history
  • Loading branch information
erikmav authored Nov 5, 2024
1 parent 1496366 commit 7b94813
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 19 deletions.
2 changes: 1 addition & 1 deletion Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
<MicrosoftBuildMinimumPackageVersion>16.11.0</MicrosoftBuildMinimumPackageVersion>
</PropertyGroup>
<ItemGroup>
<PackageVersion Include="CopyOnWrite" Version="0.4.1" Condition=" '$(TargetFramework)' != 'net46' " />
<PackageVersion Include="CopyOnWrite" Version="0.4.2" Condition=" '$(TargetFramework)' != 'net46' " />
<PackageVersion Include="Microsoft.Build" Version="$(MicrosoftBuildPackageVersion)" />
<PackageVersion Include="Microsoft.Build.Tasks.Core" Version="$(MicrosoftBuildPackageVersion)" />
<PackageVersion Include="Microsoft.Win32.Registry" Version="5.0.0" />
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Supports staging artifacts from build outputs.
[![NuGet](https://img.shields.io/nuget/v/Microsoft.Build.CopyOnWrite.svg)](https://www.nuget.org/packages/Microsoft.Build.CopyOnWrite)
[![NuGet](https://img.shields.io/nuget/dt/Microsoft.Build.CopyOnWrite.svg)](https://www.nuget.org/packages/Microsoft.Build.CopyOnWrite)

Enables Copy on Write for faster file copies.
Enables Copy on Write on Windows Dev Drive and ReFS for faster file copies.

### [Microsoft.Build.RunVSTest](src/RunTests)
[![NuGet](https://img.shields.io/nuget/v/Microsoft.Build.RunVSTest.svg)](https://www.nuget.org/packages/Microsoft.Build.RunVSTest)
Expand Down
24 changes: 7 additions & 17 deletions src/CopyOnWrite/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@

The `Microsoft.Build.CopyOnWrite` MSBuild SDK overrides the native MSBuild Copy task to add support for ReFS and Dev Drive CloneFile (Copy on Write or CoW) on Windows. It is designed to be as backwards compatible as possible and should directly replace all usages of Copy in MSBuild.

On Linux and Mac the current behavior is to always fall back to regular file copies (`File.Copy`), however `File.Copy` automatically uses CoW for [Linux](https://github.com/dotnet/runtime/pull/64264) (starting in .NET 7) and [Mac](https://github.com/dotnet/runtime/pull/79243) (.NET 8). A [similar PR](https://github.com/dotnet/runtime/pull/88695) for Windows did not make it into .NET, however there is [work underway](https://devblogs.microsoft.com/engineering-at-microsoft/copy-on-write-in-win32-api-early-access/) to integrate CoW into the Windows API in a possible future release.
*Note*: On Windows this library is being superseded by CoW support now built into the Windows 11 24H2 release, as well as [Windows Server 2025](https://learn.microsoft.com/en-us/windows-server/get-started/whats-new-windows-server-2025#block-cloning-support). Use of CoW is automatic for Dev Drive and ReFS volumes starting in these OS versions. See related notes in our [blog entry](https://devblogs.microsoft.com/engineering-at-microsoft/copy-on-write-performance-and-debugging/) and linked earlier articles. We will continue to accept bug fixes for this library, and updates for the related [`CopyOnWrite`](https://github.com/microsoft/CopyOnWrite) base package.

Linux and Mac behavior is to automatically use CoW for [Linux](https://github.com/dotnet/runtime/pull/64264) (starting in .NET 7) and [Mac](https://github.com/dotnet/runtime/pull/79243) (.NET 8). This library is not needed if you are not building on Windows.

## Usage in `Directory.Packages.Props`
This is intended to be used in a large repo already onboarded to Central Package Management. In your `Directory.Packages.props`:
Expand All @@ -30,23 +32,11 @@ If your project types don't support NuGet (e.g. `.vcxproj`), you can alternative
```

## Background
This SDK aims to improve the performance of large repositories by minimizing file copies during the build process. Many large repos have the snowball effect where Resolve Assembly Reference will add more and more dependencies that get copied along in different layers of the build graph. This is an attempt to accelerate those by taking advantage of the Copy on Write feature of the OS using this library: [https://github.com/microsoft/CopyOnWrite](https://github.com/microsoft/CopyOnWrite).

Example build, internal Microsoft repo with 758 nodes, 3989 edges, on Win11 22H2 ReFS, with the NuGet cache moved onto the same disk volume:
This SDK aims to improve the performance of large repositories by minimizing file copies during the build process. Many large repos have the snowball effect where Resolve Assembly Reference will add more and more dependencies that get copied along in different layers of the build graph. This is an attempt to accelerate those by taking advantage of the Copy on Write feature of in Windows using this library: [https://github.com/microsoft/CopyOnWrite](https://github.com/microsoft/CopyOnWrite).

Command | Value
---|---
`msbuild /p:DisableCopyOnWrite=true`| Time Elapsed 00:06:02.03
Enlistment size| 66.4 GB (71,305,402,686 bytes)
Size on disk| 66.9 GB (71,848,480,768 bytes)
Perf test results on large-sized repos can be found in this [blog post](https://devblogs.microsoft.com/engineering-at-microsoft/copy-on-write-performance-and-debugging/). That post also includes debugging information such as how to determine if a filesystem entry is a CoW link (block clone).

Command | Value
---|---
`msbuild /p:DisableCopyOnWrite=false` | **Time Elapsed 00:04:09.32**
Enlistment size| 66.4 GB (71,305,402,686 bytes)
Size on disk| **3.83 GB (3,055,628,028 bytes)**
## Using on Windows
To use this feature on Windows, you need run your build on a drive formatted with [Dev Drive](https://aka.ms/devdrive) or [ReFS](https://learn.microsoft.com/en-us/windows-server/storage/refs/refs-overview) on Windows. You should also move your package cahce to the same volume. ReFS is available on Windows Server, or on Windows 11 22H2 Enterprise and Pro SKUs. Dev Drive is available on all Windows 11 SKUs starting in 22H2 and in [Windows Server 2025](https://learn.microsoft.com/en-us/windows-server/get-started/whats-new-windows-server-2025#block-cloning-support).

See [blog post 1](https://aka.ms/EngMSDevDrive) and [blog post 2](https://aka.ms/VSDevDrive) for more information on Dev Drive, copy-on-write, and moving your package caches.

## Caveats
To use this feature, you need run on a drive formatted with [Dev Drive](https://aka.ms/devdrive) or [ReFS](https://learn.microsoft.com/en-us/windows-server/storage/refs/refs-overview) on Windows. ReFS is available on Windows Server, or on Windows 11 22H2 Enterprise and Pro SKUs. Dev Drive is available on all Windows 11 SKUs and is slated for a future Windows Server release.

0 comments on commit 7b94813

Please sign in to comment.