Add functionality for serializing depot manifests

[NicknineTheEagle]

This adds functionality for serializing depot manifests into Steam-compatible protobuf files. The output matches .manifest files found in Steam's depotcache folder unless there are non-ASCII filenames present. It seems that Steam handles filename sorting slightly differently, not quite sure what's going on there.

I'm intending to use this as a stepping stone towards migrating DepotDownloader to using Steam protobuf manifests.

[NicknineTheEagle]

@psychonic How soon is the next SteamKit release? Want to know when I can integrate these changes into DepotDownloader.

[psychonic]

I imagine not until the new Steam authentication is worked out and added. (#1125 / #1129), but I don't really manage the releases so can't say for sure.

[yaakov-h]

Usually its whenever I decide to publish one 😄

[skylayer]

Thank you for your contribution to the serialization logic of DepotManifest. I appreciate the effort you've put into implementing this feature. However, I've identified a critical performance issue within your implementation, specifically related to how unique chunks are determined:

if ( !uniqueChunks.Exists( x => x.SequenceEqual( chunk.ChunkID! ) ) )
{
    uniqueChunks.Add( chunk.ChunkID! );
}

This approach, unfortunately, leads to each existence check having a time complexity of O(N), resulting in an overall complexity of O(N^2) for this section of the code. This becomes significantly problematic for large manifests, such as those exceeding 100GB, where the serialization process could extend up to 30 minutes or more.

To address this issue and enhance the performance, I suggest utilizing a HashSet alongside a custom IEqualityComparer<byte[]>. This modification ensures that each check operates in near O(1) time, substantially reducing the total serialization time. Below is a proposed implementation:

class ByteArrayComparer : IEqualityComparer<byte[]>
{
    public bool Equals(byte[] x, byte[] y)
    {
        if (ReferenceEquals(x, y)) return true;
        if (x == null || y == null) return false;
        return x.SequenceEqual(y);
    }

    public int GetHashCode(byte[] obj)
    {
        if (obj == null) throw new ArgumentNullException(nameof(obj));
        // Adjust the hash code calculation as needed to ensure uniform distribution
        return BitConverter.ToInt32(obj, 0);
    }
}

// Usage...

var uniqueChunks = new HashSet<byte[]>(new ByteArrayComparer());

foreach (var chunk in file.Chunks)
{
    var protochunk = new ContentManifestPayload.FileMapping.ChunkData()
    {
        sha = chunk.ChunkID,
        crc = BitConverter.ToUInt32(chunk.Checksum!, 0),
        offset = chunk.Offset,
        cb_original = chunk.UncompressedLength,
        cb_compressed = chunk.CompressedLength
    };

    protofile.chunks.Add(protochunk);
    // Directly use HashSet.Add for O(1) complexity
    uniqueChunks.Add(chunk.ChunkID!);
}

By implementing this solution, we can significantly improve the performance of the serialization process for large depot manifests. I hope you find this suggestion helpful and consider integrating it into your pull request. Thank you again for your valuable contribution.

[xPaw]

This was incorrectly replacing slashes if the filename was not decrypted.

Discovered while adding a roundtrip test: 7559904