Investigate support for Unity #644
Comments
|
+1 |
|
Three +1s in the space of a few minutes suggests this is being linked to from somewhere - it would be nice if someone could provide the context. It would be even nicer if someone with Unity support could try this and inform us where there are problems :) |
|
I can only speak for myself, but in my case we are developing a mobile game with Unity and a Java backend. In order to keep our client compatible with iOS, we are using protobuf-net. Unfortunately, protobuf-net has some known bugs and does not support the latest Protobuf versions, so an official solution from Google would be very interesting for us. |
|
@jayallanjethwa: But how did you happen to reach this issue? It seems unlikely that three people would find an issue raised nearly two months ago entirely independently but within the same 5 minute window. I'm only asking so that I can try to get more context that might help resolve the issue. (I'd love to be able to support Unity - but I foresee some tricky issues, based on previous experience of bug reports.) |
|
@jskeet I started to use protobuf v3 in Unity, and I found two problems. |
|
@asuuma: Right. Supporting both .NET 3.5 and modern PCLs is painful. I'm hoping that it won't be too long before Unity moves on in terms of runtime - after all, .NET 4.0 was released a long time ago. Could you give the details of the stack trace around (I'm going to need to balance the importance of supporting a wide variety of platforms with the additional code maintenance headaches and possible performance degredations of doing so. The more I know before making this judgement call, the better...) |
|
@jskeet Yeah, everyone are hoping to move on newer runtime in near future... They opened roadmap of Unity and it says they are researching about upgrading .NET profile. |
|
Right, thanks - so that is expression tree compilation. It's definitely feasible to write a workaround for that particular one. |
|
@asuuma Could you elaborate on what you did to workaround the .NET version mismatch? |
|
@dreis2211 Oh very sorry, I missed your message. |
|
I've applied this patch (without the commented out code) and can confirm this work in unity3d 5.2.2f |
|
Note that with this patch, it won't work with CoreCLR, which is a more important initial target for us. I don't think it's feasible to support both platforms with a single binary. |
|
Agreed, but it would be nice if unity could be supported somehow untill they finally decide to move away from their ancient mono version. |
|
I wouldn't feel comfortable claiming something was "supported" via just a patch with consumers building the code themselves afterwards. This patch is a good starting point for fuller support, but I'd rather get the first version publicly released with just CoreCLR (and therefore normal desktop .NET) support to start with, then revisit. |
|
+1 |
|
+1 I've also applied the patch, and have protobuf v3.0.0-beta-2 working in my mod for Kerbal Space Program (running in Unity). I also ran the nunit tests which all passed (although I removed Google.Protobuf.Test.Compatibility) |
|
@jskeet Thanks for your statement. Do you have a rough idea of how the timeline for this could look like? |
|
I don't have any plans at the moment... I'd be very surprised to see it before protobuf 3.0 goes GA. We can look at it again after that. It's not just a matter of fixing the code itself - it's making sure we can test and maintain it appropriately. (Any sign of Unity moving off the ancient CLR they use yet? I'd hope that they're looking into CoreCLR now it's nearing release...) |
|
I think it is unlikely that Unity moves off their CLR. They have their own Mono fork and it contains a bug which is already fixed in original Mono repo 5 years ago. I guess merging branches forked at least 5 years ago would be a heavy toll. Plus they have to make their il2cpp work after the update. But still, I wish that would happen. I'd love to use protobuf and google.apis.v3 in Unity. |
|
You are able to use Protobuf just fine in unity3d though? I'm currently using it for serializing my network packages. |
|
@zeroZshadow: My guess is that you're using protobuf-csharp-port, the old proto2 implementation... which apparently worked on Unity if you were lucky. We repeatedly ran into issues to do with JIT or AOT compilation, suspecting them to be bugs with the very old version of Mono being used. |
|
@jskeet I'm using whatever is here https://github.com/google/protobuf/tree/master/csharp/src/Google.Protobuf together with proto3 |
|
@zeroZshadow: So how are you using that? I wouldn't have expected you to be able to install the NuGet package, for example - it targets CLRv4. Which version of Unity3D are you using? |
|
@jskeet I've copied that entire folder to Assets/Plugins, then applied the fix that is posted by asuuma. |
|
Okay - that makes a bit more sense. You should be aware that this isn't an officially-supported scenario at the moment. I make no guarantees about it - you may well find it works in simply situations, but that some more advanced features break :( Obviously we'd like to support Unity eventually - but the nature of the platform does make it hard to support properly, and we want to get the support for more modern platforms rock-solid first. |
|
@jtattermusch this is relevant to our ongoing work to get GRPC working in Unity. Once I started introducing more complicated messages to my project (instead of the simple Healthcheck I was running), I ran into this I'd also be interested @jskeet 's fork, but need to research how to cross-compile the necessary DLL on OSX, as the default installation instructions obviously yield dylibs. Given that this issue has been open for almost 3 years now... any chance we can drive this home? edit After replacing all instances of |
|
I was able to get grpc up and running on iOS 11 + Unity 2017 + .NET4.6, I documented the process in my blog. Hope this could help for you guys. I was thinking creating a branch for Unity support, but I was only able to get it up and running on iOS and specifically .NET 4.6, so maybe there should be a better way to support this. |
|
Di, I’ve been working with the GRPC team to merge our gRPC forks into
master. At this point we have gRPC-master working on android, iOS, windows,
Linux, and OS X with only minor tweaks.
…On Sat, Mar 24, 2018 at 8:19 AM Di ***@***.***> wrote:
I was able to get grpc up and running on iOS 11 + Unity 2017 + .NET4.6, I
documented the process in my blog
<http://chendi.me/2017/07/30/unity-grpc-plugin-en/>. Hope this could help
for you guys.
I was thinking creating a branch for Unity support, but I was only able to
get it up and running on iOS and specifically .NET 4.6, so maybe there
should be a better way to support this.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#644 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB_P1VN-fbPlFfnXAY681bWVE6zAuWWOks5thmQEgaJpZM4Fgo1X>
.
|
|
For what it’s worth, our iOS implementation was based heavily on your blog.
I cited it in the PR :)
…On Sat, Mar 24, 2018 at 8:31 AM Zane Claes ***@***.***> wrote:
Di, I’ve been working with the GRPC team to merge our gRPC forks into
master. At this point we have gRPC-master working on android, iOS, windows,
Linux, and OS X with only minor tweaks.
On Sat, Mar 24, 2018 at 8:19 AM Di ***@***.***> wrote:
> I was able to get grpc up and running on iOS 11 + Unity 2017 + .NET4.6, I
> documented the process in my blog
> <http://chendi.me/2017/07/30/unity-grpc-plugin-en/>. Hope this could
> help for you guys.
>
> I was thinking creating a branch for Unity support, but I was only able
> to get it up and running on iOS and specifically .NET 4.6, so maybe there
> should be a better way to support this.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#644 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AB_P1VN-fbPlFfnXAY681bWVE6zAuWWOks5thmQEgaJpZM4Fgo1X>
> .
>
|
|
Sounds Great! Awesome work! I'm looking forward to see grpc officially support Unity! |
|
@zaneclaes can you link to the Ad the question how to proceed with improving support:
|
|
@jtattermusch er, this is the Here's my own stacktrace to add to the mix. Note that I still have this issue if I attempt to stringify any of my messages on the iOS client. For example, the following stacktrace is triggered by |
|
@zaneclaes do you know about this one #3794? |
|
Yep; I referenced it in my last two comments @jtattermusch ; again:
|
|
Since I have a work-around by avoiding the use of the methods which trigger the error, I haven't invested in cross-compiling the PR just yet because I'm on OSX ;) |
|
@zaneclaes building the Google.Protobuf assembly should be trivial:
|
|
#3794 has been merged. |
|
Excellent, thanks @jtattermusch and @jskeet ! Sorry I didn't get to test this before it got merged; this isn't my main work and I have limited time each day ;) Look forward to NuGet being updated! |
|
Closing as this appears to be fixed. |
|
@haberman I don't think this is really "fixed." There is limited experimental support for Android alone. It still doesn't work on many platforms, and even those it does work on require a lot of voodoo and undocumented tweaking to use. |
|
@zaneclaes Can you help give a summary on what's working and what's not working? |
|
I can try. The core challenge is that
I may have also made some additional small tweaks to GRPC core, which I'll have to dig a little deeper on. But that covers the main points. FWIW, Unity has its own distribution channel (the asset store), which is probably the better route for distributing a working GRPC build to Unity users, which I'd be very interested in building/maintaining, if I could get some strong support from the rest of the community. |
|
Those sound like gRPC issues - this issue is about just protobuf, which I believe is in a rather better state for Unity now, although it still requires some code to get reflection working correctly. |
|
Oh, gosh. So sorry; I spent much more time interacting with the gRPC side of this and totally overlooked that this issue was in Protobufs. It seems that this can be closed! Apologies. |
|
Thanks for the updates! |
## protobuf-net 分析 [protobuf-net](https://github.com/protobuf-net/protobuf-net) 当前的 GC 问题有： - 序列化 - 反射。函数`ProtoBuf.Serializers.PropertyDecorator.Write`中的`property.GetValue(value, null)` - 装箱拆箱。函数`ProtoBuf.Serializers.PropertyDecorator.Write`中的`Tail.Write(value, dest)` - foreach。`ProtoBuf.Serializers.ListDecorator.Write`中的`foreach (object subItem in (IEnumerable)value)` - 反序列化GC - 反射。函数`ProtoBuf.Serializers.PropertyDecorator.Read`中的`property.SetValue` - 装箱拆箱。函数`ProtoBuf.Serializers.PropertyDecorator.Read`中的`Tail.Read(oldVal, source)` - 列表创建。函数`ProtoBuf.Serializers.ListDecorator.Read`中的`value = Activator.CreateInstance(concreteType)` - 列表扩容。函数`ProtoBuf.Serializers.ListDecorator.Read`中的`list.Add` - byte[]创建。函数`ProtoBuf.ProtoReader.AppendBytes`中的字节数组创建 - Pb对象创建。函数`ProtoBuf.Serializers.TypeSerializer.Read`中的`CreateInstance(source)` ## protobuf-net-gc-optimization 针对 protobuf-net 进行的优化 [protobuf-net-gc-optimization](https://github.com/smilehao/protobuf-net-gc-optimization) - 去反射：对指定的协议进行Hook。 - 反射产生的地方在protobuf-net的装饰类中，具体是PropertyDecorator，没有去写工具自动生成Wrap文件，而是对指定的协议进行Hook。`CustomDecorator`, `ICustomProtoSerializer`及其实现 - foreach - foreach 对列表来说改写遍历方式，没有对它进行优化，因为 Unity 5.5 以后版本这个问题就不存在了 - 无GC装箱 - 消除装箱操作。重构代码，而 protobuf-net 内部大量使用了object 进行参数传递，这使得用泛型编程来消除 GC 变得不太现实。实现了一个无 GC 版本的装箱拆箱类`ValueObject` - 使用对象池 - 对于 protobuf-net反序列化的时候会创建pb对象这一点，最合理的方式是使用对象池，Hook 住protobuf-net 创建对象的地方，从对象池中取对象，而不是新建对象，用完以后再执行回收。`IProtoPool`及实现 - 使用字节缓存池 - 对于 new byte[] 操作的 GC 优化也是一样的，只不过这里使用的缓存池是针对字节数组而非 pb 对象，实现了一套通用的字节流与字节 buffer 缓存池`StreamBufferPool`，每次需要字节buffer时从中取，用完以后放回。 ### protobuf-net-gc-optimization 的其他优化 - BetterDelegate：泛型委托包装类，针对深层函数调用树中使用泛型委托作为函数参数进行传递时代码编写困难的问题。 - BetterLinkedList：无GC链表 - BetterStringBuilder：无GC版StrigBuilder - StreamBufferPool：字节流与字节buffer缓存池 - ValueObject：无GC装箱拆箱 - ObjPool：通用对象池 关键节点： - LinkedList当自定义结构做链表节点，必须实现IEquatable<T>、IComparable<T>接口，否则Roemove、Cotains、Find、FindLast每次都有GC产生 - 所有委托必须缓存，产生GC的测试一律是因为每次调用都生成了一个新的委托 - List<T>对于自定义结构做列表项，必须实现IEquatable<T>、IComparable<T>接口，否则Roemove、Cotains、IndexOf、sort每次都有GC产生；对于Sort，需要传递一个委托。这两点的实践上面都已经说明。 ## 针对 protobuf-net-gc-optimization 的优化 [protobuf-net-gc-optimization](https://github.com/smilehao/protobuf-net-gc-optimization) 使用的`protobuf-net`是 2015 年之前的版本，当前项目使用的是 protobuf-net 2.4.5, 把 protobuf-net-gc-optimization 相关的优化合并到了 protobuf-net 2.x 版本上。 ### foreach `ProtoBuf.Serializers.ListDecorator.Write`中的`foreach (object subItem in (IEnumerable)value)` 因为 C# 不支持泛型协变，上述 foreach 循环还会产生GC，需要针对性的优化。 优化前： ```csharp foreach (object subItem in (IEnumerable)value) { if (checkForNull && subItem == null) { throw new NullReferenceException(); } Tail.Write(ValueObject.TryGet(subItem), dest); } ``` 优化后： ```csharp if (value is IList list) { for (int i = 0; i < list.Count; i++) { var subItem = list[i]; if (checkForNull && subItem == null) { throw new NullReferenceException(); } Tail.Write(ValueObject.TryGet(subItem), dest); } } else { foreach (object subItem in (IEnumerable)value) { if (checkForNull && subItem == null) { throw new NullReferenceException(); } Tail.Write(ValueObject.TryGet(subItem), dest); } } ``` ### 其他优化 `protobuf-net`的`BufferPool`在(2018.6.8)[https://github.com/protobuf-net/protobuf-net/commit/9718b9221ee0c2aa13509d0a258a0728d3fc3210#diff-3df8aa4e7ab0d7118f25612197fbe78d]修改为`弱引用(WeakReference)`实现内部缓存，之前的[老版本]为(https://github.com/protobuf-net/protobuf-net/commit/15fa224b3ceab2cdf99012d999307b3435936665#diff-3df8aa4e7ab0d7118f25612197fbe78d)。弱引用会导致 Unity Profile 时，每次调用缓存失效，创建新的对象（56B）。这里使用老的版本。 ## 需要再次确认的代码 - `ProtoBuf.BufferPool`内部有锁，用于 ProtoWriter，ProtoReader 读写, 能否优化掉 - `ProtoBuf.ProtoReader.AppendBytes`: `protobuf-net-gc-optimization`的注释（// TODO：这里还有漏洞，但是我们目前的项目不会走到这）需要再次确认 ## 测试结果 `Assets/TestScenes/TestProtoBuf/TestProtoBuf.unity` 及 `TestProtoBuf.Test5` 测试结果, 开启 deep profile： | | 序列化 GC/time | 反序列化GC GC/time | | ----------- | -------------- | ------------------ | | 优化前₁ | 0.8k/0.21ms | 1.3k/0.23ms | | 优化后₂ | 80B+56B/0.23ms | 0B+56B/0.20ms | | 再次优化后₃ | 0B | 0B | 1. https://github.com/protobuf-net/protobuf-net 代码 2. https://github.com/smilehao/protobuf-net-gc-optimization 修改合并到 protobuf-net 2.4.5 后。两次List枚举器的获取，每次40B 3. 2.4.5-gc-optimization 代码 ## google protobuf 分析 [protocolbuffers/protobuf 3.13.0](https://github.com/protocolbuffers/protobuf/tree/v3.13.0)需要 .NET Standard 2.1，Unity 只支持 .NET Standard 2.0，需要添加额外的DLL[1](protocolbuffers/protobuf#7668), [2](https://github.com/protocolbuffers/protobuf/issues/7252)。 ## 参考资料 - [九：Unity 帧同步补遗（性能优化）](https://zhuanlan.zhihu.com/p/39478710) - [Unity3D游戏GC优化总结---protobuf-net无GC版本优化实践](https://www.cnblogs.com/SChivas/p/7898166.html) - [Google protobuf 重用缓存方法](protocolbuffers/protobuf#644) - [unity 官方 GC 优化教程 - Fixing Performance Problems - 2019.3](https://learn.unity.com/tutorial/fixing-performance-problems-2019-3?uv=2019.3#)
## protobuf-net 分析 [protobuf-net](https://github.com/protobuf-net/protobuf-net) 当前的 GC 问题有： - 序列化 - 反射。函数`ProtoBuf.Serializers.PropertyDecorator.Write`中的`property.GetValue(value, null)` - 装箱拆箱。函数`ProtoBuf.Serializers.PropertyDecorator.Write`中的`Tail.Write(value, dest)` - foreach。`ProtoBuf.Serializers.ListDecorator.Write`中的`foreach (object subItem in (IEnumerable)value)` - 反序列化GC - 反射。函数`ProtoBuf.Serializers.PropertyDecorator.Read`中的`property.SetValue` - 装箱拆箱。函数`ProtoBuf.Serializers.PropertyDecorator.Read`中的`Tail.Read(oldVal, source)` - 列表创建。函数`ProtoBuf.Serializers.ListDecorator.Read`中的`value = Activator.CreateInstance(concreteType)` - 列表扩容。函数`ProtoBuf.Serializers.ListDecorator.Read`中的`list.Add` - byte[]创建。函数`ProtoBuf.ProtoReader.AppendBytes`中的字节数组创建 - Pb对象创建。函数`ProtoBuf.Serializers.TypeSerializer.Read`中的`CreateInstance(source)` ## protobuf-net-gc-optimization 针对 protobuf-net 进行的优化 [protobuf-net-gc-optimization](https://github.com/smilehao/protobuf-net-gc-optimization) - 去反射：对指定的协议进行Hook。 - 反射产生的地方在protobuf-net的装饰类中，具体是PropertyDecorator，没有去写工具自动生成Wrap文件，而是对指定的协议进行Hook。`CustomDecorator`, `ICustomProtoSerializer`及其实现 - foreach - foreach 对列表来说改写遍历方式，没有对它进行优化，因为 Unity 5.5 以后版本这个问题就不存在了 - 无GC装箱 - 消除装箱操作。重构代码，而 protobuf-net 内部大量使用了object 进行参数传递，这使得用泛型编程来消除 GC 变得不太现实。实现了一个无 GC 版本的装箱拆箱类`ValueObject` - 使用对象池 - 对于 protobuf-net反序列化的时候会创建pb对象这一点，最合理的方式是使用对象池，Hook 住protobuf-net 创建对象的地方，从对象池中取对象，而不是新建对象，用完以后再执行回收。`IProtoPool`及实现 - 使用字节缓存池 - 对于 new byte[] 操作的 GC 优化也是一样的，只不过这里使用的缓存池是针对字节数组而非 pb 对象，实现了一套通用的字节流与字节 buffer 缓存池`StreamBufferPool`，每次需要字节buffer时从中取，用完以后放回。 ### protobuf-net-gc-optimization 的其他优化 - BetterDelegate：泛型委托包装类，针对深层函数调用树中使用泛型委托作为函数参数进行传递时代码编写困难的问题。 - BetterLinkedList：无GC链表 - BetterStringBuilder：无GC版StrigBuilder - StreamBufferPool：字节流与字节buffer缓存池 - ValueObject：无GC装箱拆箱 - ObjPool：通用对象池 关键节点： - LinkedList当自定义结构做链表节点，必须实现IEquatable<T>、IComparable<T>接口，否则Roemove、Cotains、Find、FindLast每次都有GC产生 - 所有委托必须缓存，产生GC的测试一律是因为每次调用都生成了一个新的委托 - List<T>对于自定义结构做列表项，必须实现IEquatable<T>、IComparable<T>接口，否则Roemove、Cotains、IndexOf、sort每次都有GC产生；对于Sort，需要传递一个委托。这两点的实践上面都已经说明。 ## 针对 protobuf-net-gc-optimization 的优化 [protobuf-net-gc-optimization](https://github.com/smilehao/protobuf-net-gc-optimization) 使用的`protobuf-net`是 2015 年之前的版本，当前项目使用的是 protobuf-net 2.4.5, 把 protobuf-net-gc-optimization 相关的优化合并到了 protobuf-net 2.x 版本上。 ### foreach `ProtoBuf.Serializers.ListDecorator.Write`中的`foreach (object subItem in (IEnumerable)value)` 因为 C# 不支持泛型协变，上述 foreach 循环还会产生GC，需要针对性的优化。 优化前： ```csharp foreach (object subItem in (IEnumerable)value) { if (checkForNull && subItem == null) { throw new NullReferenceException(); } Tail.Write(ValueObject.TryGet(subItem), dest); } ``` 优化后： ```csharp if (value is IList list) { for (int i = 0; i < list.Count; i++) { var subItem = list[i]; if (checkForNull && subItem == null) { throw new NullReferenceException(); } Tail.Write(ValueObject.TryGet(subItem), dest); } } else { foreach (object subItem in (IEnumerable)value) { if (checkForNull && subItem == null) { throw new NullReferenceException(); } Tail.Write(ValueObject.TryGet(subItem), dest); } } ``` ### 其他优化 `protobuf-net`的`BufferPool`在(2018.6.8)[https://github.com/protobuf-net/protobuf-net/commit/9718b9221ee0c2aa13509d0a258a0728d3fc3210#diff-3df8aa4e7ab0d7118f25612197fbe78d]修改为`弱引用(WeakReference)`实现内部缓存，之前的[老版本]为(https://github.com/protobuf-net/protobuf-net/commit/15fa224b3ceab2cdf99012d999307b3435936665#diff-3df8aa4e7ab0d7118f25612197fbe78d)。弱引用会导致 Unity Profile 时，每次调用缓存失效，创建新的对象（56B）。这里使用老的版本。 ## 需要再次确认的代码 - `ProtoBuf.BufferPool`内部有锁，用于 ProtoWriter，ProtoReader 读写, 能否优化掉 - `ProtoBuf.ProtoReader.AppendBytes`: `protobuf-net-gc-optimization`的注释（// TODO：这里还有漏洞，但是我们目前的项目不会走到这）需要再次确认 ## 测试结果 `Assets/TestScenes/TestProtoBuf/TestProtoBuf.unity` 及 `TestProtoBuf.Test5` 测试结果, 开启 deep profile： | | 序列化 GC/time | 反序列化GC GC/time | | ----------- | -------------- | ------------------ | | 优化前₁ | 0.8k/0.21ms | 1.3k/0.23ms | | 优化后₂ | 80B+56B/0.23ms | 0B+56B/0.20ms | | 再次优化后₃ | 0B | 0B | 1. https://github.com/protobuf-net/protobuf-net 代码 2. https://github.com/smilehao/protobuf-net-gc-optimization 修改合并到 protobuf-net 2.4.5 后。两次List枚举器的获取，每次40B 3. 2.4.5-gc-optimization 代码 ## google protobuf 分析 [protocolbuffers/protobuf 3.13.0](https://github.com/protocolbuffers/protobuf/tree/v3.13.0)需要 .NET Standard 2.1，Unity 只支持 .NET Standard 2.0，需要添加额外的DLL[1](protocolbuffers/protobuf#7668), [2](https://github.com/protocolbuffers/protobuf/issues/7252)。 ## 参考资料 - [九：Unity 帧同步补遗（性能优化）](https://zhuanlan.zhihu.com/p/39478710) - [Unity3D游戏GC优化总结---protobuf-net无GC版本优化实践](https://www.cnblogs.com/SChivas/p/7898166.html) - [Google protobuf 重用缓存方法](protocolbuffers/protobuf#644) - [unity 官方 GC 优化教程 - Fixing Performance Problems - 2019.3](https://learn.unity.com/tutorial/fixing-performance-problems-2019-3?uv=2019.3#)
…th reflection and static code, see protocolbuffers#644
and others
quoting @jskeet:
"There's suspicion the current version of C# protobufs won't work with Unity, but we can add another build target for that later when we find a need - we can create a new minor version of the NuGet package for that with no problems.
My experience with the "old" protobuf-csharp-port is that Unity on iOS has various issues, so I think we'll want to investigate them pretty thoroughly before trying to support it anyway - and we may need to conditionally compile out some features. We'll see... but I'd like to avoid having Unity support block a dotnetcore release."
also see #638
The text was updated successfully, but these errors were encountered: