What's the future of log abstractions like ASP.NET 5 ILogger, LibLog, EventSource, ... #332
Comments
|
The |
|
thx for your comment, @herecydev ! As long as you only have one (or very few) logging abstractions, everything is ok and it would be exactly as you've described it. This is the scenario I would like to see in the future. However I'm talking about the fact that there is an increasing amount of logger abstractions and application developers have no choice than to write/find adapters for every library. Of course, this isn't the most complex task but it doesn't add any business value and is lost time IMO. I feel like there should have been more discussion with the corefx team to consolidate the existing Microsoft solutions. .NET Core would have been a great chance to deprecate/remove old technologies (e.g. Trace.Write) and promote one solution that is used by all .NET stacks. This would have helped to remove the need for 3rd party libraries like LibLog or CommonLogging. Now one can just hope that this ILogger will be good enough for this purpose and that other library vendors will move to this library when they migrate to .NET Core. PS: I would like to add a few more examples to my list of current abstractions:
|
cwe1ss
changed the title
|
There are a few logging abstractions but the idea is to stick with one, as you change the implementations out during the composition root. I like the The things you have listed are good candidates for implementations of For the mass transit situation it's an established framework with lots of users so It's harder to just say swap over. But as far as I can see nobody has suggest it to them yet. Potential problems with including RC into mass transit is the logging has evolved and is still undergoing changes so you risk alienating users. There is a proposal for using serilog so they're obviously considering the use of a "more common" abstraction. I'm sure we will start seeing more exposure of ILogger out in the wild as it matures into RTM but if you don't suggest it to the library authors don't expect them to go out on whim and implement it. They want to know what the community wants... |
|
Because .net doesn't support structural typing like GO's interfaces and diamond dependencies are problematic I am of the very strong opinion that each independently versioned library that can log stuff, should define it's own The integration packages for Common.Logging is a train wreck. If this issue https://github.com/dotnet/roslyn/issues/7844 is implemented, then what will happen is a series of I look forward to the day I can kill |
|
@damianh thx for sharing your thoughts! May I ask you a few more questions:
and thx for pointing out the roslyn link - I didn't know about that! |
|
We use the approach of having our own ILog, for internal logging, and then integration with the popular loggers (NLog, log4net, Serilog). |
|
Chris, that's my concern with EventSource as well. The concept sounds good for client applications where you only need it for actual troubleshooting scenarios. But for server-side logging, where you want to send your application logs to some place all the time, it just seems way too complicated. I'm wondering if it's "broken by design" for server applications or if it just doesn't get enough love. I really appreciate you guys taking the time to discuss this with me. I think this is a very important topic and maybe we find a way to improve it! For my current project, we decided to use We will try to write/find adapters for other ILogger (Trace, MassTransit.Logging, ...) and forward those to M.E.L or Serilog. We will not monitor any .Net EventSource providers for now. However, we will need to find a way to get some logs from Service Fabric, which uses EventSource. I think we will use Azure Diagnostics to store it in Azure Storage and we will try to fetch it from there. We don't yet have a good solution for performance/metrics - most likely we'll use Stackify. But that's a totally different story. |
|
@cwe1ss I sympathize with your struggle and wish I had better answers for you. I think some of the problem you are describing is there because we didn't had a common logging abstraction from the beginning and others because one logging abstraction doesn't rule them all. History Currently EventSource/ETW EventSource/ETW becomes even more powerful when augmented with data from the frameworks you use and your application. In a core world it is intended that if you need to consume EventSource/ETW level data, you setup a Microsoft.Extensions.Logging sink to push to EventSource/ETW. Here you will all the information you could ever want in one place. You might be asking why not just settle on using EventSource/ETW instead of Microsoft.Extensions.Logging. The problem here is that EventSource/ETW has been around for a very long time and updating programming models/interfaces here to match what we want from a modern logging platform isn't really that viable. Hence, you could look at Microsoft.Extensions.Logging existence as being this update, but done in such a way that it doesn't force you to use EventSource/ETW if you didn't want to. DiagnosticSource Traditionally this has been fine, but with the advent of newer diagnostics tools (like Glimpse) and richer requirements from others (like Visual Studios various diagnostics tools), getting access to non string data and actual rich objects has become a must. Hence DiagnosticSource was created for consumers who exist in process, are interested in distinct events (i.e. begin/end request, begin/end action, etc) and want access rich objects (i.e. the actual HttpContext object, etc). This requirement can't simply be filed by Microsoft.Extensions.Logging and EventSource/ETW and is really trying to enable a while new class of scenarios. If you are interested in knowing more about DiagnosticSource, I'm happy to fill you in more. Framework Authors Some who have ties to the operating system or azure infrastructure will choose to work with EventSource/ETW and others who are targeting OWIN based infrastructure might go with Microsoft.Owin.Logging. This all leads to the onus being on the consumer to decide which sink they want to ultimately push their data to. I hope this helps to provide a bit more context to struggles you (and other in situations like yours) are having. We do hear you and we are trying to improve things for the future, but even though we whole platform is being reset, logging is one of those things that is hard to completely reset when you start to introduce legacy components or the underlying hosting environment/infrastructure. |
Yes, I hold that opinion currently. Strong naming is still a problem in .net, though I'm not quite sure how less so in CoreCLR.
I hold the same opinion as @phatboyg on this. |
|
Thank you for this detailed post, Anthony! It's great to get some background on I read a lot about EventSource lately and I agree that it is a very powerful tool. I really like the concept and the fact that it is used by the framework itself. It would be a great place to get everything you need. But as you've said Microsoft.Extensions.Logging seems to be the easier API to use. That's why we now decided to use this in our code to create log events. From a conceptual point, I keep thinking that forwarding M.E.L to EventSource would be the right thing. (I even wrote a forwarder, in case someone needs it: #328). I'd be willing to help but I don't feel like there will be many other interested people. So there won't be a big community and there won't be many sinks I guess (which of course isn't that important because writing sinks is quite easy). My goal now is to get as much stuff as possible into M.E.L (our own logs, 3rd party ILogger) and to keep the code which actually configures the sinks in one library that's used by all of our applications. This way it should be fairly easy to switch from Serilog to EventSource. |
|
@cwe1ss I have been given the task of doing the post and I'm about half way through. I've been busy writing too much code which uses it ;) I think your path forward looks good and is the best approach in your situation. |
|
First of all I want to apologize to @cwe1ss for not replying to is question more promptly. As he suspects, the basic problem here is that logging systems don't really get 'enough love' and that is perhaps the root of most of the problem. The fact that I have so little time to respond to his very legitimate questions is just evidence of that. The other root of the problem is that logging systems are actually not as easy as you might think. You of course start out simple, you just want string logging, but then you want structure to your data, and abstraction, and contracts, and extensibility, and versioning, and mechanical processing, and filtering and complex objects, and high efficiency, and interfaces to database, and cloud storage, and coupling with other logging system, and of course on top of all this you want one homogeneous system that does all this. Indeed the .NET Framework started out with System.Diagnosics.Trace (and Debug), which was just simple string logging, but no one considers that the way forward now. We keep it because lots of code used it, and frankly while it is true that .NET Core provides an opportunity to depreciate things, it also creates a lot of work just to make the platform attractive. You only have a limited amount of manpower so you have to make choices, and when you take things away, you have to REALLY believe the long term benefit is LARGE because people will just view it as a disincentive to adopt the new platform. Thus we kept System.Diagnosics.Trace but it is more of a compatibility hook than anything else. Messages from this may flow into the other systems, but that is the extent of our long term vision for System.Diagnostics.Trace. So if we ignore System.Diagnostics.Trace we have three logging technologies
Note also, that in all cases, the most important goal of these systems is to do the things that only the framework can do easily, which is to set standard, and to instrument the framework itself. Thus the first priority is concerned with GENERATION of events not what happens to them afterward (because people outside the framework can do that). This get back to the 'enough love' issue. We do think have our priorities straight, but what it means is that 'back end' things are left as an exercise to the users and this is perhaps the main complaint people have (rightly so, but it is a point in time thing). Given that the goal of the logging system is to set standard, it is really quite unfortunate that we have three of them. Here are the justification for each: EventSource: This is the one that existed first, and has advantages that insure that it very unlike to go away. In particular
However the strong typing of EventSource has the same problems that strong typing has in programming languages. People like myself LOVE the strong typing because it allows errors to be caught. However weakly typed languages are all the vogue right now because people feel they are too complex and hard to write for. I think you will never get across-the-board agreement on the merits of typing/strong contracts. I think you can imagine a world were we only have EventSource (because you can always weaken contracts), but it has not gotten 'enough love' to match the back-end capabilities of other loggings systems. As an aside, I have heard more than one of you say EventSource as too complex or hard to use. I am interested in details on that. I suspect it is mostly around not getting 'enough love' in the form of 'off the shelf' back ends, and instructions for use, but I am only guessing, I would like to know more. DiagnosticSource. Diagnostic source is the newest logging interface. It has only existed for months so far. The main reason for its existence is that every other logging technology assumes that you want your log message to LEAVE THE PROCESS. That means it has to be SERIALIZABLE. But what if you want to 'log' 'big complex objects' (like a HttpContext, or an HttpReqest), very efficiently (e.g. for an in-process debugger/profiler). These objects are not serializable, so we have a problem, and moreover even if they were, it would be too inefficient to serialize them anyway. This is what DiagnosticSource does. It was meant to do VERY LITTLE. It really is simply an 'in process hook' publication mechanism. It is a way for code to say 'this is interesting so I have given it a name' and you can then subscribe to that point and get the hook. Arguably this is not logging in the normal sense of the word. Its clients MUST be in-process, and while you can log strings an integers, you can also log things like HttpContext. It is really more like setting up event callbacks than it is logging, but unlike normal events in C# the contract with these events is very loose, discoverable, and configurable at runtime and not compile time. Thus Diagnostic source is really 'weakly typed eventing', with the strong understanding that it is READ ONLY (you should only be looking at the objects when you get your callback). One could argue that DianosticSource is not really a logger but something more like a specialized event publishing scheme. However it does have synergy with other loggers (we are building a bridge from DiagnosticSource to EventSource right now in fact) so it certainly is relevant here. ILogger. So this is the logger that looks the most familiar to people. In fact this logger was built by the ASP.NET Team a few years go with the specific goal of ABSTRACTING EXISTING LOGGERS. The ILogger interface was specifically designed so that things like Serilog, NLog etc, could easily be plumbed underneath it. It is not surprising then that it performs that function well and got 'love' for that particular scenario (there are adapters for the Serilog and other loggers). Like EventSource, ILogger is first and foremost simply a way of decoupling the instrumentation site from the back end of the logging pipeline. Thus you only have to depend on ILogger and not Serilog or NLog etc. Since most loggers just want to print formatted strings (and the Serilog went further and allowed the arguments to the formatter to be captured and delated), this is the model that ILogger exposes. If your goal is to capture strings (or the arguments to formatted strings), and put them in some cloud search engine, this is the scenario that ILogger targets. It is also the interface that the ASP.NET team is using for a lot of its logging. So Where does this leave us? As mentioned, having three logging systems seems sub-optimal, but you can also see that there are at least reasons for having three. You can even argue that DiagnosticSource is not a logging system, but even if you consider it one, the fact that you pass object that can 'leave' the process is certainly a major design difference (and makes it very different that the other two). We actually toyed with trying to make EventSource subsume the functionality of DiagnosticSource, but ultimately decided that it would be MORE confusing to have once class with two very different scnearios (one when you log for things you expect to leave the process and one when you don't). The design we came with instead is that there would be a bridge from DiagnosticSource to EventSource. Thus EventListeners (and ETW) could listen to anything that a DiagnosticSource could produce. However there is still the problem that DiagnosticSource produces unserializable things and EventSource can only accept Serializable things. The basic solution to this is that when subscribing to a DiagnosticSource from an EventSource you can specify what piece of information you need (e.g. you might take only the URL field of the otherwise unserializable HttpRequest object) and thus convert things to data that can be serialized. So this leaves us with the question of ILogger vs EventSource. We actually spent quite a bit of time trying to resolve what the proper guidance should be here, and even now we are not really satisfied. We tried to harmonize the two but the 'traditional' logger community (e.g. Serilog, NLog etc), felt we were not adding value. On the other hand, ILogger it really no efficient enough for 'high volume' cases like the task library or networking where the contracts are actually really valueable. Thus it seems that both will exist. The mismatch can be helped with a bridge. I want to thank @cwe1ss for posting his bridge. Bridges encode many design decisions (exactly how various things are encoded), and ultimately we are likely to fiddle with those before finally setting on something, but your bridge is a useful starting point in the discussion and has been very helpful. I know it may not seem like we are doing anything (arguably we should be putting more 'love' into it), but we are making progress, and will be addressing your pull request reasonably soon). \ You certainly could imagine other bridges, like one from EventSource to ILogger. That is certainly on the table, but it is not clear what the scenario is yet. The expectation is that this would be used to dump framework information into other logger back-ends, and if that is a compelling scenario than we will probably build such a bridge (or users can). So What's the Guidance? If you live in the world of .NET Core and like classic loggers like Serialog/Nlog etc, then you certainly should be using ILogger. If you are in the framework, you are going to continue to use EventSource. We will have bridges (or you can build your own if you can't wait for us). So people can get at the information whatever the choice. Diagnostic source is also a possibility, but frankly I would not expect this to be commonly used to instrument USER code. It may be useful for users to GET some of the information from a Diagnostic Source, but again I would only expect this from the most advanced users. The guidance is certainly not as simple and complete as we would like. But more fundamentally, logging is really not 'that hard' in the sense that if you make a mistake and have to change your logging, it is not that hard to do that. Thus things can be fixed. Moreover loggers can be bridged. Thus you are never likely to be truly blocked. The main issue is one of complexity. Ideally you want all this to be 'simple', and it is sad that it is not. If you wish to ignore two of the three logging system for the sake of simplicity that is a reasonable approach. At the end of the day, logging is not an end but a means to a large end. If you get the information you need then we really don't mind how you do it. Sure it would be nice if it were simpler. We are open to suggestions.... |
|
@vancem thanks again very much for taking the time to respond and for doing so in such a detailed way. I think all of you are doing a great job and I'm really impressed by everything that is happening in .NET Core and ASP.NET Core! I also think that having an open conversation like this with so many leaders of this area is a sign that open source in the .NET community is really kicking off! Your explanations are very helpful! Your desription of Thank you again to everyone who has joined this conversation so far! As a next step, I will try to write a list of recommendations and possible next steps. Maybe we can agree on some of those and maybe we find some easy things to improve the current situation. [and to make sure all of our keystrokes in this conversation haven't been for nothing] :) |
|
I'm closing this since I haven't been able to come up with something useful and there haven't been further comments. Again, I'd like to thank everyone involved for taking the time to share his opinion! |
Hi everyone! This is a rather philosophical question but still, it would be great if I could get some opinions from you guys!
I'm currently working on a quite big new "microservices"-like project with many services. Routing all log entries into one central repository is a key to success in my opinion. Therefore I've been reading a lot about this topic lately and unfortunately the more I read the less I like the current logging ecosystem in .NET.
It's quite common nowadays to rely on many different libraries - NuGet makes it incredibly easy to just drop in one library after the other into your project. Before systems like LibLog came up, there were hard dependencies on log4net or others - so LibLog and so on were a great step forward.
However I feel like what was done wrong with the direct depencies to logging frameworks is now just put up one level and repeated for "logger abstractions" or "logging emitters" (in other words: classes like a typical ILogger that actually >create< log entries)
I'm working on a greenfield project but still I already had to deal with the following logging emitters:
None of these emitters actually store logs somewhere and if you don't configure them they just log into void. I'm having a really hard time to figure out the best way to get all of them into one repository. I'm currently routing most of them through Serilog (by writing a few adapters) but still, it doesn't work well (I don't get some EventSource entries, ...). What also frustrates me about this is the fact that you can't easily reuse this logic throughout your applications since every application depends on different abstractions. This makes it really painful in a microservice environment. I don't want my developers to think about this topic whenever they create a new service.
I'm also frustrated that all of this configuration is just for one technology stack in a very big logging architecture. There are so many other things that have to be aggregated as well. (infrastructure logging like nagios, windows event logs, IIS logs, ...)
So I don't understand why there isn't one good solution in the .NET framework itself for such an important topic. System.Diagnostics.Tracing.EventSource seems to be the most recent solution but it feels like it doesn't get enough attention and the tooling around it isn't very good in my opinion. Its rather static nature also makes it somehow hard to adopt in a system with many components. Much has been improved lately, but there are still many things that are hard to understand. E.g. would the ASP.NET 5 ecosystem have one EventSource per library? ... I still haven't got my head around it completely...
And if there's no sufficient solution in the framework itself, why haven't we as a community managed to reach 1 or 2 de-facto standards within the 15 years of .NET?
What do you think where we should head as a community?
Should we invest more into a built-in system like "EventSource" which seems to be the recommended solution from the .NET team? Its adoption within the framework itself basically makes it a "problem" we have to solve anyway in my opinion.
Should a system like LibLog be able to aggregate some of these other abstractions??
Should we actually go "back" and have hard dependencies on a system like Serilog that can emit and store events?
Or do we need yet another layer which basically does the opposite as the existing systems and acts like an aggregater who can read events from Trace, TraceSource, EventSource, LibLog, Common.Logging, ASP.NET 5 ILogger, ... and offers them in one central stream which then can be consumed by e.g. log4net/serilog - and how would such a system handle DLL dependencies??
And really important - where would be a good place to discuss this issue with the major stakeholders? Because the most important thing in my opinion is to NOT end up with yet another ILogger.
@nblumhardt, @vancem, @davidfowl, @damianh - As owners/main contributors to some of these systems, I hope it's ok to cc you for this question especially. Is this something you think should be discussed in more detail??
Or do you think I'm seeing a problem where there isn't one? :)
I'd really appreciate your feedback!!
The text was updated successfully, but these errors were encountered: