aspnet-async first implementation

[slorion]

This benchmark suite is using the new asynchronous features of .NET 4.0 and MVC4, and also a little known JSON serializer (Jil).

[msmith-techempower]

There already exists a test with the name 'aspnet'. Could this be changed to 'aspnet-async'?

[slorion]

Sure, I just committed the change! I did not realize that the framework property referred to the benchmark name.

[slorion]

Maybe not the best place to ask, but would it be ok to tweak machine level ASP.NET configuration via the benchmark script ? There are some threading properties that can only be set machine-wide and have a big impact on IIS performance under heavy load. http://msdn.microsoft.com/en-us/library/ee377050.aspx lists some of them.

[msmith-techempower]

I don't think there is anything wrong with tweaking the IIS settings to be more performant so long as they are production-quality changes (read: not just hax that would make the results look better but not something that would be used on a production machine in practice).

[pdonald]

Previous async implementations/discussions:

#499
#272 (comment)
#315

[slorion]

According to DevART, their dotConnect provider for PostgreSQL does implement async operations (http://www.devart.com/dotconnect/postgresql/features.html), so I am curious to see how it fares under load. Testing with SQL Server database would still be nice thought as I suspect that is the most common db used with asp.net. As for MySQL, the official provider also supports async since 6.8.0 (https://blogs.oracle.com/MySqlOnWindows/entry/mysql_connector_net_6_8).

[pdonald]

I just had a quick look at the MySQL connector source code and the "async support" is just wrapping existing synchronous methods with Task.Factory.StartNew.

I'm not an expert on asynchrony, but I believe this all new code is still blocking and does not use I/O completion ports which means all these new asynchronous methods will actually be a little slower because of all the overhead. Correct me if I'm wrong.

[slorion]

Well, sadly you are right. I should have known better and not assumed the calls were truly asynchronous. I checked the code of both the MySQL and dotConnect providers and they are both simply wrapping synchronous calls.

So besides the new JSON serializer and unless a SQL Server database is added as a possible database to the benchmark, this code would basically serve as testing how performance is affected by using the async MVC controllers, combined with the overhead of wrapping database operations. DevART also claims that their provider is faster than npgsql, so it would also let us see how that claim holds in the specific benchmark scenarios.

[hamiltont]

@slorion If you rebase this onto master, and update the .travis.yml file to add a line for aspnet-async, then Travis-CI will verify your changes. We should be good to merge once you pass verification!

EDIT: Just realized this is windows-only. The Travis system can't verify it unless you add a linux-only test option, using something like mono+xsp

[slorion]

Hello, sorry for the delay in answering. Before going ahead, I would like to know if SQL Server will be used as a database in a future benchmark, because as far as I know, it is the only database that supports truly asynchronous calls in the .NET world. In Java/Scala, there is pgjdbc-ng or postgresql-async, but no such driver exists for .NET.

[hamiltont]

@slorion I don't know the answer directly, but I do know this directory exists, which seems to indicate interest in making sqlserver available. I have no idea if these setup scripts 1) work or 2) are using during a round

[slorion]

Ok, thank you! Beyond that, I think there may be licensing issues and also, SQL Server (and other major vendors) have pretty strict rules regarding benchmarks. If you could have a definite answer, that would be great.

[slorion]

If not, what I could do is make another pull request based on an existing aspnet benchmark, with the changes that may improve it.

[hamiltont]

One of the TE guys will have to give a definite answer

About this bit:

pretty strict rules regarding benchmarks

that's fascinating. What kind of rules? Where could I see a quick summary?

I'm guessing they prohibit stuff like publishing benchmark results without approval,
but I'm curious if they also attempt to extend that to benchmarks that use their
server internally. I can't speak for TE here, but it seems to me that this project
benchmarks frameworks, which may happen to use their servers, but this project
isn't doing any sort of metric gathering on the server and therefore is not a
benchmark of their software. Gray area though...

make another pull request based on an existing aspnet benchmark

Oh man I would love that, but let me tell you why:

The project is suffering from having a number of open PRs that
are only testable in windows (I think like 40% of the open ones). We don't have a
lot of active windows-capable contributors at the moment, which is really a bummer.
I really want to see how these frameworks stack up. We've had a
ton of improvements to our ability to rapidly test linux-based tests, so if we could
get aspnet (or any other C# / .NET based system)
to turn on inside of linux then we could actually start to both close the existing .NET
PRs and really support rapidly closing and new PRs that are put in.
#968 is my (noob) attempt to get aspnet to run under mono on linux. With
@pdonald 's help we appear to have reached the point where we are executing the
build and run correctly, but the aspnet framework appears to have been struck by some
code rot and is not compiling for us.

If that's not your cup of tea, there is an open issue #1038 to add continuous integration support
for windows. At the moment, this is a bit stagnant due to a lack of windows-capable contributors. I'm the one that setup the linux continuous integration (Travis-CI), so if you're interested in taking a crack at setting this up I could offer advice.

[slorion]

About publishing benchmark results issue, yes it is indeed a gray area and I am far from an expert on the subject. I can point you to this SO question though: http://stackoverflow.com/questions/12115397/is-it-against-license-to-publish-oracle-and-sql-server-performance-test

As for giving a hand, I must admit my time is very limited at the moment and I was more thinking about doing small improvements to the stripped down version of aspnet benchmark to improve their performance. That said, if you point to specific projects that are not compiling on Mono, I can take a look. It may have to do with new .NET/C# features. Are you using the very latest Mono version ?

I would venture a guess that the people working with the Microsoft stack would be more interested in seeing how .NET works with Microsoft CLR integrated with SQL Server since it is usually what is happening in production (Stack Overflow being one very visible example of that). Not that Mono is not great, it is often underestimated I think, but its use is quite limited compared to Windows/IIS.

[hamiltont]

As for giving a hand, I must admit my time is very limited at the moment and I was more thinking about doing small improvements

Very understandable, just wanted to point out that while a small PR for aspnet would help, it might be months before it could be merged because there's no one to test it (see #587, #762, #803, #804, #968, #833 for a small sample of the open windows (or mono) PRs)

A very significant contribution would be helping us get at least one of the .NET projects running under mono properly, because then at least we could test these PRs and verify that the code changes compile, run, and pass verification.

While I know that "results under mono" sounds a drab when compared to setting it as it would be in production, that is a huge step towards making windows deployment and continuous integration a lot more realistic for our team of linux-scripters. Basically, figuring out powershell isn't too far fetched, but figuring out .NET if you only work with/in linux 100% of the time is a large challenge (when we already have ~15 other languages to learn :-P)

That said, if you point to specific projects that are not compiling on Mono, I can take a look.

C#/aspnet is the number one target here. Here is the latest log that I know of. We are using mono 3.6.0 IIRC

[slorion]

I took a look at the log and all I see are 404 errors when trying to execute the tests. The app source code seems fine, including the routing, so the issue seems to be about registering the web site properly in nginx or XSP. Unfortunately, I am not familiar in how to setup that part.

[hamiltont]

Good to know, thanks

[bhauer]

Regarding licensing terms, some quick notes:

• This is not a database benchmark. Results collected from database-connected test types treat the database as incidental and by design run trivial queries. We are concerned with the overhead of the platform's database driver and ORM. However, as you've pointed out, this could be a gray area.
• If the latest SQL Server's licensing terms indeed do restrict publishing results, we would want to get confirmation from Microsoft that they are comfortable with it being included going forward. I believe we've had a few people fairly well-connected with (if not necessarily employed by) Microsoft review the project. Perhaps we could get one of them to find out who would be able to bless this project.
• Is this document applicable? This is the document I had read when I originally heard of possible Microsoft licensing constraints, and it seems specifically to allow such tests as long as the methodology is clear; the results are reproducible; the environment details are published; source code is available; and the test is done according to best practices. I believe we meet those requirements. However, I am not certain that SQL Server is covered by the ".NET framework."

[slorion]

SQL Server is licensed under another document which contains a clause similar to "You must obtain Microsoft's prior written approval to disclose to a third party the results of any benchmark test of the software." See http://www.microsoft.com/en-us/download/details.aspx?id=2803 for the 2008 R2 version (newer version on MSDN do not seem to contain Standard/Enterprise license terms).

I think your best course of action would be to contact Microsoft and ask them directly. You may also ask them if they would grant you an Enterprise license because the Standard edition performance is often quite below Enterprise (caused by limitations on hardware and features). To which extent with the kind of processing and hardware you test, I do not know, but it should be non trivial. See http://msdn.microsoft.com/en-us/library/cc645993.aspx

[msmith-techempower]

I tend to agree with @bhauer on this one -- we are not benchmarking database performance as much as a stack's ability to communicate with any database in a trivial-but-realistic way.

We want to include MSSQL for anyone who wants to use it, but if it is a source of licensing pain then I would not have any qualms with quietly removing it.

[bhauer]

Given the time budget we have, the easy option of simply not including SQL Server for now is my preference. We can highlight this discussion in the Round 10 announcement notes, and we might be able to catch the eye of people who would like to see SQL Server included, and that in turn may get some Microsoft people to weigh in.

In other words, I'd rather invite them to voluntarily grant permission rather than ask them for special concessions.

[slorion]

I do agree with @msmith-techempower and @bhauer that there should not be any issue with publishing benchmark results, I just cannot give a definite answer on that legal aspect. Maybe you can publish the results and in the very worst case, Microsoft will ask to remove them ? There is still the issue of having either a Standard or Enterprise license. I don't think a MSDN development license is allowed in this case.

[msmith-techempower]

Closing this pull request due to inactivity - please rebase off master and make the appropriate updates and issue a new PR.