umbracoLock timeout exceeded frequently for members #8433
Comments
|
Thanks for the report. 8.6.4 will have some fixes directly relating to this https://github.com/umbraco/Umbraco-CMS/issues?q=label%3Arelease%2F8.6.4+is%3Aclosed But it's great you've provided a stack trace as we'll need to investigate further with the member logins |
nul800sebastiaan
added
state/needs-investigation
|
Thank you. Looking at this PR #8398 I'm not sure it addresses this problem specifically. Please let me know what other information I can provide to assist identifying a solution. One thing we are doing to reduce the number of ReadLocks on Members is to now internally cache members so we will reduce our calls to read members from the database significantly. Prior we were accessing Members directly from the Member Service on the scale of 10s of times per session.
|
|
Hi @TimZander - we would love it if you could give the upcoming 8.6.4 release a spin, we think it will fix your issue and a few other ones as well. The 8.6.4 build we intend to release is now available from the Artifacts on this build (where it says 3 published): https://dev.azure.com/umbraco/Umbraco%20Cms/_build/results?buildId=43999&view=results We need to await test results from real-life usage to actually test this on your environments to make sure we have now eliminated the problems even in setups that might have different configurations on what we've tested it on. This needs to happen before we can release this build, so the sooner that happens, the sooner we can release! 😄 Please give it a spin and let us know how it goes! |
|
In your case, I'd recommend first upgrading to 8.6.3 and from 8.6.3 to 8.6.4, the only differences are the dlls so you can drop in the new dlls on your target test environment. |
|
Thank you, I will try this today |
|
I have tried replacing the dlls after upgrading to 8.6.3, but it is now asking me to run an upgrade from 8.6.4 from 8.6.3. Does this make any changes to the database, or can I just change the web.config value to 8.6.4 to skip this? |
|
You can just run the installer it doesn't modify your database for patch releases. |
|
Thanks, this has been fine in our testing environment, but we have been unable to replicate the lock timeout in anything but production, so I will be applying these changes to production in a few hours and will see if it makes a difference |
|
Initially the number of theses errors seem to be reduced today, but not gone entirely. I would say they are roughly 1/3 of what they were previously. So far this quantity is not causing issues with stability or operation of the site, but I will continue monitoring |
|
We have a site with over 26000 content items (at least that many rows in umbracoContent), and are having problems with lock request time outs as well and the content tree won't render in the backoffice. The SQL ActivityMonitor shows a lot of lock waits.
That Exception happened shortly after an editor tried to move some content nodes which errored out Which happend shortly after something went really wrong with the move I'm throwing this up here in case it helps or is even related. It's also quite possible I'm doing something I shouldn'. I have a number of events on Published/Saving/transformingIndexValues content to do various things such as send push notices, store some computed data in the ExternalIndex etc. I'll keep reviewing the code I've put into this site, if this sounds related, i'm happy to try 8.6.4. |
|
@cleversolutions yes 8.6.4 will improve what you are seeing with the fixes to scheduled publishing. It may not fix everything entirely, it just depends on what's happening in your project. I would highly recommend you try the 8.6.4 build, it will definitely improve the lock contention for content. Please let us know any results |
|
For long running operations like Move, lets say you are moving a huge amount of data, we cannot avoid SQL lock timeouts. We use SQL locks as the distributed locking mechanism. For long running operations the best we could do in the future would be to improve the handling of these errors and the UI so that we could detect long running operations and tell the user what is going on. There's certainly improvements in parts of the code that probably don't need to take a distributed lock or just not eagerly take one (like scheduled publishing was doing) which will all go a long way to improving this. |
|
Hi @TimZander I've created a PR that you can find here #8525 And other good news is that you can see in the PR that I've fixed a long outstanding TODO in the code to ensure that members are cached by their username and not just by their ID so you can remove your own caches for that. Since you were already caching members this way you would have already seen the performance improvements but perhaps you'll see more I'm unsure. For others that haven't added their own caches and have member heavy sites this will reduce queries tremendously since calls are made to get a member by username on each request. I'll wait until tomorrow for the build to finish since it's late here but if you are around, the latest build for this is here https://dev.azure.com/umbraco/Umbraco%20Cms/_build/results?buildId=45100&view=results If it's successful you will be able to get the artifacts for that build and test it out. If you do let me know how it goes, else i'll respond tomorrow once the build is complete. |
|
@Shazwazza Awesome! Thank you, I will review and apply the artifact ASAP. Would this be expected to be rolled into 8.6.4 when that is released? |
|
@TimZander unfortunately no we've missed that deadline window so this will be an 8.6.5 release |
Shazwazza
changed the title
|
@Shazwazza OK, thanks. |
|
Here's the artifacts if you haven't checked yet https://dev.azure.com/umbraco/Umbraco%20Cms/_build/results?buildId=45100&view=artifacts&type=publishedArtifacts |
|
Thanks @Shazwazza I downloaded the artifacts yesterday as soon as the build completed and have been running them in our staging env. Early next week I'll apply to production and hope for the best! |
|
@TimZander Hi Tim, |
|
@clausjensen I will be applying to production in 9-10 hours. I will follow up in 24 hours with observed results Thanks! |
|
As of this morning we are no longer seeing many of this issue with umbracoLock timeout, however now I am seeing a number of other new exceptions. and and still a few of these |
|
@unaligned We are depending on people trying out these updates on their environments as they are impossible for us to replicate. We're doing everything we can to make this a safe update, there's additional unit tests, etc. But before we put it into a release we'd love to know if it has the intended effect. As to your question, not to be flippant but.. we expect to see them in a NuGet release as soon as we can get some real-life testing done. 🙂 |
|
@nul800sebastiaan @Shazwazza |
|
I just released this in our production environment because we were seeing significant numbers of errors this morning, and it was affecting the backoffice. Things seem be be working more smoothly, but I have another error to report. See stack trace below. |
|
And I'm also still getting some lock request timeouts: |
|
Thanks @unaligned, that is fairly good news regarding these errors:
This one is strange since this isn't a lock timeout, this is an query SQL timeout originating from user lookups Are you able to tell me how many rows you have in your
Do you have the full stack trace of where this query is originating? I mentioned before above (#8433 (comment)) that the lock timeouts in some cases won't be preventable. These SQL locks are used as distributed locks (like c# locks but distributed and with timeouts as to not lock indefinitely). There's probably more places to reduce the contention for these locks like I've been doing with all of the above changes which will help but to do that I need to know where the lock contentions are originating. |
|
@Shazwazza We have 9313 records in the Edit: Oh I see, they're sessions, not user records. |
|
@unaligned those are use login sessions, they are kept for 15 days. Anytime a new user login session is created old ones are purged that are 15 days old. You could double check that is occurring and see if all records in that table aren't older than 15 days? That isn't an enormous amount of records to do a query against and the query is done against the PK so the query is fast, I was just wondering if there was somehow an enormous amount of records in there resulting in a query timeout. I've debugged this query locally and there's no real way to improve the query performance since it's a very simple query against the PK of the table. |
|
Hi @unaligned + @TimZander, Do you have any updates on how your site is working with all of these fixes applied? Are you continually seeing the same errors and if so do you have any info on replication steps and stack traces? Thanks! |
|
Hi @Shazwazza we are still seeing some issues with lock request time out exceeded. I am including the most verbose logs we have of those exceptions Also the issue with duplicate key insert At times it is from where we are calling member.save() The offending code from above is as follows: |
|
Thanks @TimZander , Are you getting less errors with the changes done so far? Some additional questions:
I will re-visit the duplicate key issue and will look to change the lock timeout period (configurable) since it is quite low. |
|
@Shazwazza good suggestion, I'm going to take a deeper look at where and under what circumstances we are saving members. As a general rule we are rarely(but sometimes) saving members automatically, more often we are saving members when a user changes a property on the member such as contact phone number. And yes, we are getting significantly less error counts than prior to the changes |
Yes, when we are updating contact info we may be updating phone and email addresses, and currently we handle those with two unique memberservice.save(member) calls. This will likely be making our code more chatty than necessary. Since we are using the same member object to do those saves I have reduced it to one save at the end of the work. Also we have been assigning roles within a foreach, I have changed that to now assign roles in bulk changed to Another potential pain point is in saving passwords. We are currently using the member service to save passwords like so But I see that is depreciated. It seems the preferred way is using your UmbracoIdentity package https://github.com/Shazwazza/UmbracoIdentity/wiki |
|
@TimZander great stuff, reducing any DB calls will make a big difference. Also keep in mind that you can wrap multiple service calls in a custom Scope which means they will all execute in the same transaction which will be faster too so if you are still requiring multiple service calls in the same method that is an option. MemberService.SavePassword is obsolete the because the Service layer is purely for working with the Database it was never intended to perform any Security/Member/User related logic. In v8 and earlier versions, that responsibility has always been with the ASP.Net membership providers. Those membership providers wrap the MemberService. Unfortunately the deprecated MemberService.SavePassword wrapped the Membership provider ... so now you see there's a circular dependency and no clear separation of responsibility which is why its deprecated (shouldn't have been there in the first place). It's also deprecated because it doesn't support the default password policies which require the old password to change the member's password. My UmbracoIdentity is to change the entire membership security layer to use ASP.NET Identity instead of the old membership providers. This is entirely optional and is up to you if you need/want that and have any use for ASP.NET Identity like if you needed external login/OAuth support. If you don't need ASP.NET Identity then you don't need that package. To change a members password in v8 (and previous versions) you should use the Membership Provider, or you can use Umbraco's MembershipHelper object which is exposed on most base Umbraco classes as |
nul800sebastiaan
added
release/8.7.1
and removed
release/8.6.5
release/8.7.1
state/needs-investigation
|
We'll mark this specific issue as fixed in #8525 - it will be a part of the upcoming versions 8.6.5, 8.7.1 and then all versions higher than that. |
|
Still getting lots of "Lock request time out period exceeded" issues running on 8.9 version. It is happening randomly all across the application including both front end and the backoffice. Please help... Here are few stack traces: {"@t":"2020-12-08T08:21:33.6409996Z","@mt":"Exception ({InstanceId}).","@l":"Error","@x":"System.Data.SqlClient.SqlException (0x80131904): The service has encountered an error processing your request. Please try again. Error code 40549.\r\nThe service has encountered an error processing your request. Please try again. Error code 40549.\r\nA severe error occurred on the current command. The results, if any, should be discarded.\r\n at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action {"@t":"2020-12-08T08:22:50.2511498Z","@mt":"An unhandled exception occurred","@l":"Error","@x":"Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.\n\n-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.\n\n-> System.Data.SqlClient.SqlException: A severe error occurred on the current command. The results, if any, should be discarded.\r\nA severe error occurred on the current command. The results, if any, should be discarded.\n at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action |
|
In our case (Umbraco 8.10.1) we get the "Lock request time out period exceeded" whenever bots are crawling the site and causing a lot of load... System.Data.SqlClient.SqlException
Does anyone know any workarounds for this? This is really business critical. We need a solution or any kind of workarounds here. @vladicaivanovski is your website running on Azure? |
|
I've been getting this more and more since 8.9 and it's progressively getting worse (even after upgrading to 8.11 today). Setup is Dedicated server running SQL Server 2017 (so no azure). Always happens after doing a few publish events (updating content) - only way to fix it is to restart SQL Server which is far from ideal, god knows how i'd do this if it was Azure. Log entry: 2021-01-28 11:49:02,112 [P15776/D2/T17] ERROR Umbraco.Web.Editors.ContentController - Unhandled controller exception occurred for request 'http://site.com.local/umbraco/backoffice/UmbracoApi/Content/PostSave' |
Yes. |
umbracoLock table timing out frequently
Umbraco version
I am seeing this issue on Umbraco version: 8.5.5
Reproduction
If you're filing a bug, please describe how to reproduce it. Include as much
relevant information as possible, such as:
Bug summary
I am seeing many cases of Lock request time out period exceeded.
Up to 2000/day which is causing significant issues using our site.
Looking at database statistics I observed that we are exceeding the hard coded 1800ms timeout on many days.
Specifics
Umbraco is running on Windows Server 2019 on premises
The database is running on Azure, P2 scale.
Steps to reproduce
Any time we have relatively high activity. This often occurs when logging a member in, but it may happen in other instances as that is the most visible scenario
Expected result
umbracoLock performs quickly with no timeouts
Actual result
The text was updated successfully, but these errors were encountered: