NFS based temporary lockups under load #1060
Comments
|
Can you describe the NFS load which is causing the problem? It appears to be issuing lots of fsyncs() which will probably be slow without a dedicated ZIL device. The only thing which makes me think there might be more going on here is your huge CPU usage. If it was just IOPs bound I wouldn't expect to see that. We could look in to this more closely if you have a reproducer for the problem. |
|
The load is an oracle database using NFS storage. 8K recordsize is used for the zvol created. We have a large number of cores on these systems (80) and a large amount of data (Lidar pointclouds) being written into the database. When the system goes out of this phase (which seems to last about 5-15 min at a time) , load drops down to a normal level of about 3-5. |
|
#922 again? |
|
What is your pool configuration? Also, which version of ZFSOnLinux is this? Have you tried HEAD? |
|
This could be related to #922, @msmitherdc if you get a chance you could try the following patches and see if it improves things. https://github.com/behlendorf/spl/branches/stats |
|
@msmitherdc The suggested patches have been merged in to master. If you get a chance please update to the latest master source and let us know if it helps. |
|
The patches seems to have made a difference however the load has been lower. I'll rebuild with master and try to stress it this weekend but it does look promising. From: Brian Behlendorf <[email protected]mailto:[email protected]> @msmitherdchttps://github.com/msmitherdc The suggested patches have been merged in to master. If you get a chance please update to the latest master source and let us know if it helps. — |
|
Just make sure you use the latest SPL and ZFS |
|
On sha 4c837f0 and under high load (oracle database with lots of large object write activity) INFO: task txg_sync:5783 blocked for more than 120 seconds. — |
|
@msmitherdc The stack shows the txg_sync thread hung in a likely a write IO is it possible it's being starved by a huge number of read IOs in your workload. I assume that eventually the txg sync completes? |
|
That is correct. It eventually does complete. Are we just over saturating the IO? We were able to maintain this volume under solaris. From: Brian Behlendorf <[email protected]mailto:[email protected]> @msmitherdchttps://github.com/msmitherdc The stack shows the txg_sync thread hung in a likely a write IO is it possible it's being starved by a huge number of read IOs in your workload. I assume that eventually the txg sync completes? — |
|
It's hard to say. The fact that your hung in zio_wait() just indicates that you've submitted an IO to the pipeline and are waiting for it to complete. Exactly why it's taking so long would require more digging. Recently I've seen something very similar to this at LLNL (very very rarely) where ZFS clearly submits the IO to the block layer but it takes 100s of seconds for ZFS to be notified that it completed. In this case I strongly suspect a problem in either the driver, transport layer or SCSI mid layer. Unfortunately, I haven't run it to ground yet. You could check if something similar is happening on your systems by running 'zpool events -v'. You should see a delay event which was generated for this really slow IO. It will include the total delay time (in miliiseconds) that the IO took after it was handled off to the block layer. |
|
I do see wait events showing up showing up after a large number of statechange events Nov 10 2012 17:35:16.795457000 resource.fs.zfs.statechange Nov 10 2012 17:35:17.362036000 resource.fs.zfs.statechange Nov 10 2012 17:35:17.591634000 ereport.fs.zfs.vdev.unknown From: Brian Behlendorf <[email protected]mailto:[email protected]> It's hard to say. The fact that your hung in zio_wait() just indicates that you've submitted an IO to the pipeline and are waiting for it to complete. Exactly why it's taking so long would require more digging. Recently I've seen something very similar to this at LLNL (very very rarely) where ZFS clearly submits the IO to the block layer but it takes 100s of seconds for ZFS to be notified that it completed. In this case I strongly suspect a problem in either the driver, transport layer or SCSI mid layer. Unfortunately, I haven't run it to ground yet. You could check if something similar is happening on your systems by running 'zpool events -v'. You should see a delay event which was generated for this really slow IO. It will include the total delay time (in miliiseconds) that the IO took after it was handled off to the block layer. — |
|
No good, we're looking for a |
|
@msmitherdc @behlendorf Considering this is old (almost a year and a half) and against an old/ancient version of ZoL and kernel, should we keep this open or close it as stale? |
|
close it as stale |
Bumps [syn](https://github.com/dtolnay/syn) from 2.0.26 to 2.0.27. - [Release notes](https://github.com/dtolnay/syn/releases) - [Commits](dtolnay/syn@2.0.26...2.0.27) --- updated-dependencies: - dependency-name: syn dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
on rc11
When this happens, NFS is non responsive but will eventually return in 5-15 minutes. When it does happen, I see messages like below
Top will look like this
This is RedHat EL 6.2 - 2.6.32-220.el6.x86_64
The text was updated successfully, but these errors were encountered: