master panics #201
Comments
|
And previously exporting did this: |
|
The first panic, we have: which allocates with PUSHPAGE, but even if we did return NULL here, we would panic in Then we call; which does; again, PUSHPAGE, and even if NULL, assigning tr would panic, not the call to We have seen a few panics in the vnop_write path though, could be an issue there? |
|
Second panic So, it definitely initialises |
|
Actually, do you think I could convince you to run the kexts from todays DMG? http://lundman.net/ftp/osx.zfs/osx.zfs-signed-20140627.dmg Nothing special in it, but rules out the compile environment and related bits. |
|
Well the most recent master just took out half of two mirrored vdevs in two pools and resilvering will take a lonnnnnnnnnnnnng time, so I'm a bit keen to avoid having another vdev taken out. Mere panics have never done that. |
|
(In fact the bigger shock was a pool that never imports automatically (it's deliberately never in zpool.cache, ever) managed to get imported, and that is one of the pools resilvering.) |
|
Yeah, I think 95ff805 changed the kext so we don't load cache file, but also changed zfs.util to "list pools" + "import pools". You don't want the new zfs.util to run, until we can make it better. Just the two kexts from dmg is what I mean, no zed, no zfs.util. But I can understand the hesitation. We will have to think about what to do about pools that users dont want imported. |
|
I can do that in the alternate boot environment I have on the same machine, but I'll wait to see how long the resilvering takes. "We will have to think about what to do about pools that users dont want imported." Please also think about not astonishing people who work with other zfs implementations, none of which import pools automatically unless they're in zpool.cache. (e.g., think of cachefile pool property) |
|
Yes, that is the discussion. One, the cache file controls what imports on boot. But also on OSX after boot, if you plug in a device with ZFS on it, the OSX users expect it to be imported and mounted. We have to follow this as well, even if it is not done on other platforms (by default). |
|
Ok, I now have two ways I think I can provoke two different panics (one a double panic that leaves nothing sensible in the crash dump, one that seems to), so I'm now in an alternate boot environment that has never had zfs installed on it. download and mount today's dog, run load_zfs.sh, zpool import: look ok. i'll carry on with tests and update here later. |
|
PS: If you're going to depart wholeheartedly from having a zpool.cache, that's fine, but the documentation job will be important. If you're going to do a small departure, say, automatically import ONLY (!!!) single-device pools, that's fine too. However, I personally would prefer to be as close to openzfs as possible. Another option would be an o3x-specific pool feature that would allow automatic import if active. If it's important that a pool with the feature active be imported on another zfs implementation, sell the idea to openzfs developers. |
|
Hmm... while not conclusive, I haven't gotten it to crash yet. |
|
Ah, but the pools' cache vdevs are ONLINE but are not actually being written to. |
|
Having kext reading cachefile seems to be problematic on OSX since the device names/orders change. Neither master nor vdev-iokit handle that rename yet. One thought is to take the cachefile reading out of kext (no auto import there) and let zfs.util handle it. Just adding some code to zfs.util to read the cachefile, and import only those in there. |
|
YAY! YAY! A panic! |
|
vdev-iokit has a big operational advantage at the moment in that shortly after boot it reliably imports pools with many vdevs and leaves them all (including the cache vdevs) working correctly, every time. master practically always has issues shortly after boot: FAULTED pools, cache vdevs not working, etc. |
|
At the time of the crash above, two pools were resilvering, one of those was the source of some zfs send traffic. A third pool was being beaten up by diskutil verifyvolume concurrently on two zvols. And a fourth was doing minor I/O. IIRC, I think I was doing about 3k IOPS. |
|
So you are saying that vdev_iokit handles the device renaming and autoimport on boot just fine? |
|
Yes, that is what I am saying (about vdev-iokit). Given the number of times I've restarted lately, it is a huge advantage to vdev-iokit. Oh, back and loaded the kexts again. (no cachefile exists to be clear). At import time the cache vdevs again are ONLINE but are not used. Ordinarily (and again, always in vdev-iokit) there would be a couple meg written to them as fast as you could run zpool iostat -v upon import. |
|
Yes! Another panic. This was exporting CLATM after its resilver had finished. Trinity was naturally still resilvering and was also the source of a zfs send. |
|
This time, "zpool import -N Trinity" (after loading the ftp kexts as above) results in cache vdevs not only being ONLINE but also seeing traffic |
|
Certainly seeing that Do you have a second machine/VM that you can run lldb from on panic? or should we see if darwin still can send cores to tftp.. |
|
If 10.8 lldb can talk to 10.9 kernel, then yes, otherwise I could do the tftp trick relatively easily (is it a boot argument?), or cough up another 10.9 box, but probably none of the above until some hours from now. |
|
It can be picking about lldb version, I'm not sure. As long as the kernel debug dmg is exactly the same it might work. Or I can connect from here, but that is awkward with timezones and all that. |
|
Tricky. We'll have to come back to it when I am more functional (timezones...) The job that sends snapshots from 52 datasets to a mother machine just terminated correctly. The main difference is that it was running while the system had very little load (just a resilver, really). Doing the same job while the machine is under heavy load (multiple resilvers/scrubs, heavy activity on zvols) tends to coincide with a panic. Although it cannot be a factor in this boot environment, I do run zfs snapshots on several datasets about once a minute, and the last syslog entry before several BOOT_TIME entries has often been from zfs-auto-snapshot. There are a couple of other tasks (that are independent of any zfs send/receive or snapshot activity) that trigger a panic as noted in the long vdev-iokit thread with @evansus -- I will try to get around to doing that with the ftp kexts tomorrow. He's also been trying to reproduce them, I think. I don't know whether the panics have a single common source or whether heavy load exposes multiple problems. I'm (now) pretty confident it's not hardware or build environment related, though. |
|
Indeed, if you happen to get a panic while we are both on (and pool can be down for ~10mins) log on to irc and maybe we can sort something out. If you are in a LAN, we can create your account on o3x.org for ssh tunnel. Meanwhile, I am wondering if there is some way that It is true the taskq code in SPL is almost a straight paste job from MacZFS, and should probably be inspected for heavy-load cases. But since we used to spawn ~270 threads for each pool, perhaps it is not related. |
|
Do you remember if you still had panics when NOT doing one of the tasks? Like if not zfs send, or if not using zvol? Hmm I guess I would only suspect zvol in that list. Do panics happen with no zvols in use? |
|
Well I seem to be able to provoke a panic in a relatively short timeframe by deliberately firing up real traffic-generating jobs concurrently -- not exactly on demand and not an easy-to-reduce test case, but maybe easier than deciding on what IRC client to use and where I'm leaking my public IP addresses. :-) In answer to your question above, yes, I have had panics when not doing zfs send, and when there have been no pools with zvols imported at all. Beating up zvols makes panics happen more quickly, but it's not necessary. The last panic ( #201 (comment) ) was zvol-free. |
|
Another trigger, for example, is high inbound P2P (the relevant thing is many multiple writing threads rather than one fast source) load onto a slow-to-write pool. No zvols or zfs send involved. Opening hundreds of Safari and Chrome windows and tabs has also been an apparent trigger, mostly because of the cache activity of the browsers vs a squid cache that is on the same pool. Again that's lots of threads/processes writing a lot. |
|
Again, was AFK for a bit and returned to this. There was a big resilver going on, but nothing else out of the ordinary (the load was very light, for instance). |
|
Another slightly different panic, again below zio somewhere, again during a scrub of ssdpool (since that is both fast and is implicated in panics). (zfs @ issue197 + 95374b6 ; spl @ issue201 openzfsonosx/spl@9124b49 ; updating to the latest issue201 in moments) |
|
we have something big we are tracking down at the moment, probably have something for you to try tomorrow. |
|
openzfsonosx/spl@ffdef74 panics every time in the sequence:
(it also fails with zpool import ssdpool) |
|
I've changed the returning same addy twice panic to a printf; it does not happen enormously often so far, mostly during dataset mounts. (edit: "zfs list" on its own triggers one "returning same addy twice" per dataset) |
|
It is the zfs send panics, we are returning memory, again, that is not free. so anything can happen. tracking it down now |
|
@ilovezfs I think the panics are now taken care of with the spl changes. Should we close this and migrate the zfs.fs / importing / zpool-status-shows-wrong-devices stuff to a new issue? |
|
@rottegift perhaps we should use good-old #167 ? |
|
@ilovezfs Sure. It's easy and not too high impact to arbitrarily physically detach and re-attach the cache vdevs which are probably the worst case for renumbering and recovering from that; removing one of the three-way-mirror disks is also not too disruptive and is also likely to lead to renumbering. That also seems to fit with taking several minutes at system startup to probe and reprobe all the disks while rarely actually importing non-trivial pools and sometimes getting the device names wrong on manual import. |
|
@rottegift Please give https://github.com/openzfsonosx/zfs/tree/delayprobe a try |
|
Sure, I'll do so a bit later today. (I'm mainly waiting on a large zfs send/recv). |
|
I got a double panic during the export phase when shutting down to reboot with the delayprobe branch after about 1d14h uptime without problem. Sadly the panic report is entirely useless. |
|
During export? Probably unrelated/new to what we were seeing before at least |
|
@ilovezfs - delayprobe branch imported everything correctly -- all devices ONLINE, all log vdevs accumulating data per zpool iostat -v. Nice. All pools imported (it would be nice to have a way to exclude pools from automatic import somehow) and in arbitrary serial order. Three minutes from nothing to all-imported-OK though is pretty good. |
|
@lundman : well, hopefully it'll recur. Late during shutdown is the best time for a panic, although double panics are always gonna be guesswork. |
|
@rottegift delayprobe isn't doing anything about renumbering after we're already imported. But what it should do, in addition to the import -a during boot, is import a pool if you connect it later. As for limiting what pools get imported, if you change the command in zpool-import-all.sh from zpool import -a to zpool import -c /etc/zfs/zpool.cache -a that may give you what you're looking for. And you can limit what would be imported later if you connect the pool later, by #define'ing ZFS_AUTOIMPORT_ZPOOL_CACHE_ONLY in zfs_util.c. |
|
@rottegift I made the changes I suggested, so that you can try that out: |
|
Ok the commit is pretty straightforward (I'm glad you changed the #ifndef DEBUG logic, defining DEBUG by hand in zfs/cmd/zfs_util/zfs_util.c is easy). I'll try it now and hope It Just Works because I won't be able to spend much time with it for the next 14 hours or so. |
|
@lundman : one normal shutdown since then and now one waiting with blinking lights showing activity on one pool with "ZFS: Reclaim thread is being slow (20001)" logged to the console shortly after verbose shutdown printed "continuing" (which was more than fifteen minutes ago). Not a panic but it's an a awfully long time to shut down cleanly. |
|
Got bored, forced reboot. @ilovezfs : One of the four pools in the cachefile imported in DEGRADED state, the rest did not import at all. debug log entries for zfs.util at |
|
With ssdpool imported and up OK and Donkey still in the DEGRADED state above, did a shutdown -r now. When the system came back up, nothing got imported at all, and these are the only zfs.util log entries. Subsequent "zpool import -N {poolname}" is doing the right thing. |
|
Oops, correction. There were more entries in syslog that weren't being shown by Console app. I should know better, sorry. |
|
@rottegift Yeah, this is not mysterious. It turns out that zpool import -c /etc/zpool.cache -a only does a verbatim import, so looks like zpool import -a will be the way to go for now, with no means of specifying pools that should not be imported. Perhaps at a later date, we can consider such an enhancement if there's significant interest. So the remaining decision is whether, subsequent to the initial zpool import -a during boot, to permit zfs.util to probe and import any pool that gets connected later, or only pools named in the cache file. As a side note, it looks like there may be a bug with zpool import -c when the disk numbers don't match the verbatim disk numbers in the cache file. With it expecting disk5, but the device actually at disk6, check this out: joe-vm:~ joe$ sudo zpool import -c /etc/zfs/zpool.cache -a It's possible it's not a bug per se, in that it's try to find the pool on disk5 (all zeros), but it would be nice to at least have a better error. I suspect this is an upstream issue. |
|
@rottegift Ah, so that bug is caused by the disk5 placeholder being too small (I made it 32 M). Using a larger placeholder for disk5, I get joe-vm:~ joe$ sudo zpool import -c /etc/zfs/zpool.cache -a as you'd expect. |
|
@ilovezfs - I can live with "try to import everything attached at boot time" for now (and until O3X catches up with the latest illumos feature flags, which likely will come from ZoL in some weeks or months). However, I would really like an easy way to not import a pool whose devices are attached later. The particular usage case in mind is when attaching physical disks that are meant to be plumbed up to a virtual machine, but which are importable-in-principle by O3X on the native machine. (I figure that you guys might like that too for if and when you test with vdevs on physical devices rather than files and zvols). |
|
@rottegift Well, limiting the auto-import-laters to only pools listed in cache file should take care of that, no? |
|
@ilovezfs Yes, I think that would be fine. |
|
@rottegift The changes we discussed are merged to master now. |
|
I don't think this needs to be open. If any of the various things above become live problems again, I'll make a new issue. |
zfs mount -a did this:
The text was updated successfully, but these errors were encountered: