zdb -R :d decompress flag is doing everything wrong. zdb_read_block()

[JuliaVixen]

This might be a bug report for the illumos team, rather than zfsonlinux, but I'm sure that this report will eventually get to the right person.

I've got a couple of zpools with a single checksum error, on a single file on them, for whatever reason. And I'm trying to just dump a copy of the corrupt file out with zdb, so I can compare it with my good (from backup) copy to find out what changed. Curiously, zdb doesn't just have an option to dump out all the data contents of a file if given a znode or blkptr.

So, normally, the trick is to just look up the inode, I mean znode for the file (ls -i), and get the DVA's of the indirect blocks.
zdb -ddddd pool 123456
(And some -bbbbbb too if you want to get more detail.)
Then you write a very short perl script, or shell script to chop out all of the L0 DVA[0]=<0:23afca2bc000:134000> parts, and feed them back into zdb -R kinda like this:
zdb -R pool 0:23afca2bc000:134000:r > blocks/0_23afca2bc000_134000.bin
And then cat all the blocks back together, if you didn't just use ">>" instead.

BUT!

If you have compression turned on, these raw chunks of file will be compressed. Well, there's a "d" flag you can add, which should decompress the data, right? So...

I speculate that when you hand zdb just a raw DVA to read, it has no idea about it's logical size, or physical size, or compression algorithm, or checksum, because all of that info is in the block pointer, and it doesn't know which block pointer is pointing at this DVA. All zdb knows is you gave it an address.

So, when you do zdb -R pool 0:d919ae8c000:1c000:dr, zdb will brute-force attempt to decompress that block with every single compression algorithm known to zfs, until one of the decompression functions returns without an error... AND! also attempt the decompression from every possible starting offset in the block. (This explains why CPU load goes up to 100% for twenty minutes to dump a few MB of data.)

Once upon a time, this would probably, eventually, give you a correctly decompressed block... And then came LZ4...

Actually, the problem isn't with LZ4, it's actually with the zero-run-length-encoder. Basically, any random data you hand to ZLE, is always doing to be decompressable without error. So this brute-force, try-every-decompression-function, method will always return your block, ZLE decompressed, as soon as it tries that.... which it will try, before trying LZ4.

It goes in this order (from include/sys/zio.h):

enum zio_compress {
        ZIO_COMPRESS_INHERIT = 0,
        ZIO_COMPRESS_ON,
        ZIO_COMPRESS_OFF,
        ZIO_COMPRESS_LZJB,
        ZIO_COMPRESS_EMPTY,
        ZIO_COMPRESS_GZIP_1,
        ZIO_COMPRESS_GZIP_2,
        ZIO_COMPRESS_GZIP_3,
        ZIO_COMPRESS_GZIP_4,
        ZIO_COMPRESS_GZIP_5,
        ZIO_COMPRESS_GZIP_6,
        ZIO_COMPRESS_GZIP_7,
        ZIO_COMPRESS_GZIP_8,
        ZIO_COMPRESS_GZIP_9,
        ZIO_COMPRESS_ZLE,
        ZIO_COMPRESS_LZ4,
        ZIO_COMPRESS_FUNCTIONS
};

And the particular brute-force loop is this one from cmd/zdb/zdb.c zdb_read_block()

                enum zio_compress c;
                void *pbuf2 = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL);
                void *lbuf2 = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL);

                bcopy(pbuf, pbuf2, psize);

                VERIFY(random_get_pseudo_bytes((uint8_t *)pbuf + psize,
                    SPA_MAXBLOCKSIZE - psize) == 0);

                VERIFY(random_get_pseudo_bytes((uint8_t *)pbuf2 + psize,
                    SPA_MAXBLOCKSIZE - psize) == 0);

                for (lsize = SPA_MAXBLOCKSIZE; lsize > psize;
                    lsize -= SPA_MINBLOCKSIZE) {
                        for (c = 0; c < ZIO_COMPRESS_FUNCTIONS; c++) {
                                if (zio_decompress_data(c, pbuf, lbuf,
                                    psize, lsize) == 0 &&
                                    zio_decompress_data(c, pbuf2, lbuf2,
                                    psize, lsize) == 0 &&
                                    bcmp(lbuf, lbuf2, lsize) == 0)
                                        break;
                        }
                        if (c != ZIO_COMPRESS_FUNCTIONS)
                                break;
                        lsize -= SPA_MINBLOCKSIZE;
                }

A month ago, I hacked zdb.c to just do LZ4 first, and that mostly worked -- the output block was much more correct. But, then I also needed to pass it the psize and lsize to really get the output block right, and by this point I'm thinking maybe it would be better to just modify zdb_read_block() to take a blkptr as input, and extract all this stuff from there. And then I went on a trip, and I haven't gotten back to working on this at all...

There is an unimplemented "c" checksum check flag, which would also be super-duper useful for me in this situation, because then I can just find the individual block which is corrupt, and only need to examine just that... Of course, this is probably unimplemented, because you'll either need to pass the whole checksum to zdb on the command line... or get zdb to look it up from a user supplied blkptr.

[tuxoko]

I also notice the possible problem with ZLE when I was trying to figure out the 16MB/infinite loop issues (#4955/#4956). Though in the master, lsize will start from psize so zle is much less likely to success on pseudo random data.

[roopa01]

This might be a bug report for the illumos team, rather than zfsonlinux, but I'm sure that this report will eventually get to the right person.

I've got a couple of zpools with a single checksum error, on a single file on them, for whatever reason. And I'm trying to just dump a copy of the corrupt file out with zdb, so I can compare it with my good (from backup) copy to find out what changed. Curiously, zdb doesn't just have an option to dump out all the data contents of a file if given a znode or blkptr.

So, normally, the trick is to just look up the inode, I mean znode for the file (ls -i), and get the DVA's of the indirect blocks.
zdb -ddddd pool 123456
(And some -bbbbbb too if you want to get more detail.)
Then you write a very short perl script, or shell script to chop out all of the L0 DVA[0]=<0:23afca2bc000:134000> parts, and feed them back into zdb -R kinda like this:
zdb -R pool 0:23afca2bc000:134000:r > blocks/0_23afca2bc000_134000.bin
And then cat all the blocks back together, if you didn't just use ">>" instead.

BUT!

If you have compression turned on, these raw chunks of file will be compressed. Well, there's a "d" flag you can add, which should decompress the data, right? So...

I speculate that when you hand zdb just a raw DVA to read, it has no idea about it's logical size, or physical size, or compression algorithm, or checksum, because all of that info is in the block pointer, and it doesn't know which block pointer is pointing at this DVA. All zdb knows is you gave it an address.

So, when you do zdb -R pool 0:d919ae8c000:1c000:dr, zdb will brute-force attempt to decompress that block with every single compression algorithm known to zfs, until one of the decompression functions returns without an error... AND! also attempt the decompression from every possible starting offset in the block. (This explains why CPU load goes up to 100% for twenty minutes to dump a few MB of data.)

Once upon a time, this would probably, eventually, give you a correctly decompressed block... And then came LZ4...

Actually, the problem isn't with LZ4, it's actually with the zero-run-length-encoder. Basically, any random data you hand to ZLE, is always doing to be decompressable without error. So this brute-force, try-every-decompression-function, method will always return your block, ZLE decompressed, as soon as it tries that.... which it will try, before trying LZ4.

It goes in this order (from include/sys/zio.h):

enum zio_compress {
        ZIO_COMPRESS_INHERIT = 0,
        ZIO_COMPRESS_ON,
        ZIO_COMPRESS_OFF,
        ZIO_COMPRESS_LZJB,
        ZIO_COMPRESS_EMPTY,
        ZIO_COMPRESS_GZIP_1,
        ZIO_COMPRESS_GZIP_2,
        ZIO_COMPRESS_GZIP_3,
        ZIO_COMPRESS_GZIP_4,
        ZIO_COMPRESS_GZIP_5,
        ZIO_COMPRESS_GZIP_6,
        ZIO_COMPRESS_GZIP_7,
        ZIO_COMPRESS_GZIP_8,
        ZIO_COMPRESS_GZIP_9,
        ZIO_COMPRESS_ZLE,
        ZIO_COMPRESS_LZ4,
        ZIO_COMPRESS_FUNCTIONS
};

And the particular brute-force loop is this one from cmd/zdb/zdb.c zdb_read_block()

                enum zio_compress c;
                void *pbuf2 = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL);
                void *lbuf2 = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL);

                bcopy(pbuf, pbuf2, psize);

                VERIFY(random_get_pseudo_bytes((uint8_t *)pbuf + psize,
                    SPA_MAXBLOCKSIZE - psize) == 0);

                VERIFY(random_get_pseudo_bytes((uint8_t *)pbuf2 + psize,
                    SPA_MAXBLOCKSIZE - psize) == 0);

                for (lsize = SPA_MAXBLOCKSIZE; lsize > psize;
                    lsize -= SPA_MINBLOCKSIZE) {
                        for (c = 0; c < ZIO_COMPRESS_FUNCTIONS; c++) {
                                if (zio_decompress_data(c, pbuf, lbuf,
                                    psize, lsize) == 0 &&
                                    zio_decompress_data(c, pbuf2, lbuf2,
                                    psize, lsize) == 0 &&
                                    bcmp(lbuf, lbuf2, lsize) == 0)
                                        break;
                        }
                        if (c != ZIO_COMPRESS_FUNCTIONS)
                                break;
                        lsize -= SPA_MINBLOCKSIZE;
                }

A month ago, I hacked zdb.c to just do LZ4 first, and that mostly worked -- the output block was much more correct. But, then I also needed to pass it the psize and lsize to really get the output block right, and by this point I'm thinking maybe it would be better to just modify zdb_read_block() to take a blkptr as input, and extract all this stuff from there. And then I went on a trip, and I haven't gotten back to working on this at all...

There is an unimplemented "c" checksum check flag, which would also be super-duper useful for me in this situation, because then I can just find the individual block which is corrupt, and only need to examine just that... Of course, this is probably unimplemented, because you'll either need to pass the whole checksum to zdb on the command line... or get zdb to look it up from a user supplied blkptr.

[roopa01]

I have same issue on decompressing lz4 compressed data block. ZDB does not decompress data block gives
error " ZLE decompression was selected. If you suspect the results are wrong,
try avoiding ZLE by setting and exporting ZDB_NO_ZLE="true" ".
I have exporting this variable but issue is not resolved .How can i resolve this issue. I am working on ubuntu 20.04 .