API to get condensed tree of modified blocks between two txg #6

mynktl · 2018-03-02T07:57:56Z

Description

API to get condense tree of modified blocks between two txg
Changes in ZVOL ZAP update/read API regarding microzap

Added uzfs_txg_block_diff API, which will return condensed tree of changed blocks
between two txg.
In ZVOL ZAP update/read API, value changed to 64bit unsigned integer instead of char *.

How Has This Been Tested?

Unit test are added to test this change with uzfs_test.
To test this change :

./cmd/uzfs_test/uzfs_test -T 3 -t 10 -n 5
./cmd/uzfs_test/uzfs_test -T 4 -t 10

Types of changes

New feature (non-breaking change which adds functionality)

Checklist:

My code follows the ZFS on Linux code style requirements.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.
All commit messages are properly formatted and contain Signed-off-by.
Change has been approved by a ZFS on Linux member.

- changes in zvol zap update/read API for microzap checks Signed-off-by: mayank <[email protected]>

Signed-off-by: mayank <[email protected]>

vishnuitta · 2018-03-04T01:16:52Z

lib/libzpool/uzfs_zap.c

@@ -61,8 +73,8 @@ uzfs_update_zap_entries(void *zvol, const uzfs_zap_kv_t **array,

 	for (i = 0; i < count; i++) {
 		kv = array[i];
-		VERIFY0(zap_update(os, ZVOL_ZAP_OBJ, kv->key, 1, kv->size,
-		    kv->value, tx));
+		VERIFY0(zap_update(os, ZVOL_ZAP_OBJ, kv->key, kv->size, 1,


I understand our kv->size is 8 currently. But, its better to be (.., 1, kv->size,..) @mynktl incase if kv->size becomes greater than 8 later on. Same goes to below also.

I kept it this way to avoid fat zap upgrade.

vishnuitta · 2018-03-04T01:24:47Z

lib/libzpool/uzfs_zap.c

@@ -50,6 +51,17 @@ uzfs_update_zap_entries(void *zvol, const uzfs_zap_kv_t **array,
 	int err;
 	int i = 0;

+	/*
+	 * check if key length is greater than MZAP_NAME_LEN.
+	 * key with MZAP_NAME_LEN+ length will convert microzap


Thanks @mynktl for these changes, so that, it won't become fat zap. Comment need to be fixed at this line.

vishnuitta · 2018-03-04T06:04:16Z

lib/libzpool/uzfs_mtree.c

+};
+
+void
+add_to_mblktree(avl_tree_t *tree, uint64_t boffset, uint64_t blen)


why tree called as mblktree..? also, add comments at each logic.

it contains modified block's details, so i kept it as mblktree. I will add comments at each tree operation/logic.

add_to_txg_diff_tree

vishnuitta · 2018-03-05T10:18:31Z

lib/libzpool/uzfs_mtree.c

+	snprintf(snapname, sizeof (snapname), "%s@%s%llu", zv->zv_name,
+	    TXG_DIFF_SNAPNAME, now);
+
+	error = dsl_pool_hold(snapname, FTAG, &dp);


why don't we use dmu_objset_own rather than dsl_pool_hold & dsl_dataset_hold?

here, we are traversing snapshot only, so we can avoid dmu_objset_own. Regarding snapshot destruction while we are traversing snapshot, we can use dsl_dataset_long_hold.

vishnuitta · 2018-03-05T10:37:18Z

cmd/uzfs_test/uzfs_txg_diff.c

+} wblkinfo_t;
+
+int
+del_from_mblktree(avl_tree_t *tree, uint64_t offset, uint64_t len)


operations on tree can go into one file in lib/libzpool/

in production code, we are not using this API. So, i didn't keep it in lib/libzpool. Once we need this, we will move this to lib/libzpool.

vishnuitta

changes looks good @mynktl .. but, we need to change data structures and functions names. Also, some more unit testing needed to see whether the traversal is returning correct values.

vishnuitta · 2018-03-06T09:05:33Z

lib/libzpool/uzfs_mtree.c

+
+
+int
+uzfs_txg_block_diff(zvol_state_t *zv, uint64_t start_txg, uint64_t end_txg,


uzfs_get_txg_diff_tree

vishnuitta · 2018-03-06T09:11:38Z

lib/libzpool/uzfs_mtree.c

+
+#define	TXG_DIFF_SNAPNAME	"tsnap"
+
+struct diff_txg_blk {


uzfs_txg_diff_cb_args_t

vishnuitta · 2018-03-06T09:13:43Z

lib/libzpool/uzfs_mtree.c

+}
+
+int
+uzfs_changed_block_cb(spa_t *spa, zilog_t *zillog, const blkptr_t *bp,


uzfs_txg_diff_cb

vishnuitta · 2018-03-06T09:16:35Z

lib/libzpool/uzfs_mtree.c

+#define	TXG_DIFF_SNAPNAME	"tsnap"
+
+struct diff_txg_blk {
+	avl_tree_t *tree;


uzfs_txg_diff_tree

vishnuitta · 2018-03-06T09:16:52Z

lib/libzpool/uzfs_mtree.c

+}
+
+static int
+zvol_blk_off_cmpr(const void *arg1, const void *arg2)


uzfs_txg_diff_tree_compare

vishnuitta · 2018-03-06T09:23:22Z

cmd/uzfs_test/uzfs_txg_diff.c

+}
+
+void
+uzfs_zvol_txg_diff_blk_test(void *arg)


uzfs_txg_diff_test

Signed-off-by: mayank <[email protected]>

mynktl · 2018-03-06T19:31:53Z

@vishnuitta I have updated PR with review comments.
I have made following changes:

changed functions and data structures name.
changes in functionality test for txg_diff_tree test, increased io_count from 5 to 5-100.
changes in zap_update API regarding zap_entry's value check which was missing earlier.

vishnuitta · 2018-03-07T08:53:58Z

cmd/uzfs_test/uzfs_txg_diff.c

 			io_num++;
 			blk_offset = uzfs_random(vol_blocks - 16);
+			/*
+			 * make sure offset is aligned to block size
+			 */
 			offset = ((blk_offset * blksz + block_size) /


offset can be just blk_offset * block_size

also, have a test case with less active size (probably for 30K or 50K blocks), so that, collisions/merges of tree can be verified

vishnuitta · 2018-03-07T08:54:15Z

cmd/uzfs_test/uzfs_txg_diff.c

+					offset_matched = B_TRUE;
+					break;
+				}
+				if (temp_blk->offset + temp_blk->len ==


this condition is not required at all

vishnuitta · 2018-03-07T08:55:20Z

cmd/uzfs_test/uzfs_txg_diff.c

@@ -166,16 +186,37 @@ uzfs_zvol_txg_diff_blk_test(void *arg)

 	while (i++ < test_iterations) {
 		count = 0;
+		max_io = MAX(uzfs_random(100), 5);


make it as 10K..

vishnuitta

changes looks good Mayank. Just take care of those minor comments.

modified blocks between two txg - fix for active_size/vol_size argument parsing in uzfs_test Signed-off-by: mayank <[email protected]>

[CCS-68] travis integration with ZoL

- API to get condensed tree of modified blocks between two txg Signed-off-by: mayank <[email protected]>

- API to get condensed tree of modified blocks between two txg

1b6fd5b

- changes in zvol zap update/read API for microzap checks Signed-off-by: mayank <[email protected]>

mynktl force-pushed the cstor49 branch from 2fa8ab6 to 1b6fd5b Compare March 2, 2018 07:59

travis build fix

48cdaaf

Signed-off-by: mayank <[email protected]>

mynktl changed the title ~~- API to get condensed tree of modified blocks between two txg~~ API to get condensed tree of modified blocks between two txg Mar 2, 2018

jkryl approved these changes Mar 2, 2018

View reviewed changes

vishnuitta reviewed Mar 4, 2018

View reviewed changes

vishnuitta reviewed Mar 5, 2018

View reviewed changes

vishnuitta suggested changes Mar 5, 2018

View reviewed changes

vishnuitta reviewed Mar 6, 2018

View reviewed changes

lib/libzpool/uzfs_mtree.c Outdated

#define TXG_DIFF_SNAPNAME "tsnap"

struct diff_txg_blk {

Copy link

vishnuitta Mar 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uzfs_txg_diff_cb_args_t

vishnuitta reviewed Mar 6, 2018

View reviewed changes

lib/libzpool/uzfs_mtree.c Outdated

#define TXG_DIFF_SNAPNAME "tsnap"

struct diff_txg_blk {

avl_tree_t *tree;

Copy link

vishnuitta Mar 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uzfs_txg_diff_tree

vishnuitta reviewed Mar 6, 2018

View reviewed changes

lib/libzpool/uzfs_mtree.c Outdated

}

static int

zvol_blk_off_cmpr(const void *arg1, const void *arg2)

Copy link

vishnuitta Mar 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uzfs_txg_diff_tree_compare

vishnuitta reviewed Mar 6, 2018

View reviewed changes

cmd/uzfs_test/uzfs_txg_diff.c Outdated

}

void

uzfs_zvol_txg_diff_blk_test(void *arg)

Copy link

vishnuitta Mar 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uzfs_txg_diff_test

mayank added 2 commits March 6, 2018 22:28

changes according to review comments

1cc6ce4

Signed-off-by: mayank <[email protected]>

resolving confilicts from zfs-0.7-release

1e67fc0

Signed-off-by: mayank <[email protected]>

vishnuitta reviewed Mar 7, 2018

View reviewed changes

vishnuitta approved these changes Mar 7, 2018

View reviewed changes

mayank and others added 2 commits March 7, 2018 16:43

- removing fix io count from functionality test of

8b9cbad

modified blocks between two txg - fix for active_size/vol_size argument parsing in uzfs_test Signed-off-by: mayank <[email protected]>

Merge branch 'zfs-0.7-release' into cstor49

fb2fec8

vishnuitta merged commit 399d4f2 into mayadata-io:zfs-0.7-release Mar 7, 2018

jkryl pushed a commit that referenced this pull request Jun 26, 2018

[CCS-68] Travis integration with ZoL (#6)

83e551d

[CCS-68] travis integration with ZoL

jkryl pushed a commit that referenced this pull request Jun 26, 2018

API to get condensed tree of modified blocks between two txg (#6)

6076c64

- API to get condensed tree of modified blocks between two txg Signed-off-by: mayank <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API to get condensed tree of modified blocks between two txg #6

API to get condensed tree of modified blocks between two txg #6

mynktl commented Mar 2, 2018

vishnuitta Mar 4, 2018

mynktl Mar 6, 2018

vishnuitta Mar 4, 2018

mynktl Mar 5, 2018

vishnuitta Mar 4, 2018

mynktl Mar 5, 2018

vishnuitta Mar 6, 2018

vishnuitta Mar 5, 2018

mynktl Mar 6, 2018

vishnuitta Mar 5, 2018

mynktl Mar 5, 2018

vishnuitta left a comment

vishnuitta Mar 6, 2018 •

edited

Loading

vishnuitta Mar 6, 2018

vishnuitta Mar 6, 2018 •

edited

Loading

vishnuitta Mar 6, 2018

vishnuitta Mar 6, 2018

vishnuitta Mar 6, 2018

mynktl commented Mar 6, 2018

vishnuitta Mar 7, 2018

vishnuitta Mar 7, 2018

vishnuitta Mar 7, 2018

vishnuitta Mar 7, 2018

vishnuitta left a comment



		int
		uzfs_txg_block_diff(zvol_state_t *zv, uint64_t start_txg, uint64_t end_txg,

API to get condensed tree of modified blocks between two txg #6

API to get condensed tree of modified blocks between two txg #6

Conversation

mynktl commented Mar 2, 2018

Description

How Has This Been Tested?

Types of changes

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vishnuitta left a comment

Choose a reason for hiding this comment

vishnuitta Mar 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vishnuitta Mar 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mynktl commented Mar 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vishnuitta left a comment

Choose a reason for hiding this comment

vishnuitta Mar 6, 2018 •

edited

Loading

vishnuitta Mar 6, 2018 •

edited

Loading