Split snapshot controller using beta APIs #182

xing-yang · 2019-10-21T12:39:16Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
This PR splits the snapshot controller into two, a common controller to be deployed on the controller node, and a sidecar controller to be deployed together with the CSI driver.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
This PR depends on the snapshot API beta changes.

Does this PR introduce a user-facing change?:

Split the snapshot controller into two controllers, a common snapshot controller and a sidecar snapshot controller. Only the sidecar controller should be deployed with the CSI driver.

yuxiangqian

some initial thoughts.

yuxiangqian · 2019-10-23T21:00:42Z

pkg/common_controller/snapshot_controller.go

+// points to a specific VolumeSnapshotContent and the VolumeSnapshotContent also
+// points back for this VolumeSnapshot.
+//
+// The dynamic snapshot creation is multi-step process: first controller triggers


this comment no longer applies.

yuxiangqian · 2019-10-23T21:04:23Z

pkg/common_controller/snapshot_controller.go

+	klog.V(5).Infof("syncContent: check if we should remove Finalizer for VolumeSnapshotContent[%s]", content.Name)
+	// It is a deletion candidate if DeletionTimestamp is not nil and
+	// VolumeSnapshotContentFinalizer is set.
+	if utils.IsContentDeletionCandidate(content) {


the deletion workflow needs to be revisited a bit here.
after split, common controller will no longer be removing finalizer on volumesnapshotcontent, rather the sidecar should be doing it.

yuxiangqian · 2019-10-23T21:19:13Z

pkg/common_controller/snapshot_controller.go

+		}
+	}
+
+	if utils.NeedToAddContentFinalizer(content) {


should the finalizer only be added after binding?

I still think this should be added after binding, i.e., after line 97?
unless the sidecar will act as following:
if deleteTimestamp set && uid == "", exec deletion policy, and then remove finalizer

Most of the finalizers here are following the in-tree pv/pvc finalizers where they are added very early, but let's revisit this later.

yuxiangqian · 2019-10-23T21:21:34Z

pkg/common_controller/snapshot_controller.go

+		snapshot = nil
+	}
+
+	if snapshot == nil {


this if clause is no longer needed as deletion will be handled by sidecar controller

yuxiangqian · 2019-10-24T20:36:57Z

pkg/common_controller/snapshot_controller.go

+		content := obj.(*crdv1.VolumeSnapshotContent)
+		if content.Spec.VolumeSnapshotRef.Name == snapshot.Name &&
+			content.Spec.VolumeSnapshotRef.Namespace == snapshot.Namespace &&
+			content.Spec.VolumeSnapshotRef.UID == snapshot.UID &&


It might be missing one case here for pre-existing snapshot binding. UID of content.Spec.VolumeSnapshotRef will be empty, it should also return the content as a match.

nvm, findContentfromStore should handle, these two functions needs a bit renaming in future version

yuxiangqian · 2019-10-24T22:16:59Z

pkg/common_controller/snapshot_controller.go

+	}
+
+	// Set AnnBoundByController if it is not set yet
+	if !metav1.HasAnnotation(contentClone.ObjectMeta, utils.AnnBoundByController) {


Is this annotation still needed in the new workflow.

yuxiangqian · 2019-10-24T22:19:47Z

pkg/common_controller/snapshot_controller.go

+		snapshotCopy.Status.BoundVolumeSnapshotContentName = &snapshotContent.Name
+
+		// Set AnnBoundByController if it is not set yet
+		if !metav1.HasAnnotation(snapshotCopy.ObjectMeta, utils.AnnBoundByController) {


are these annotations still needed?

yuxiangqian · 2019-10-24T22:22:18Z

pkg/common_controller/snapshot_controller.go

+	}
+
+	// TODO(xyang): update snapshot status
+	if snapshotContent.Status != nil {


as you pointed out before, snapshot needs to be re-read from API server for the status update to be successful.
and since there is a retry loop, this line should be removed.

I think the BoundVolumeSnapshotContentName on VolumeSnapshot.Status needs to be updated regardless whether snapshotContent.Status == nil or not.
this function is effectively a call to bind and update.

Sure

yuxiangqian · 2019-10-24T22:23:08Z

pkg/common_controller/snapshot_controller.go

+
+		// TODO(xyang): update snapshot status
+		if newContent.Status != nil {
+			_, err = ctrl.updateSnapshotStatus(snapshot, newContent)


these is a retry loop, this line is a duplication?

yuxiangqian · 2019-10-25T00:23:45Z

pkg/common_controller/snapshot_controller.go

+		updated = true
+	} else {
+		newStatus = snapshotObj.Status.DeepCopy()
+		if newStatus.BoundVolumeSnapshotContentName == nil {


It might worth checking whether if BoundVolumeSnapshotContentName == boundContentName to decide whether or not to continue here? That case it should be invalid?

yuxiangqian

still in process, first pass on common controller

yuxiangqian · 2019-10-29T18:31:46Z

pkg/common_controller/snapshot_controller.go

 	obj, found, err := ctrl.snapshotStore.GetByKey(snapshotName)
 	if err != nil {
 		return err
 	}
 	if !found {
-		klog.V(4).Infof("synchronizing VolumeSnapshotContent[%s]: snapshot %s not found", content.Name, snapshotRefKey(content.Spec.VolumeSnapshotRef))
+		klog.V(4).Infof("synchronizing VolumeSnapshotContent[%s]: snapshot %s not found", content.Name, utils.SnapshotRefKey(&content.Spec.VolumeSnapshotRef))


maybe reuse "snapshotName"? Ditto everywhere

yuxiangqian · 2019-10-29T18:36:01Z

pkg/common_controller/snapshot_controller.go

 		// Treat the content as bound to a missing snapshot.
 		snapshot = nil
+	} else {


not sure whether this else block is needed.
if syncContent is not doing binding, i.e., in pre-existing VSC case, does it make sense to do the status update here? syncSnapshot will be doing it anyways?

I tested create snapshot several times and observed that the status update in snapshot happens a lot later than status update in the content without this block of code here. I had to keep typing "kubectl describe volumesnapshot" to check the snapshot status and wait for it to be in sync with content status (this is much slower than before the controller split).

So I added updateSnapshotStatus here, and observed that snapshot status update happens as soon as the content status update.

I don't know if there is a better way to trigger snapshot status update to happen as soon as content status update.

Maybe I can add this line "ctrl.snapshotQueue.Add(snapshotName)" to trigger the snapshot status update instead of adding this block here. Will give that a try.

yuxiangqian · 2019-10-29T18:39:03Z

pkg/common_controller/snapshot_controller.go

+	// sidecar controller.
+	// Snapshot won't be deleted until content is deleted
+	// due to the finalizer
+	if snapshot == nil || utils.IsSnapshotDeletionCandidate(snapshot) {


There is one case missed here to handle the following steps:

create a VSC (pre-existing snapshot, uid == "")

delete the VSC
in this case, delete VSC will not be handled as it returns in previous checks.

I was wondering how to handle that as well as we don't have a way to validate anything about snapshot in the sidecar controller any more.

This means if admin has made a mistake about the VSC he/she created, he/she can't delete it without first binding it to a snapshot.

We probably should just check the deletion timestamp and deletion policy in syncContent in the sidecar controller and delete the content if both conditions meet? Otherwise admin could be stuck with a VSC that he/she wants to delete. It is likely that an admin could make mistakes when creating a VSC and wants to delete it and create a new one.

Maybe this "ShouldDelete" annotation should only apply to dynamic case? For static case, we just check deletion timestamp and deletion policy?

If timestamp set in content:

If snapshot uid="", sidecar syncContent deletes it because no finalizer.

If snapshot uid is set, sidecar syncContent checks "ShouldDelete" annotation and then delete.

If we only add finalizer for "bound" snapshots which means that having finalizer == "bound", how about this

If DeletionTimestamp set in content:

if no finalizer, do nothing (actually this case should not exist, because VSC will be deleted immediately if no finalizer

has a finalizer: if annotation (How about name it as "BoundVolumeSnapshotIsDeleted") exists, depending on DeletionPolicy, delete physical snapshot (or not), remove finalizer.

Discussed with Jing again about this. We can let the common controller remove snapshot uid when it is going to delete the content but don't add the ShouldDelete annotation.
If deletion timestamp is set in the content:

if snapshot uid="", sidecar syncContent invokes the CSI driver to delete the physical snapshot.

if snapshot uid is set, don't delete.

yuxiangqian · 2019-10-29T18:39:33Z

pkg/common_controller/snapshot_controller.go

+	// Snapshot won't be deleted until content is deleted
+	// due to the finalizer
+	if snapshot == nil || utils.IsSnapshotDeletionCandidate(snapshot) {
+		if content.Spec.DeletionPolicy != crdv1.VolumeSnapshotContentDelete {


in this case, should the finalizer on VolumeSnapshot be removed?

Changed the logic to only add snapshot bound finalizer if snapshot is bound to a content and deletion policy is delete.

yuxiangqian · 2019-10-29T18:40:42Z

pkg/common_controller/snapshot_controller.go


-	if isSnapshotDeletionCandidate(snapshot) {
+	// Check is snapshot deletionTimestamp is set and any finalizer is on
+	if utils.IsSnapshotDeletionCandidate(snapshot) {
 		// Volume snapshot should be deleted. Check if it's used
 		// and remove finalizer if it's not.
 		// Check if a volume is being created from snapshot.
 		isUsed := ctrl.isVolumeBeingCreatedFromSnapshot(snapshot)


rather inUse?

yuxiangqian · 2019-10-29T23:33:34Z

pkg/common_controller/snapshot_controller.go

+// and removed it if needed.
+func (ctrl *csiSnapshotCommonController) checkandRemoveSnapshotSourceFinalizer(snapshot *crdv1.VolumeSnapshot) error {
+	// Get snapshot source which is a PVC
+	pvc, err := ctrl.getClaimFromVolumeSnapshot(snapshot)


per above, anyways the PVC has already been found out.

yuxiangqian · 2019-10-29T23:37:10Z

pkg/common_controller/snapshot_controller.go

-		klog.Error(strerr)
-		ctrl.eventRecorder.Event(newSnapshot, v1.EventTypeWarning, "CreateSnapshotContentFailed", strerr)
-		return nil, newControllerUpdateError(snapshotKey(snapshot), err.Error())
+		updateSnapshot, err := ctrl.clientset.SnapshotV1beta1().VolumeSnapshots(snapshot.Namespace).Update(snapshotCopy)


this will not update the status of the snapshot? BoundVolumeSnapshotContentName needs to be updated by invoke UpdateStatus()
since Status will be updated later, this Update call mostly will not do anything. Is this still needed?

That's right. I think we don't need this any more.

yuxiangqian · 2019-10-29T23:48:29Z

pkg/common_controller/snapshot_controller.go

-	updated := false
+// UpdateSnapshotStatus updates snapshot status based on content status
+func (ctrl *csiSnapshotCommonController) updateSnapshotStatus(snapshot *crdv1.VolumeSnapshot, content *crdv1.VolumeSnapshotContent) (*crdv1.VolumeSnapshot, error) {
+	klog.V(5).Infof("updateSnapshotStatus[%s]", utils.SnapshotKey(snapshot))
 	if content.Status == nil {


should BoundContentName be updated even if the status of content is nil?

That's my question too.

yuxiangqian · 2019-10-29T23:53:56Z

pkg/common_controller/snapshot_controller.go

+	}
+	if removeBoundFinalizer {
+		snapshotClone.ObjectMeta.Finalizers = slice.RemoveString(snapshotClone.ObjectMeta.Finalizers, utils.VolumeSnapshotBoundFinalizer, nil)
+	}
 	_, err := ctrl.clientset.SnapshotV1beta1().VolumeSnapshots(snapshotClone.Namespace).Update(snapshotClone)


Update returns the latest copy of snapshotclone on success, rather do snapshotClone, err = ......

yuxiangqian · 2019-10-29T23:55:02Z

pkg/common_controller/snapshot_controller.go

-	if err != nil {
-		return newControllerUpdateError(pvcClone.Name, err.Error())
+func (ctrl *csiSnapshotCommonController) contentExists(snapshot *crdv1.VolumeSnapshot) (bool, error) {
+	if snapshot.Status.BoundVolumeSnapshotContentName == nil {


is snapshot.Status == nil checking needed here?

yuxiangqian

on going

yuxiangqian · 2019-10-30T20:51:09Z

pkg/sidecar_controller/snapshot_controller_base.go

+	// Map of scheduled/running operations.
+	runningOperations goroutinemap.GoRoutineMap
+
+	createSnapshotContentRetryCount int


the sidecar will not be creating SnapshotContent, there two variable names seems to be not appropriate

Actually these are not needed any more. Removed.

yuxiangqian · 2019-10-30T20:53:43Z

pkg/sidecar_controller/snapshot_controller_base.go

+
+// verify whether the driver specified in VolumeSnapshotContent matches the controller's driver name
+func (ctrl *csiSnapshotSideCarController) isDriverMatch(content *crdv1.VolumeSnapshotContent) bool {
+	if content.Spec.Source.VolumeHandle == nil && content.Spec.Source.SnapshotHandle == nil {


It's ok to have it check here, a webhook will be added to enforce either one of those fields should have been specified when creating in API server

yuxiangqian · 2019-10-30T20:55:53Z

pkg/sidecar_controller/snapshot_controller_base.go

+		return false
+	}
+	snapshotClassName := content.Spec.VolumeSnapshotClassName
+	if snapshotClassName != nil {


what about default class?
In cases where a snapshot class name is not specified in content, the default snapshot class (if exists) should be used to verify against the sidecar snapshotterName instead of return true simply.

The check for default class is in the common controller.

@xing-yang that case only covers the dynamic case, however does not cover the pre-existing case, and even for dynamic case,
sidecar controller should not assume the existence of the driver name? Rather I think it should refer to the default snapshotclass
if the field is not provided.
We should have the sidecar controller verifying it as well.

@yuxiangqian sorry, I don't know how to reply to this, so just directly adding it here. Finding default snapshot class is the job of the common controller. I don't think we should ask the sidecar to repeat that.

yuxiangqian · 2019-10-30T20:59:01Z

pkg/sidecar_controller/snapshot_controller_base.go

+	_ = ctrl.contentStore.Delete(content)
+	klog.V(4).Infof("content %q deleted", content.Name)
+
+	snapshotName := utils.SnapshotRefKey(&content.Spec.VolumeSnapshotRef)


this check is no longer needed?
I am also not sure whether the content should be removed from contentStore at this point.
A VSC will only be deleted after DeletionPolicy has been carried. Maybe it still worth keeping in the contentStore?
the following comments do not apply as well

We are done with the content. It is already deleted from API, so should be removed from cache store.

yuxiangqian · 2019-10-30T20:59:16Z

pkg/sidecar_controller/snapshot_controller_base.go

+		klog.V(5).Infof("deleteContent[%q]: content not bound", content.Name)
+		return
+	}
+	// sync the snapshot when its content is deleted.  Explicitly sync'ing the


comments no longer apply

yuxiangqian · 2019-10-30T21:27:36Z

pkg/sidecar_controller/snapshot_controller.go

+	var snapshotID string
+
+	if content.Spec.Source.VolumeHandle == nil && content.Spec.Source.SnapshotHandle != nil {
+		klog.V(5).Infof("checkandUpdateBoundSnapshotStatusOperation: checking whether snapshot is pre-bound to content [%s]", content.Name)


log does not match

yuxiangqian · 2019-10-30T21:27:56Z

pkg/sidecar_controller/snapshot_controller.go

+	var driverName string
+	var snapshotID string
+
+	if content.Spec.Source.VolumeHandle == nil && content.Spec.Source.SnapshotHandle != nil {


first check is not necessary

Why? We need to differentiate between static vs dynamic case.

this check means as long as VolumeHandle exists, its a dynamic snapshot.
will checking: content.Spec.Source.SnapshotHandle != nil be just sufficient?
those two fields should be enforced by API server later. I think

yuxiangqian · 2019-10-30T21:29:46Z

pkg/sidecar_controller/snapshot_controller.go

+		if content.Spec.Source.SnapshotHandle != nil {
+			snapshotID = *content.Spec.Source.SnapshotHandle
+		}
+	} else if content.Spec.Source.VolumeHandle != nil {


If content.Status.SnapshotHandle already exists, we should just call ListSnapshots?

ListSnapshots is optional. We discussed that we always want to call CreateSnapshot to find out snapshot status if possible.

yuxiangqian · 2019-10-30T21:31:42Z

pkg/sidecar_controller/snapshot_controller.go

+	// content.Status will be created for the first time after a snapshot
+	// is created by the CSI driver. If content.Status is not nil,
+	// we should update content status without creating snapshot again.
+	if content.Status != nil && content.Status.Error != nil && content.Status.Error.Message != nil && !isControllerUpdateFailError(content.Status.Error) {


minor thing, I'd move all nil checks into isControllerUpdateFailError

yuxiangqian · 2019-10-30T21:33:32Z

pkg/sidecar_controller/snapshot_controller.go

+}
+
+// updateSnapshotContentSize update the restore size for snapshot content
+func (ctrl *csiSnapshotSideCarController) updateSnapshotContentSize(content *crdv1.VolumeSnapshotContent, size int64) error {


jingxu97 · 2019-10-31T18:44:54Z

pkg/common_controller/snapshot_controller.go

+		// If content exists, set DeletionTimeStamp on the content;
+		// content won't be deleted immediately due to the finalizer
+		if content != nil && deleteContent {
+			err := ctrl.clientset.SnapshotV1beta1().VolumeSnapshotContents().Delete(*snapshot.Status.BoundVolumeSnapshotContentName, &metav1.DeleteOptions{})


what if *snapshot.Status is nil (status is lost case?),

Actually I don't need to rely on snapshot.Status here. We have content so I can use content.Name directly.

saad-ali

/approve

Will let @yuxiangqian do final review and lgtm

saad-ali · 2019-10-31T21:52:54Z

pkg/sidecar_controller/snapshot_controller_base.go

+		runningOperations: goroutinemap.NewGoRoutineMap(true),
+		resyncPeriod:      resyncPeriod,
+		contentStore:      cache.NewStore(cache.DeletionHandlingMetaNamespaceKeyFunc),
+		contentQueue:      workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "csi-snapshotter-content"),


Let's make sure there are finalizers in case the sidecar crashes and the content object is deleted

sure. Adding content finalizer is handled by the common controller and the removal is handled by the sidecar right now.

saad-ali · 2019-10-31T22:18:17Z

pkg/sidecar_controller/snapshot_controller.go

+func (ctrl *csiSnapshotSideCarController) syncContent(content *crdv1.VolumeSnapshotContent) error {
+	klog.V(5).Infof("synchronizing VolumeSnapshotContent[%s]", content.Name)
+
+	if metav1.HasAnnotation(content.ObjectMeta, utils.AnnShouldDelete) {


For GA, this annotation should become a first class field probably in VolumeSnapshotContent.Status.

saad-ali · 2019-11-03T03:51:02Z

cmd/csi-snapshotter-common/Dockerfile

@@ -0,0 +1,6 @@
+FROM gcr.io/distroless/static:latest


Eventually we should consider putting csi-snapshotter-common in to its own repo.

saad-ali · 2019-11-03T03:54:58Z

deploy/kubernetes/common-snapshot-controller.yaml

+# This YAML file shows how to deploy the common snapshot controller
+
+---
+kind: StatefulSet


We used to do StatefulSet because there was no leader election. Is this still the best choice? CC @misterikkit thoughts?

saad-ali · 2019-11-03T03:59:35Z

pkg/utils/util_test.go


 import (
 	crdv1 "github.com/kubernetes-csi/external-snapshotter/pkg/apis/volumesnapshot/v1beta1"
 	"k8s.io/api/core/v1"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	//metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"


Remove if not needed.

pkg/utils/util.go

k8s-ci-robot · 2019-11-03T04:05:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: saad-ali, xing-yang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [saad-ali,xing-yang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

yuxiangqian · 2019-11-04T22:38:14Z

pkg/common_controller/snapshot_controller.go

+		}
+	}
+
+	if utils.NeedToAddContentFinalizer(content) {


I still think this should be added after binding, i.e., after line 97?
unless the sidecar will act as following:
if deleteTimestamp set && uid == "", exec deletion policy, and then remove finalizer

Most of the finalizers here are following the in-tree pv/pvc finalizers where they are added very early, but let's revisit this later.

yuxiangqian · 2019-11-04T22:58:34Z

pkg/common_controller/snapshot_controller.go

+
+	var err error
+	// Check is snapshot deletionTimestamp is set and any finalizer is on
+	if utils.IsSnapshotDeletionCandidate(snapshot) {


it is a bit confusing to me when to call this function, it's name does not match well with the logic, maybe move the utils.IsSnapshotDeletionCandidate to caller to check?

could you shed some light on why existences of "VolumeSnapshotAsSourceFinalizer" OR "VolumeSnapshotBoundFinalizer"
finalizers are checked in this call "IsSnapshotDeletionCandidate"? is not deletion timestamp itself enough?
I am thinking about cases when there newly created snapshto was never bound to a content (i.e., content failed to be created), how would it be deleted?

This follows the in-tree pv/pvc finalizers pattern:
https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/protectionutil/utils.go#L25

one thing I can think of is to save the API call if there is no finalizer added, we will revisit this later.

yuxiangqian · 2019-11-04T22:59:48Z

pkg/common_controller/snapshot_controller.go

+		klog.V(5).Infof("syncSnapshot[%s]: set DeletionTimeStamp on content.", utils.SnapshotKey(snapshot))
+		// If content exists, set DeletionTimeStamp on the content;
+		// content won't be deleted immediately due to the finalizer
+		if content != nil && deleteContent {


what if inUse = true?
set "DeletionTimestamp" in API server would trigger sidecar to execute deletion policy and cause data lost if the system is trying to create a volume from the snapshot?

Added the check.

yuxiangqian · 2019-11-04T23:13:29Z

pkg/common_controller/snapshot_controller.go

+// It checks if contents exists, it checks if snapshot has bi-directional binding, it checks if
+// finalizers should be added or removed, and it checks if content should be deleted and deletes it
+// if needed.
+func (ctrl *csiSnapshotCommonController) processFinalizersandDeleteContent(snapshot *crdv1.VolumeSnapshot) error {


maybe beneficial enough to break this function into two based on whether a snapshot is a deletion candidate or not.
mixing adding/deleting into one single function is kinda of confusing.

yuxiangqian · 2019-11-05T00:35:53Z

pkg/common_controller/snapshot_controller.go

+func (ctrl *csiSnapshotCommonController) checkandAddSnapshotFinalizers(snapshot *crdv1.VolumeSnapshot, snapshotBound bool, deleteContent bool) {
+	addSourceFinalizer := false
+	addBoundFinalizer := false
+	if utils.NeedToAddSnapshotAsSourceFinalizer(snapshot) {


Should not this finalizer being added only when a VolumeSnapshot object is being used as source to restore a volume?
It seems to me it will be always added in this routine regardless whether it's under using or not.

This follows the in-tree pv/pvc finalizers pattern. Even though it will be added early, it will be deleted when necessary.

yuxiangqian · 2019-11-05T00:45:47Z

pkg/common_controller/snapshot_controller.go

+				// we delete content before the snapshot
+				klog.V(5).Infof("Content for snapshot %s not found. It may be already deleted as expected.", uniqueSnapshotName)
+			} else {
+				// TODO(xyang): may not need this check any more


in this case, we should be updating snapshot.status based on content.status?

Whenever there is an update on the content status, we will be triggering a snapshot status update. So it should not need it here.

yuxiangqian · 2019-11-05T00:53:24Z

pkg/common_controller/snapshot_controller.go

-	} else {
-		klog.Errorf("failed to getCreateSnapshotInput %s without a snapshot class", snapshot.Name)
-		return nil, nil, "", nil, fmt.Errorf("failed to take snapshot %s without a snapshot class", snapshot.Name)
+		klog.Infof("Added protection finalizer to persistent volume claim %s", pvc.Name)


the else piece seems to be gone, nvm.

yuxiangqian · 2019-11-05T00:56:10Z

pkg/common_controller/snapshot_controller.go

+	}
+	for _, snap := range snapshots {
+		// Skip static bound snapshot without a PVC source
+		if snap.Spec.Source.PersistentVolumeClaimName == nil && snap.Spec.Source.VolumeSnapshotContentName == nil {


should be snap.Spec.Source.VolumeSnapshtoContentName != nil

yuxiangqian · 2019-11-05T00:56:48Z

pkg/common_controller/snapshot_controller.go

+}
+
+// isPVCBeingUsed checks if a PVC is being used as a source to create a snapshot
+func (ctrl *csiSnapshotCommonController) isPVCBeingUsed(snapshot *crdv1.VolumeSnapshot) bool {


lets take in a PersistentVolumeClaim object as input, anyways the pvc has been found out before this call

yuxiangqian · 2019-11-05T00:57:11Z

pkg/common_controller/snapshot_controller.go

+}
+
+// removePVCFinalizer removes a Finalizer for VolumeSnapshot Source PVC.
+func (ctrl *csiSnapshotCommonController) removePVCFinalizer(snapshot *crdv1.VolumeSnapshot) error {


lets take in a PersistentVolumeClaim object as input, anyways the pvc has been found out before this call

yuxiangqian · 2019-11-05T19:52:04Z

pkg/common_controller/snapshot_controller.go

+
+	var err error
+	// Check is snapshot deletionTimestamp is set and any finalizer is on
+	if utils.IsSnapshotDeletionCandidate(snapshot) {


one thing I can think of is to save the API call if there is no finalizer added, we will revisit this later.

yuxiangqian · 2019-11-05T21:00:24Z

pkg/common_controller/snapshot_controller.go

-	} else { // snapshot.Source.Spec.VolumeSnapshotContentName == nil
-		// find a matching volume snapshot content
+	} else { // snapshot.Spec.Source.VolumeSnapshotContentName == nil - dynamically creating snapshot
+		klog.V(5).Infof("before getMatchSnapshotContent for snapshot %s", uniqueSnapshotName)
 		if contentObj := ctrl.getMatchSnapshotContent(snapshot); contentObj != nil {


error out should be good, I think an error event/message should be populated in VolumeSnapshot.Status as well.

Added

yuxiangqian · 2019-11-05T21:15:59Z

pkg/common_controller/snapshot_controller.go

+
+	// If PVC is not being deleted and finalizer is not added yet, a finalizer should be added to PVC until snapshot is created
+	klog.V(5).Infof("createSnapshotContent: Check if PVC is not being deleted and add Finalizer for source of snapshot [%s] if needed", snapshot.Name)
+	err := ctrl.ensurePVCFinalizer(snapshot)


"ensurePVCFinalizer" should return error when "pvc.ObjectMeta.DeletionTimestamp != nil" -> the PVC/PV are getting deleted from K8s
so that snapshot will fail.

yuxiangqian · 2019-11-05T21:20:15Z

pkg/common_controller/snapshot_controller.go

+		content := obj.(*crdv1.VolumeSnapshotContent)
+		if content.Spec.VolumeSnapshotRef.Name == snapshot.Name &&
+			content.Spec.VolumeSnapshotRef.Namespace == snapshot.Namespace &&
+			content.Spec.VolumeSnapshotRef.UID == snapshot.UID &&


nvm, findContentfromStore should handle, these two functions needs a bit renaming in future version

yuxiangqian · 2019-11-05T21:21:40Z

pkg/common_controller/snapshot_controller.go

+		return false
+	}
+	for _, pvc := range pvcList {
+		if pvc.Spec.DataSource != nil && len(pvc.Spec.DataSource.Name) > 0 && pvc.Spec.DataSource.Name == snapshot.Name {


please remove unnecessary len check

Done

yuxiangqian · 2019-11-05T21:26:20Z

pkg/common_controller/snapshot_controller.go

+	}
+
+	// TODO(xyang): update snapshot status
+	if snapshotContent.Status != nil {


I think the BoundVolumeSnapshotContentName on VolumeSnapshot.Status needs to be updated regardless whether snapshotContent.Status == nil or not.
this function is effectively a call to bind and update.

Sure

yuxiangqian · 2019-11-05T21:34:08Z

pkg/common_controller/snapshot_controller.go

+
+// SetDefaultSnapshotClass is a helper function to figure out the default snapshot class from
+// PVC/PV StorageClass and update VolumeSnapshot with this snapshot class name.
+func (ctrl *csiSnapshotCommonController) SetDefaultSnapshotClass(snapshot *crdv1.VolumeSnapshot) (*crdv1.VolumeSnapshotClass, *crdv1.VolumeSnapshot, error) {


this function does not work for a pre-existing snapshot which does not have StorageClass supplied.
its called at:

external-snapshotter/pkg/common_controller/snapshot_controller_base.go

Line 213 in 4954e80

newSnapshot, err := ctrl.checkAndUpdateSnapshotClass(snapshot)

For pre-existing snapshot, it does not have PVC, thus there will not be storage class

Changed it. Now we don't return error if default snapshot class not found for pre-provisioned snapshots.

yuxiangqian · 2019-11-05T22:51:42Z

pkg/common_controller/snapshot_controller.go

+}
+
+func (ctrl *csiSnapshotCommonController) contentExists(snapshot *crdv1.VolumeSnapshot) (*crdv1.VolumeSnapshotContent, error) {
+	if snapshot.Status == nil || snapshot.Status.BoundVolumeSnapshotContentName == nil {


@jingxu97 I think Jing posted a concern w.r.t the cases where status is lost. only refer to the Status.BoundVolumeSnapshotContentName seems to be not sufficient.
my rough thoughts to get conentName is:
if snapshot.Status == nil || snapshot.Status.BoundVolumeSnapshotContentName == nil --> use BoundVolumeSnapshotContentName
else if snapshot.Spec.VolumeSnapshotContentName != nil -> use VolumeSnapshotContentName --> this is to handle pre-existing case where binding did not happen?
else for dynamic creation case (to avoid situations like controller crashes before VS.Status has been updated), use the generated name.

yuxiangqian · 2019-11-05T22:54:28Z

pkg/common_controller/snapshot_controller.go

+// verifySnapshotClass gets the VolumeSnapshotClass from VolumeSnapshot.
+// It also detects if snapshotter in the VolumeSnapshotClass is the same as
+// the snapshotter in external controller.
+func (ctrl *csiSnapshotCommonController) verifySnapshotClass(snapshot *crdv1.VolumeSnapshot) (*crdv1.VolumeSnapshot, error) {


returning a new snapshot is not useful here.

This function is not needed any more.

Makefile

cmd/csi-snapshotter-common/main.go

cmd/csi-snapshotter-sidecar/main.go

deploy/kubernetes/common-snapshot-controller.yaml

deploy/kubernetes/rbac-common.yaml

deploy/kubernetes/setup-csi-snapshotter.yaml

pkg/utils/util.go

msau42 · 2019-11-06T02:56:29Z

I did a very quick review. Did not look into the controller logic at all.

cmd/csi-snapshotter/main.go

pkg/utils/util.go

yuxiangqian

in general LGTM, two minor corner cases to be fixed and then I think we are in a good shape for the first cut.

Fixed the two cases:

pre-provisioned snapshots with no snapshot class, don't return nil there
removed the un-used csi capability check function from snapshot-controller/main.go

yuxiangqian · 2019-11-07T23:08:00Z

cmd/snapshot-controller/main.go

+	return rest.InClusterConfig()
+}
+
+func supportsControllerCreateSnapshot(ctx context.Context, conn *grpc.ClientConn) (bool, error) {


this should be moved to sidecar controller. In common controller, driver validation is no longer needed. common controller does not need to know anything about CSI.

Removed. This is not called by common controller and sidecar controller already has this function.

yuxiangqian · 2019-11-07T23:21:51Z

pkg/common-controller/snapshot_controller.go

+		snapshot = nil
+	} else {
+		// Check if content status is set to true and update snapshot status if so
+		if snapshot != nil && content.Status != nil && content.Status.ReadyToUse != nil && *content.Status.ReadyToUse == true {


I am ok with this check for now, in next round we should be verifying snapshot.Status need to be updated or not instead of just the ReadyToUse field.
i.e., if snapshot.Status.ReadyToUse == true, no-op here

Not sure if that is necessary as snapshot status will be updated eventually. This one is added in particular to address the problem that when content status becomes ready, there is a long wait before snapshot status becomes ready. "ReadyToUse" is usually the last thing we are waiting for. Let's look at this again later.

yuxiangqian · 2019-11-08T00:01:54Z

pkg/common-controller/snapshot_controller.go

+}
+
+// removeSnapshotFinalizer removes a Finalizer for VolumeSnapshot.
+func (ctrl *csiSnapshotCommonController) removeSnapshotFinalizer(snapshot *crdv1.VolumeSnapshot, removeSourceFinalizer bool, removeBoundFinalizer bool) error {


if both removeSourceFinalizer and removeBoundFinalizer are false, or if there is not update on snapshotClone.ObjectMeta.Finalizers, this function should just return to avoid API call.

ok to optimize in later version

good point. Fixed.

yuxiangqian · 2019-11-08T00:05:31Z

pkg/common-controller/snapshot_controller.go

+// syncReadySnapshot checks the snapshot which has been bound to snapshot content successfully before.
+// If there is any problem with the binding (e.g., snapshot points to a non-exist snapshot content), update the snapshot status and emit event.
+func (ctrl *csiSnapshotCommonController) syncReadySnapshot(snapshot *crdv1.VolumeSnapshot) error {
+	if !utils.IsBoundVolumeSnapshotContentNameSet(snapshot) {


based on the comments, should an event been added here? or is this call really needed? util.IsSnapshotReady(snapshot) should have checked?

I made those util functions more explicit. So IsSnapshotReady only checks the ReadyToUse flag. It doesn't check the BoundVolumeSnapshotContentName.

Let's revisit this function later.

yuxiangqian · 2019-11-08T00:20:06Z

pkg/common-controller/snapshot_controller.go

+				}
+				// NOTE: this is not an error now because we delete content before the snapshot
+				klog.V(5).Infof("Content for snapshot %s not found. It may be already deleted as expected.", uniqueSnapshotName)
+			} else {


In this case, should snapshot.Status be updated just in case?

ok to optimize later. adding status update logic here only saves unnecessary createSnapshot call

yuxiangqian · 2019-11-08T00:31:46Z

pkg/common-controller/snapshot_controller.go

+			klog.V(4).Infof("Skipping static bound snapshot %s when checking PVC %s/%s", snap.Name, pvc.Namespace, pvc.Name)
+			continue
+		}
+		if snap.Spec.Source.PersistentVolumeClaimName != nil && pvc.Name == *snap.Spec.Source.PersistentVolumeClaimName && (snap.Status == nil || snap.Status.ReadyToUse == nil || (snap.Status.ReadyToUse != nil && *snap.Status.ReadyToUse == false)) {


minor: (snap.Status == nil || snap.Status.ReadyToUse == nil || (snap.Status.ReadyToUse != nil && *snap.Status.ReadyToUse == false)) is the same as !IsSnapshotReady(snap)

yuxiangqian · 2019-11-08T00:35:58Z

pkg/common-controller/snapshot_controller.go

+		return nil, fmt.Errorf("Could not bind snapshot %s and content %s, the VolumeSnapshotRef does not match", snapshot.Name, content.Name)
+	} else if content.Spec.VolumeSnapshotRef.UID != "" && content.Spec.VolumeSnapshotRef.UID != snapshot.UID {
+		return nil, fmt.Errorf("Could not bind snapshot %s and content %s, the VolumeSnapshotRef does not match", snapshot.Name, content.Name)
+	} else if content.Spec.VolumeSnapshotRef.UID != "" && content.Spec.VolumeSnapshotClassName != nil {


minor, no need to update:
content.Spec.VolumeSnapshotClassName != nil seems to be redundant.
also better to also verify namespace of snapshot as well. We will do it in next round.

yuxiangqian · 2019-11-08T00:44:09Z

pkg/sidecar-controller/snapshot_controller.go

+		klog.V(4).Infof("updating VolumeSnapshotContent[%s] error status: cannot update internal cache %v", content.Name, err)
+		return err
+	}
+	// Emit the event only when the status change happens


move event emitting before storeContentUpdate.

yuxiangqian · 2019-11-08T00:47:52Z

pkg/sidecar-controller/snapshot_controller.go

+		}
+		// For pre-provisioned snapshot, snapshot class is not required
+		klog.V(5).Infof("getCSISnapshotInput for content [%s]: no VolumeSnapshotClassName provided for pre-provisioned snapshot", content.Name)
+		return nil, nil, nil


returning here could cause issue for the following case:
pre-existing snapshot, no snapshot class supplied in content, however does have credential annotation.
simply remove this return statement should just fix

yuxiangqian · 2019-11-08T00:48:57Z

pkg/sidecar-controller/snapshot_controller.go

+			return nil, err
+		}
+		driverName = content.Spec.Driver
+		if content.Spec.Source.SnapshotHandle != nil {


redundant check

yuxiangqian · 2019-11-08T21:26:10Z

/lgtm
This is a preliminary version to unblock core feature enabling. We will need to revisit some of the implementations right after the core merging.
tests have been conducted on hostpath CSI, however not enough has been done for cloud storage vendors. looking also to add more e2e tests on core to cover that.
thanks for the hardwork @xing-yang
thanks for reviewing @msau42 @saad-ali @jingxu97

670bb0e Merge pull request kubernetes-csi#229 from marosset/fix-codespell-errors 35d5e78 Merge pull request kubernetes-csi#219 from yashsingh74/update-registry 63473cc Merge pull request kubernetes-csi#231 from coulof/bump-go-version-1.20.5 29a5c76 Merge pull request kubernetes-csi#228 from mowangdk/chore/adopt_kubernetes_recommand_labels 8dd2821 Update cloudbuild image with go 1.20.5 1df23db Merge pull request kubernetes-csi#230 from msau42/prow 1f92b7e Add ginkgo timeout to e2e tests to help catch any stuck tests 2b8b80e fixing some codespell errors c10b678 Merge pull request kubernetes-csi#227 from coulof/check-sidecar-supported-versions 72984ec chore: adopt kubernetes recommand label b055535 Header bd0a10b typo c39d73c Add comments f6491af Script to verify EOL sidecar version 4133d1d Merge pull request kubernetes-csi#226 from msau42/cloudbuild 8d519d2 Pin buildkit to v0.10.6 to workaround v0.11 bug with docker manifest 6e04a03 Merge pull request kubernetes-csi#224 from msau42/cloudbuild 26fdfff Update cloudbuild image 6613c39 Merge pull request kubernetes-csi#223 from sunnylovestiramisu/update 0e7ae99 Update k8s image repo url 77e47cc Merge pull request kubernetes-csi#222 from xinydev/fix-dep-version 155854b Fix dep version mismatch 8f83905 Merge pull request kubernetes-csi#221 from sunnylovestiramisu/go-update 1d3f94d Update go version to 1.20 to match k/k v1.27 e322ce5 Merge pull request kubernetes-csi#220 from andyzhangx/fix-golint-error b74a512 test: fix golint error 901bcb5 Update registry k8s.gcr.io -> registry.k8s.io aa61bfd Merge pull request kubernetes-csi#218 from xing-yang/update_csi_driver 7563d19 Update CSI_PROW_DRIVER_VERSION to v1.11.0 a2171be Merge pull request kubernetes-csi#216 from msau42/process cb98782 Merge pull request kubernetes-csi#217 from msau42/owners a11216e add new reviewers and remove inactive reviewers dd98675 Add step for checking builds b66c082 Merge pull request kubernetes-csi#214 from pohly/junit-fixes b9b6763 filter-junit.go: fix loss of testcases when parsing Ginkgo v2 JUnit d427783 filter-junit.go: preserve system error log 38e1146 prow.sh: publish individual JUnit files as separate artifacts 78c0fb7 Merge pull request kubernetes-csi#208 from jsafrane/skip-selinux 36e433e Skip SELinux tests in CI by default 348d4a9 Merge pull request kubernetes-csi#207 from RaunakShah/reviewers 1efc272 Merge pull request kubernetes-csi#206 from RaunakShah/update-prow 7d410d8 Changes to csi prow to run e2e tests in sidecars cfa5a75 Merge pull request kubernetes-csi#203 from humblec/test-vendor 4edd1d8 Add RaunakShah to CSI reviewers group 7ccc959 release tools update to 1.19 d24254f Merge pull request kubernetes-csi#202 from xing-yang/kind_0.14.0 0faa3fc Update to Kind v0.14.0 images ef4e1b2 Merge pull request kubernetes-csi#201 from xing-yang/add_1.24_image 4ddce25 Add 1.24 Kind image 7fe5149 Merge pull request kubernetes-csi#200 from pohly/bump-kubernetes-version 70915a8 prow.sh: update snapshotter version 31a3f38 Merge pull request kubernetes-csi#199 from pohly/bump-kubernetes-version 7577454 prow.sh: bump Kubernetes to v1.22.0 d29a2e7 Merge pull request kubernetes-csi#198 from pohly/csi-test-5.0.0 41cb70d prow.sh: sanity testing with csi-test v5.0.0 c85a63f Merge pull request kubernetes-csi#197 from pohly/fix-alpha-testing b86d8e9 support Kubernetes 1.25 + Ginkgo v2 ab0b0a3 Merge pull request kubernetes-csi#192 from andyzhangx/patch-1 7bbab24 Merge pull request kubernetes-csi#196 from humblec/non-alpha e51ff2c introduce control variable for non alpha feature gate configuration ca19ef5 Merge pull request kubernetes-csi#195 from pohly/fix-alpha-testing 3948331 fix testing with latest Kubernetes e4dab7f Merge pull request kubernetes-csi#194 from yselkowitz/registry-k8s-io 84a4d5a Move from k8s.gcr.io to registry.k8s.io 9a0260c fix boilerplate header 37d1104 Merge pull request kubernetes-csi#191 from pohly/go-1.18 db917f5 update to Go 1.18 335339f Merge pull request kubernetes-csi#187 from mauriciopoppe/remove-eol-windows-versions 890b87a Merge pull request kubernetes-csi#188 from pwschuurman/update-release-notes-docs 274bc9b Update Sidecar Release Process documentation to reference latest syntax for release-notes tool 87b6c37 Merge pull request kubernetes-csi#185 from Garima-Negi/fix-OWNERS-files f1de2c6 Fix OWNERS file - squashed commits 59ae38b Remove EOL windows versions from BUILD_PLATFORMS 5d66471 Merge pull request kubernetes-csi#186 from humblec/sp d066f1b Correct prow.sh typo and make codespell linter pass 762e22d Merge pull request kubernetes-csi#184 from pohly/image-publishing-troubleshooting 81e26c3 SIDECAR_RELEASE_PROCESS.md: add troubleshooting for image publishing 31aa44d Merge pull request kubernetes-csi#182 from chrishenzie/csi-sanity-version f49e141 Update csi-sanity test suite to v4.3.0 d9815c2 Merge pull request kubernetes-csi#181 from mauriciopoppe/armv7-support 05c1801 Add support to build arm/v7 images 4aedf35 Merge pull request kubernetes-csi#178 from xing-yang/timeout 2b9897e Increase build timeout 51d3702 Merge pull request kubernetes-csi#177 from mauriciopoppe/kind-image-1.23 a30efea Add kind image for 1.23 a6a1a79 Merge pull request kubernetes-csi#176 from pohly/go-1.17.3 0a2cf63 prow.sh: bump Go to 1.17.3 fc29fdd Merge pull request kubernetes-csi#141 from pohly/prune-replace-optional 5b9a1e0 Merge pull request kubernetes-csi#175 from jimdaga/patch-1 5eeb602 images: use k8s-staging-test-infra/gcb-docker-gcloud b46691a go-get-kubernetes.sh: make replace statement pruning optional git-subtree-dir: release-tools git-subtree-split: 670bb0e

…sion Update csi-sanity test suite to v4.3.0

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Oct 21, 2019

k8s-ci-robot requested review from lpabon and saad-ali October 21, 2019 12:39

xing-yang mentioned this pull request Oct 21, 2019

WIP: Split the controller into common and sidecar controllers - based on Snapshot Beta APIs #178

Closed

xing-yang force-pushed the new_beta_split_controller branch from 025d873 to 2035cfc Compare October 23, 2019 16:49

yuxiangqian reviewed Oct 25, 2019

View reviewed changes

xing-yang force-pushed the new_beta_split_controller branch from 2035cfc to ac7f89e Compare October 25, 2019 19:22

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 25, 2019

xing-yang force-pushed the new_beta_split_controller branch from ac7f89e to ef3e675 Compare October 26, 2019 14:34

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 26, 2019

xing-yang force-pushed the new_beta_split_controller branch 7 times, most recently from 7e85b30 to 855f6ea Compare October 29, 2019 17:04

yuxiangqian reviewed Oct 29, 2019

View reviewed changes

xing-yang force-pushed the new_beta_split_controller branch 2 times, most recently from 8e84325 to 7261d2e Compare October 30, 2019 04:34

yuxiangqian reviewed Oct 30, 2019

View reviewed changes

xing-yang force-pushed the new_beta_split_controller branch from 7261d2e to fc7ba19 Compare October 31, 2019 17:35

jingxu97 reviewed Oct 31, 2019

View reviewed changes

xing-yang force-pushed the new_beta_split_controller branch 3 times, most recently from bf9a419 to bc596b3 Compare November 3, 2019 01:02

saad-ali approved these changes Nov 3, 2019

View reviewed changes

xing-yang force-pushed the new_beta_split_controller branch from bc596b3 to f49fe34 Compare November 3, 2019 15:35

yuxiangqian reviewed Nov 5, 2019

View reviewed changes

xing-yang mentioned this pull request Nov 5, 2019

Remove driver name from Snapshotter CreateSnapshot return values #190

Closed

xing-yang force-pushed the new_beta_split_controller branch from f49fe34 to 4954e80 Compare November 5, 2019 14:59

yuxiangqian reviewed Nov 5, 2019

View reviewed changes

msau42 reviewed Nov 6, 2019

View reviewed changes

xing-yang force-pushed the new_beta_split_controller branch 2 times, most recently from 04f6eb0 to 13f7ea6 Compare November 7, 2019 04:31

Split snapshot controller using beta APIs

6308420

xing-yang force-pushed the new_beta_split_controller branch from 13f7ea6 to 6308420 Compare November 7, 2019 16:50

msau42 reviewed Nov 7, 2019

View reviewed changes

cmd/csi-snapshotter/main.go Outdated Show resolved Hide resolved

pkg/utils/util.go Outdated Show resolved Hide resolved

yuxiangqian reviewed Nov 8, 2019

View reviewed changes

Address review comments

9000e9c

k8s-ci-robot assigned yuxiangqian Nov 8, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 8, 2019

k8s-ci-robot merged commit fa239cb into kubernetes-csi:master Nov 8, 2019

xing-yang mentioned this pull request Nov 11, 2019

Separate common controller logic from the sidecar #140

Closed

ialidzhikov mentioned this pull request Apr 30, 2020

Improve csi-snapshotter ClusterRole kubernetes-sigs/aws-ebs-csi-driver#491

Merged

andyzhangx mentioned this pull request Oct 29, 2020

Improve csi-snapshotter ClusterRole kubernetes-sigs/azuredisk-csi-driver#581

Closed

edreed mentioned this pull request Dec 11, 2020

chore: Remove unnecessary permissions from external-snapshotter role. kubernetes-sigs/azuredisk-csi-driver#612

Merged

4 tasks

dr0pdb mentioned this pull request Dec 18, 2020

chore: remove unnecessary permissions from external snapshotter role kubernetes-sigs/azurefile-csi-driver#501

Merged

4 tasks

andyzhangx pushed a commit to andyzhangx/external-snapshotter that referenced this pull request Jun 5, 2024

Merge pull request kubernetes-csi#182 from chrishenzie/csi-sanity-ver…

31aa44d

…sion Update csi-sanity test suite to v4.3.0

Split snapshot controller using beta APIs #182

Split snapshot controller using beta APIs #182

Conversation

xing-yang commented Oct 21, 2019 • edited Loading

yuxiangqian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian Nov 4, 2019 • edited by xing-yang Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian Nov 5, 2019 • edited by xing-yang Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xing-yang Oct 30, 2019 • edited Loading

Choose a reason for hiding this comment

xing-yang Oct 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xing-yang Oct 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xing-yang Oct 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian Oct 29, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian Nov 1, 2019 • edited by xing-yang Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xing-yang Oct 31, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saad-ali left a comment

Choose a reason for hiding this comment

xing-yang commented Oct 21, 2019 •

edited

Loading

yuxiangqian Nov 4, 2019 •

edited by xing-yang

Loading

yuxiangqian Nov 5, 2019 •

edited by xing-yang

Loading

xing-yang Oct 30, 2019 •

edited

Loading

xing-yang Oct 30, 2019 •

edited

Loading

xing-yang Oct 30, 2019 •

edited

Loading

xing-yang Oct 30, 2019 •

edited

Loading

yuxiangqian Oct 29, 2019 •

edited

Loading

yuxiangqian Nov 1, 2019 •

edited by xing-yang

Loading

xing-yang Oct 31, 2019 •

edited

Loading