Update Extended Resource doc wrt cluster-level resources #7759

yguo0905 · 2018-03-16T00:31:36Z

This is to update the Extended Resource doc wrt cluster-level resources after kubernetes/kubernetes#60332 is released in 1.10.

/assign @vishh
/assign @jiayingz

yguo0905 · 2018-03-16T00:33:13Z

/assign @bsalamat

k8sio-netlify-preview-bot · 2018-03-16T00:33:35Z

Deploy preview for kubernetes-io-vnext-staging ready!

Built with commit 75395a7

https://deploy-preview-7759--kubernetes-io-vnext-staging.netlify.com

tengqm · 2018-03-16T03:17:49Z

docs/concepts/configuration/manage-compute-resources-container.md

-JSON-Pointer. For more details, see
-[IETF RFC 6901, section 3](https://tools.ietf.org/html/rfc6901#section-3).
-{: .note}
-


Line 391-423 above is about advertising an extended resource given that these resources are not discoverable by Kubernetes. Why are we deleting this content?

I think it should be the Device Plugin that advertises the node-level resources. If there's a legit use case of directly patching node objects with HTTP requests, let me know. I will add this back.

In the previous paragraphs, it is mentioned that software licenses can be managed as extended resources, no?
I am not aware of any efforts to support that using device plugins.

tengqm

Please clarify the reason for some of the changes made.

Bradamant3 · 2018-03-16T20:37:42Z

/assign

jiayingz · 2018-03-16T20:45:48Z

/lgtm

yguo0905 · 2018-03-19T16:07:09Z

@Bradamant3, can you please take a look?

Bradamant3

copyedits for clarity and compliance with style guide

Bradamant3 · 2018-03-21T23:05:06Z

docs/concepts/configuration/manage-compute-resources-container.md

+
+**Example:**
+
+The below scheduler policy configuration indicates that the cluster-level


Sorry for late review. Please change:
s/below scheduler policy configuration/following configuration for a scheduler policy

Bradamant3 · 2018-03-21T23:05:46Z

docs/concepts/configuration/manage-compute-resources-container.md

+
+The below scheduler policy configuration indicates that the cluster-level
+extended resource "example.com/foo" is handled by scheduler extender.
+ - The scheduler will only send a pod to the scheduler extender iff the pod


s/will only send a pod to the scheduler extender iff/sends a pod to the scheduler extender only if

Bradamant3 · 2018-03-21T23:07:06Z

docs/concepts/configuration/manage-compute-resources-container.md

+extended resource "example.com/foo" is handled by scheduler extender.
+ - The scheduler will only send a pod to the scheduler extender iff the pod
+   requests "example.com/foo".
+ - The `ignoredByScheduler` indicates that scheduler will skip checking the


s/indicates that scheduler will skip checking/field specifies that the scheduler does not check

Bradamant3 · 2018-03-21T23:07:41Z

docs/concepts/configuration/manage-compute-resources-container.md

-remain in the `PENDING` state as long as the resource request cannot be met by
-any node.
+A Pod is scheduled only if all of the resource requests are satisfied, including
+CPU, memory and any Extended Resources. The Pod will remain in the `PENDING`


s/will remain/remains

tengqm · 2018-03-22T01:42:55Z

docs/concepts/configuration/manage-compute-resources-container.md

-JSON-Pointer. For more details, see
-[IETF RFC 6901, section 3](https://tools.ietf.org/html/rfc6901#section-3).
-{: .note}
-


In the previous paragraphs, it is mentioned that software licenses can be managed as extended resources, no?
I am not aware of any efforts to support that using device plugins.

tengqm · 2018-03-22T01:46:44Z

docs/concepts/configuration/manage-compute-resources-container.md

+  "apiVersion": "v1",
+  "extenders": [
+    {
+      ...


This line is making the JSON invalid, please remove.

Bradamant3 · 2018-03-22T16:47:39Z

@yguo0905 some technical questions are still not answered, and I've asked for copyedits. Can you get these changes in right away? Thanks!

yguo0905 · 2018-03-22T18:15:49Z

In the previous paragraphs, it is mentioned that software licenses can be managed as extended resources, no?
I am not aware of any efforts to support that using device plugins.

Software license is an example of cluster-level resource, which is handled by scheduler extender, rather than device plugin.

yguo0905 · 2018-03-22T18:16:26Z

PTAL.

Bradamant3 · 2018-03-22T23:07:27Z

/approve

@tengqm if you're satisfied with latest changes/responses to your question/s, can you please add an /lgtm so we can get this merged? Thank you, and thanks again to @yguo0905 for bearing with last-minute copyedits.

tengqm · 2018-03-23T01:22:12Z

@Bradamant3, @yguo0905 I'm still not convinced that 1) all node level extended resources can/should be covered by the device plugin framework, 2) software license is a cluster-level resource to be managed.

There are cases where software are licensed on a per-core basis, per-seat basis. That is why I think the software license example is not a good one, although I agree there are cases where modeling them as cluster-level resources makes sense. It just doesn't have to be a cluster-level resource.

I'd suggest we avoid doing changes which imply some decisions are made for the users.

yguo0905 · 2018-03-23T05:25:41Z

all node level extended resources can/should be covered by the device plugin framework

I think device plugin is designed to do so. Can you please give an example where a node level extended resource shouldn't be handled by device plugin?

software license is a cluster-level resource to be managed.

GPUs and software licences are used as examples. It is possible that different software licenses have different requirements. It is also possible that users expose their GPUs as a network service, where GPUs become a cluster-level resource. I can avoid mentioning both examples. WDYT?

tengqm · 2018-03-23T06:14:18Z

Well, here are two examples:

I have a log analysis solution (a game system, a remote desktop system etc) that streams data to/from each node but I have to restrict the number of concurrent connections from each node. The number of connections is thus a consumable, opaque integer resources to be managed so my sever won't crash.
I bought a software from a vendor who refuses to grant me free license if I'm running it on a (virtual) system with more than 4 cores. I have to carefully schedule number of Pods running that software so that I don't have to pay extra money.

The key is this ... extended resources is a much broader term than devices, which is a much broader term than GPUs. I don't see how device plugins as a replacement of extended resources.

Bradamant3 · 2018-03-24T01:47:56Z

@tengqm @yguo0905 Release is on track for Monday, March 26. Can you please get agreement about needed changes and get them in no later than noon Eastern US time that day? I still need to merge and publish the branch after all PRs are merged. Otherwise, this one will have to wait until after release and get rebased to master. Thanks!

Bradamant3 · 2018-03-26T13:13:54Z

Folks, I can help sort some of the outstanding issues here, because a quick look suggests that some of the confusion/disagreement lies in the way things are worded, not in the technical accuracy. But I can't do it today; we need to ship the release first. I'll get a revision in as soon as I can, but it will take a bit of time. Diffs for doc changes this big are not always helpful :-(

Adding Docs Review: Open Issues label to help me find this PR after release, if needed. If y'all get agreement today, I'll remove it.

tengqm · 2018-03-26T13:45:47Z

@Bradamant3 Thanks for the heads up.

yguo0905 · 2018-03-26T16:58:43Z

@tengqm, thanks for the example.
@Bradamant3, sorry for the delay.
PTAL.

heckj · 2018-03-26T20:22:46Z

/assign
/hold
/approve

Bradamant3 · 2018-03-26T20:34:41Z

/hold cancel

/lgtm

note that lgtm is per earlier lgtms that the bot either dropped or ignored

k8s-ci-robot · 2018-03-26T20:34:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bradamant3, heckj, jiayingz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Bradamant3,heckj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* 1.10 update (#7151) * Fix partition value expected behaviour explanation (#7123) Fixes issue #7057 * Correct "On-Premise" to "On-Premises" * Updates the Calico installation page (#7094) * All files for Haufe Groups case study (#7051) * Fix typo (#7127) * fix typo of device-plugins.md (#7106) * fix broken links (#7136) * Updated configure-service-account (#7147) Error from server resolved by escaping kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "myregistrykey"}]}' JSON string by '\' * Remove docs related to 'require-kubeconfig' (#7138) With kubernetes/kubernetes#58367 merged, v1.10 will not use the "require-kubeconfig" flag. The flag has become a no-op solely to ensure existing deployments won't break. * Added Verification Scenario for a Pod that Uses a PVC in Terminating State (#7164) The below PR: kubernetes/kubernetes#55873 modified scheduler in such a way that scheduling of a pod that uses a PVC in Terminating state fails. That's why verification of such scenario was added to documentation. * fix LimitPodHardAntiAffinityTopology name (#7221) * Document the removal of the KubeletConfigFile feature gate (#7140) With kubernetes/kubernetes#58978 merged, the said feature gate is removed. This PR removes texts related to the gate and revises the Feature Gates reference to reflect this change. * deprecate three admission controller (#7363) * Document the removal of Accelerators feature gate (#7389) The `Accelerators` feature gate will be removed in 1.11. 1.10 will be its last mile. References: kubernetes/kubernetes#57384 * Update local storage docs for beta (#7473) * Document that HugePages feature gate is Beta (#7387) The `HugePages` feature gate has graduated to Beta in v1.10. This PR documents this fact. * Add HyperVContainer feature gates (#7502) * Remove the beta reference from Taints and Tolerations doc (#7493) * Kms provider doc (#7479) * Kms provider doc * issue# 7399, Create KMS-provider.md and update encrypt-data.md * address review comments * Document that Device Plugin feature is Beta (1.10) (#7512) * Add docs for CRD features for 1.10 (#7439) * Add docs for CRD features for 1.10 * Add CustomResourcesSubresources to list of feature gates * Add latest changes to custom resources doc * Add crds as abbreviated alias (#7437) * Bring PVC Protection Feature to Beta (#7165) * Bring PVC Protection Feature to Beta The PR: kubernetes/kubernetes#59052 brought PVC Protection feature to beta. That's why the documentation is updated accordingly. * The PVC Protection feature was renamed to Storage Protection. That's why the documentation is updated. * promote PodNodeSelector to stable; document detailed behavior (#7134) * promote PodNodeSelector to stable; document detailed behavior * respond to feedback * Update CPU manager feature enabling (#7390) With `CPUManager` feature graduating to beta. No explicit enabling is required starting v1.10. References: kubernetes/kubernetes#55977 * Adding block volumeMode documentation for local volumes. (#7531) Code review comments. Changed property to field. Address tech review comment. * remove description kubectl --show-all (#7574) --show-all has been deprecated and set to true by default. kubernetes/kubernetes#60210 * fix description about contribute style guide (#7592) * fix description about KUBECONFIG (#7589) s/envrionment/environment * fix description about cni (#7588) s/simultanously/simultaneously/ * fix description about MutatingAdmissionWebhook and ValidatingAdmissionWebhook (#7587) * fix description about persistent volume binding (#7590) s/slighty/slightly/ * Doc change for configurable pod resolv.conf Beta (#7611) * fix description about out of resource handling (#7597) s/threshhold/threshold * fix description about zookeeper (#7598) s/achive/achieve * fix description about kubeadm (#7594) s/compatability/compatibility/ * fix description about kubeadm (#7593) * fix description about kubeadm implementation details (#7595) * fix description about api concepts (#7596) * Storage Protection was renamed to Storage Object in Use Protection (#7576) * Storage Protection was renamed to Storage Object in Use Protection The K8s PR: kubernetes/kubernetes#59901 renamed Storage Protection to Storage Object in Use Protection. That's why the same is also renamed in the documentation. * Moved Storage Object in Use Protection admission plugin description down according to alphabetic order. * Use PSP from policy API group. (#7562) * update kubeletconfig docs for v1.10, beta (#7561) * Update port-forwarding docs (#7575) * add pv protection description (#7620) * fix description about client library (#7634) * Add docs on configuring NodePort IP (#7631) * Document that LocalStorageCapacityIsolation is beta (#7635) A follow-up to the kubernetes/kubernetes#60159 change which has promoted the `LocalStorageCapacityIsolation` feature gate to Beta. * Update CoreDNS docs for beta (#7638) * Update CoreDNS docs for beta * Review comments * Fix typo (#7640) * Update feature gates move to beta (#7662) * Added the inability to use colon ':' character as environment variable names and described workaround (#7657) * merge master to 1.10, with fixes (#7682) * Flag names changed (s/admission-control/enable-admission-plugins); disable-admissions-plugin entry added; removed reference to admission controller/plugins requiring set order (for v1.10), redundant example enabling specific plugin, and redundant version-specific info (#7449) * Documentation for MountPropagation beta (#7655) * Remove job's scale-related operations (#7684) * authentication: document client-go exec plugins (#7648) * authentication: document client-go exec plugins * Update authentication.md * Update local ephemeral storage feature to beta (#7685) Update local ephemeral storage feature to beta * Update docs for windows container resources (#7653) * add server-side print docs (#7671) * Create a task describing Pod process namespace sharing (#7489) * Add external metrics to HPA docs (#7664) * Add external metrics to HPA docs * Update horizontal-pod-autoscale-walkthrough.md * Apply review comments to HPA walkthrough * remove description about "scale jobs" (#7712) * CSI Docs for K8s v1.10 (#7698) * Add a warning about increased memory consumption for audit logging feature. (#7725) Signed-off-by: Mik Vyatskov <[email protected]> * Update Audit Logging documentation for 1.10 (#7679) Signed-off-by: Mik Vyatskov <[email protected]> * Fix stage names in audit logging documentation (#7746) Signed-off-by: Mik Vyatskov <[email protected]> * Feature gate update for release 1.10 (#7742) * State in the docs that the value of default Node labels are not reliable. (#7794) * Kill the reference to --admission-control option (#7755) The `--admission-control` option has been replaced by two new options in v1.10. This PR kills the last appearance of the old option in the doc. * Pvcprotection toc (#7807) * Refreshing installation instructions (#7495) * Refreshing installation instructions Added conjure-up. Updated displays and juju versions to current versions. * Updated anchors * Fixed image value version typo (#7768) Was inconsistent with other values * Update flocker reference to the github repo (#7784) * Fix typo in federation document (#7779) * an user -> a user (#7778) * Events are namespaced (#7767) * fix 'monitoring' link lose efficacy problem' (#7764) * docs/concepts/policy/pod-security-policy.md: minor fix. (#7659) * Update downward-api-volume-expose-pod-information.md (#7771) * Update downward-api-volume-expose-pod-information.md The pod spec puts the downward api files into /etc/podinfo, not directly in /etc. Updated docs to reflect this fact. * Update downward-api-volume-expose-pod-information.md One more spot needed fixing. * Update downward-api-volume-expose-pod-information.md Yet another fix, in the container example. * Add Amadeus Case Study (#7783) * Add Amadeus Case Study * add Amadeus logo * Fixed Cyrillic с in 'kube-proxy-cm' (#7787) There was a typo (wrong character) in kube-proxy-cm.yaml - Cyrillic с (UTF-8 0x0441) was used instead of Latin c. * install-kubectl: choose one installation method (#7705) The previous text layout suggested that all installations had to be done, one after another. * Update install-kubeadm.md (#7781) Add note to kubeadm install instruction to help install in other arch i.e. aarch64, ppc64le etc. * repair failure link (#7788) * repair failure link * repair failure link * do change as required * Update k8s201.md (#7777) * Update k8s201.md Change instructions to download yams files directly from the website (as used in other pages.) Added instructions to delete labeled pod to avoid warnings in the subsequent deployment step. * Update k8s201.md Added example of using the exposed host from the a node running Kubernetes. (This works on AWS with Weave; not able to test it on other variations...) * Gramatical fix to kompose introduction (#7792) The original wording didn't through very well. As much of the original sentence has been preserved as possible, primarily to ensure the kompose web address is see both in text and as a href link. * update amadeus.html (#7800) * Fix a missing word in endpoint reconciler section (#7804) * add toc entry for pvcprotection downgrade issue doc * Pvcprotection toc (#7809) * Refreshing installation instructions (#7495) * Refreshing installation instructions Added conjure-up. Updated displays and juju versions to current versions. * Updated anchors * Fixed image value version typo (#7768) Was inconsistent with other values * Update flocker reference to the github repo (#7784) * Fix typo in federation document (#7779) * an user -> a user (#7778) * Events are namespaced (#7767) * fix 'monitoring' link lose efficacy problem' (#7764) * docs/concepts/policy/pod-security-policy.md: minor fix. (#7659) * Update downward-api-volume-expose-pod-information.md (#7771) * Update downward-api-volume-expose-pod-information.md The pod spec puts the downward api files into /etc/podinfo, not directly in /etc. Updated docs to reflect this fact. * Update downward-api-volume-expose-pod-information.md One more spot needed fixing. * Update downward-api-volume-expose-pod-information.md Yet another fix, in the container example. * Add Amadeus Case Study (#7783) * Add Amadeus Case Study * add Amadeus logo * Fixed Cyrillic с in 'kube-proxy-cm' (#7787) There was a typo (wrong character) in kube-proxy-cm.yaml - Cyrillic с (UTF-8 0x0441) was used instead of Latin c. * install-kubectl: choose one installation method (#7705) The previous text layout suggested that all installations had to be done, one after another. * Update install-kubeadm.md (#7781) Add note to kubeadm install instruction to help install in other arch i.e. aarch64, ppc64le etc. * repair failure link (#7788) * repair failure link * repair failure link * do change as required * Update k8s201.md (#7777) * Update k8s201.md Change instructions to download yams files directly from the website (as used in other pages.) Added instructions to delete labeled pod to avoid warnings in the subsequent deployment step. * Update k8s201.md Added example of using the exposed host from the a node running Kubernetes. (This works on AWS with Weave; not able to test it on other variations...) * Gramatical fix to kompose introduction (#7792) The original wording didn't through very well. As much of the original sentence has been preserved as possible, primarily to ensure the kompose web address is see both in text and as a href link. * update amadeus.html (#7800) * Fix a missing word in endpoint reconciler section (#7804) * add toc entry for pvcprotection downgrade issue doc * revert TOC change * Release 1.10 (#7818) * Refreshing installation instructions (#7495) * Refreshing installation instructions Added conjure-up. Updated displays and juju versions to current versions. * Updated anchors * Fixed image value version typo (#7768) Was inconsistent with other values * Update flocker reference to the github repo (#7784) * Fix typo in federation document (#7779) * an user -> a user (#7778) * Events are namespaced (#7767) * fix 'monitoring' link lose efficacy problem' (#7764) * docs/concepts/policy/pod-security-policy.md: minor fix. (#7659) * Update downward-api-volume-expose-pod-information.md (#7771) * Update downward-api-volume-expose-pod-information.md The pod spec puts the downward api files into /etc/podinfo, not directly in /etc. Updated docs to reflect this fact. * Update downward-api-volume-expose-pod-information.md One more spot needed fixing. * Update downward-api-volume-expose-pod-information.md Yet another fix, in the container example. * Add Amadeus Case Study (#7783) * Add Amadeus Case Study * add Amadeus logo * Fixed Cyrillic с in 'kube-proxy-cm' (#7787) There was a typo (wrong character) in kube-proxy-cm.yaml - Cyrillic с (UTF-8 0x0441) was used instead of Latin c. * install-kubectl: choose one installation method (#7705) The previous text layout suggested that all installations had to be done, one after another. * Update install-kubeadm.md (#7781) Add note to kubeadm install instruction to help install in other arch i.e. aarch64, ppc64le etc. * repair failure link (#7788) * repair failure link * repair failure link * do change as required * Update k8s201.md (#7777) * Update k8s201.md Change instructions to download yams files directly from the website (as used in other pages.) Added instructions to delete labeled pod to avoid warnings in the subsequent deployment step. * Update k8s201.md Added example of using the exposed host from the a node running Kubernetes. (This works on AWS with Weave; not able to test it on other variations...) * Gramatical fix to kompose introduction (#7792) The original wording didn't through very well. As much of the original sentence has been preserved as possible, primarily to ensure the kompose web address is see both in text and as a href link. * update amadeus.html (#7800) * Fix a missing word in endpoint reconciler section (#7804) * Partners page updates (#7802) * Partners page updates * Update to ZTE link * Make using sysctls a task instead of a concept (#6808) Closes: #4505 * add a note when mount a configmap to pod (#7745) * adjust a note format (#7812) * Update docker-cli-to-kubectl.md (#7748) * Update docker-cli-to-kubectl.md Edited the document for adherence to the style guide and word usage. * Update docker-cli-to-kubectl.md * Incorporated the changes suggested. * Mount propagation update to include docker config (#7854) * update overridden config for 1.10 (#7847) * update overridden config for 1.10 * fix config file per comments * Update Extended Resource doc wrt cluster-level resources (#7759)

k8s-ci-robot assigned jiayingz and vishh Mar 16, 2018

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 16, 2018

k8s-ci-robot requested review from xiangpengzhao and zhangxiaoyu-zidif March 16, 2018 00:31

k8s-ci-robot assigned bsalamat Mar 16, 2018

tengqm reviewed Mar 16, 2018

View reviewed changes

tengqm requested changes Mar 16, 2018

View reviewed changes

Bradamant3 added this to the 1.10 milestone Mar 16, 2018

k8s-ci-robot assigned Bradamant3 Mar 16, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 16, 2018

zacharysarah added Needs Tech Review labels Mar 20, 2018

Bradamant3 suggested changes Mar 21, 2018

View reviewed changes

Bradamant3 removed Needs Docs Review labels Mar 21, 2018

tengqm requested changes Mar 22, 2018

View reviewed changes

tengqm reviewed Mar 22, 2018

View reviewed changes

yguo0905 force-pushed the client-2 branch from 7ebfa1c to e8f9538 Compare March 22, 2018 18:13

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 22, 2018

Bradamant3 approved these changes Mar 22, 2018

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 22, 2018

Bradamant3 added the Docs Review: Open Issues label Mar 26, 2018

yguo0905 force-pushed the client-2 branch 3 times, most recently from 7c50d2a to de4692a Compare March 26, 2018 16:57

Update Extended Resource doc wrt cluster-level resources

75395a7

yguo0905 force-pushed the client-2 branch from de4692a to 75395a7 Compare March 26, 2018 17:02

k8s-ci-robot assigned heckj Mar 26, 2018

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 26, 2018

heckj removed the Docs Review: Open Issues label Mar 26, 2018

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Mar 26, 2018

k8s-ci-robot merged commit 4be379c into kubernetes:release-1.10 Mar 26, 2018


		Example:

		The below scheduler policy configuration indicates that the cluster-level

Update Extended Resource doc wrt cluster-level resources #7759

Update Extended Resource doc wrt cluster-level resources #7759

Conversation

yguo0905 commented Mar 16, 2018

yguo0905 commented Mar 16, 2018

k8sio-netlify-preview-bot commented Mar 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tengqm left a comment

Choose a reason for hiding this comment

Bradamant3 commented Mar 16, 2018

jiayingz commented Mar 16, 2018

yguo0905 commented Mar 19, 2018

Bradamant3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bradamant3 commented Mar 22, 2018

yguo0905 commented Mar 22, 2018

yguo0905 commented Mar 22, 2018

Bradamant3 commented Mar 22, 2018

tengqm commented Mar 23, 2018

yguo0905 commented Mar 23, 2018

tengqm commented Mar 23, 2018

Bradamant3 commented Mar 24, 2018 • edited Loading

Bradamant3 commented Mar 26, 2018 • edited Loading

tengqm commented Mar 26, 2018

yguo0905 commented Mar 26, 2018

heckj commented Mar 26, 2018

Bradamant3 commented Mar 26, 2018

k8s-ci-robot commented Mar 26, 2018

k8sio-netlify-preview-bot commented Mar 16, 2018 •

edited

Loading

Bradamant3 commented Mar 24, 2018 •

edited

Loading

Bradamant3 commented Mar 26, 2018 •

edited

Loading