Added PSU util for Mellanox platforms, added PSU CLI to SNMP #1170

andrii-savka · 2017-11-20T20:28:40Z

- What I did
Added PSU plugin for Mellanox platforms.
Installed sonic-utilities to SNMP container for PSU status querying

- How I did it
Implemented PSU plugin for Mellanox platforms
Added sonic-utilities dependencies to SNMP component

- Description for the changelog
Implementation of PSU CLI and SNMP support

- A picture of a cute animal (not mandatory but encouraged)

lguohan · 2017-11-20T21:23:19Z

dockers/docker-snmp-sv2/Dockerfile.j2

@@ -1,5 +1,8 @@
 FROM docker-config-engine

+RUN echo 'deb http://debian-archive.trafficmanager.net/debian jessie-backports main' >> /etc/apt/sources.list && \


why do we need add backports into the config-engine? which package requires this backport repo?

Removed unused.

lguohan · 2017-11-20T21:24:10Z

dockers/docker-snmp-sv2/Dockerfile.j2

-RUN apt-get update && apt-get install -y libperl5.20 libpci3 libwrap0 \
-            libexpat1-dev \
-            curl gcc && \
+RUN apt-get update && apt-get install -y --force-yes libperl5.20 libpci3 libwrap0 \


why do we need force-yes?

It was required by python click default group but I will double check and remove if no needed.

@lguohan
It was required by following packets:
WARNING: The following packages cannot be authenticated!
python-click python-click-default-group python-tabulate
E: There are problems and -y was used without --force-yes

jleveque · 2017-11-21T02:39:52Z

rules/docker-snmp-sv2.mk

@@ -3,7 +3,7 @@
 DOCKER_SNMP_SV2 = docker-snmp-sv2.gz
 $(DOCKER_SNMP_SV2)_PATH = $(DOCKERS_PATH)/docker-snmp-sv2
 ## TODO: remove LIBPY3_DEV if we can get pip3 directly
-$(DOCKER_SNMP_SV2)_DEPENDS += $(SNMP) $(SNMPD) $(PY3) $(LIBPY3_DEV)
+$(DOCKER_SNMP_SV2)_DEPENDS += $(SNMP) $(SNMPD) $(PY3) $(LIBPY3_DEV) $(SONIC_UTILS)


Ultimately, there should be no need to install the sonic-utilities package in the SNMP docker. SNMP shouldn't rely on the CLI, but rather just the PSU plugin. However, the plugin bases currently live in the sonic-utilities repo. If we could move all of our Python modules into their own repo/Debian package (e.g., "sonic-python-modules"), we could simply install that package here, and wouldn't need to install all of the sonic-utilities dependencies.

@jleveque @lguohan
From my point of view, the refactoring of sonic-utilities dependencies can be done separately as we need PSU feature available ASAP.
I use the psuutil CLI to avoid the code duplication. In case to use plugin directly, we need to copy ("duplicate") the logic for loading vendor and platform specific plugin to SNMP MIB implementation.

I agree we need this ASAP. This was simply intended as a comment to consider for the future. I did not intend for you to make this change now.

If there is plan to refactor in near future, please add a TODO comment as above. #Closed

@andrii-savka: FYI, from inside the SNMP docker, you would not need to duplicate the logic for loading vendor and platform specific plugin, because each docker container mounts the appropriate platform and hwsku directories to /usr/share/sonic/platform/ and /usr/share/sonic/hwsku/, respectively. Therefore, from inside the docker, you can simply load the plugin from /usr/share/sonic/platform/plugins.

I agree with Qi that you should add TODO comments as necessary so that we remember this future change.

@jleveque @qiluo-msft
The "TODO" mark is added.
Regarding the plugin usage instead of CLI.
Currently, we can't avoid we sonic-utilities installing to SNMP docker because psu_base.py which required by vendor specific plugin is placed in sonic-util. repo.

Anyway, do we still need to use plugin directly from "/usr/share/sonic/platform/plugins/" instead of CLI in SNMP?

One of the main idea is that SNMP agent knows nothing about PSU implementation. It just uses a CLI to get a PSU status with no matter whether PSU CLI is plugin, C app or some other script.

Agreed to use PSU plugin in SNMP with current dependencies.

jleveque · 2017-11-21T02:41:52Z

device/mellanox/x86_64-mlnx_msn2100-r0/plugins/psuutil.py

+except ImportError, e:
+    raise ImportError (str(e) + "- required module not found")
+
+class PSUutil(PsuBase):


Suggest changing case of class name to PsuUtil to match CamelCase of SfpUtil

See matching suggestion here: sonic-net/sonic-utilities#152 (comment)

jleveque · 2017-11-21T02:42:00Z

device/mellanox/x86_64-mlnx_msn2410-r0/plugins/psuutil.py

+except ImportError, e:
+    raise ImportError (str(e) + "- required module not found")
+
+class PSUutil(PsuBase):


Suggest changing case of class name to PsuUtil to match CamelCase of SfpUtil

See matching suggestion here: sonic-net/sonic-utilities#152 (comment)

jleveque · 2017-11-21T02:42:06Z

device/mellanox/x86_64-mlnx_msn2700-r0/plugins/psuutil.py

+except ImportError, e:
+    raise ImportError (str(e) + "- required module not found")
+
+class PSUutil(PsuBase):


Suggest changing case of class name to PsuUtil to match CamelCase of SfpUtil

See matching suggestion here: sonic-net/sonic-utilities#152 (comment)

jleveque · 2017-11-21T02:42:10Z

device/mellanox/x86_64-mlnx_msn2740-r0/plugins/psuutil.py

+except ImportError, e:
+    raise ImportError (str(e) + "- required module not found")
+
+class PSUutil(PsuBase):


Suggest changing case of class name to PsuUtil to match CamelCase of SfpUtil

See matching suggestion here: sonic-net/sonic-utilities#152 (comment)

jleveque

I have merged the related sonic-utilities PR. Please update the sonic-utilities submodule pointer in this PR.

qiluo-msft · 2017-11-28T21:23:33Z

device/mellanox/x86_64-mlnx_msn2100-r0/plugins/psuutil.py

+        Retrieves the oprational status of power supply unit (PSU) defined
+                by index <index>
+
+        :param index: An integer, index of the PSU of which to query status


index [](start = 15, length = 5)

Is it 1-based? Add some comment #Closed

The base of index has been added.

qiluo-msft · 2017-11-28T21:24:19Z

device/mellanox/x86_64-mlnx_msn2100-r0/plugins/psuutil.py

+        if index and index > 0 and index <= self.get_num_psus():
+            return True
+
+        return False


return index and index > 0 and index <= self.get_num_psus() #Closed

qiluo-msft · 2017-11-28T21:26:47Z

device/mellanox/x86_64-mlnx_msn2100-r0/plugins/psuutil.py

+        :param index: An integer, index of the PSU of which to query status
+        :return: Boolean, True if PSU is plugged, False if not
+        """
+        if index and index > 0 and index <= self.get_num_psus():


index [](start = 11, length = 5)

isinstance(x, int) #Closed

Replaced with isinstance(index, int)

qiluo-msft · 2017-11-28T21:30:22Z

device/mellanox/x86_64-mlnx_msn2700-r0/plugins/psuutil.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python


Exactly same file as msn2410. Please use a symbol link. #Closed

Replaced to a symbol link.

qiluo-msft · 2017-11-28T21:36:50Z

device/mellanox/x86_64-mlnx_msn2410-r0/plugins/psuutil.py

+except ImportError, e:
+    raise ImportError (str(e) + "- required module not found")
+
+class PsuUtil(PsuBase):


PsuUtil [](start = 6, length = 7)

I see many code duplication in mellanox plugins. Is it possible to reduce them? Extract base class for mlnx? #Closed

Why do we need a base class for Mellanox in Mellanox plugin ?
The base class won't be used anymore.

@qiluo-msft: I don't understand what you intend to accomplish here. Could you elaborate?

The base class is simply an abstract class that each platform-specific plugin should implement.

(refine original comment) #Closed

qiluo-msft · 2017-11-28T21:39:42Z

dockers/docker-snmp-sv2/Dockerfile.j2

-            libexpat1-dev \
-            curl gcc && \
+RUN apt-get update && apt-get install -y --force-yes libperl5.20 libpci3 libwrap0 \
+            libexpat1-dev grub2-common bash-completion curl gcc \


grub2-common bash-completion [](start = 26, length = 28)

I think many dependencies are not needed if correctly refactored. If yes, please add a TODO comment. #Closed

The added packets are required by sonic utilities.
The "TODO" comment is added.

qiluo-msft · 2017-11-28T21:44:02Z

dockers/docker-snmp-sv2/Dockerfile.j2

@@ -1,5 +1,7 @@
 FROM docker-config-engine

+RUN echo 'deb [arch=amd64] http://packages.microsoft.com/repos/sonic-dev/ jessie main' >> /etc/apt/sources.list


sonic-dev/ [](start = 63, length = 10)

Why you need sonic-dev repo? #Closed

sonic-dev repo is necessary to download python-click and python-tabulate packages which are dependencies of sonic-utilities. This can all be removed in the future once this uses the PSU plugin directly (see my comment above).

Is it possible to install them by pip? #Closed

Unfortunately, dependency issues arise when we have Debian packages that rely on Python packages and vice-versa (for example: sonic-net/sonic-utilities#124).

qiluo-msft

As comments.

jleveque · 2017-11-30T19:09:49Z

dockers/docker-snmp-sv2/Dockerfile.j2

-RUN apt-get update && apt-get install -y libperl5.20 libpci3 libwrap0 \
-            libexpat1-dev \
-            curl gcc && \
+## TODO: remove click modules and grub2-common bash-completion if we can use PSU util directly


Update TODO here, since you are now using the PSU plugin directly.

TODO: remove grub2-common, bash-completion, python-pip, python-click-default-group, python-click, python-natsort and python-tabulate once we no longer need to install sonic-utilities for PSU plugin support

It's still required to install sonic-utilities for PsuBase accessing.

Text changed

jleveque · 2017-11-30T19:11:13Z

rules/docker-snmp-sv2.mk

@@ -3,7 +3,8 @@
 DOCKER_SNMP_SV2 = docker-snmp-sv2.gz
 $(DOCKER_SNMP_SV2)_PATH = $(DOCKERS_PATH)/docker-snmp-sv2
 ## TODO: remove LIBPY3_DEV if we can get pip3 directly
-$(DOCKER_SNMP_SV2)_DEPENDS += $(SNMP) $(SNMPD) $(PY3) $(LIBPY3_DEV)
+## TODO: replace SONIC_UTILS if we can use PSU util directly


Update TODO here.

TODO: replace SONIC_UTILS with appropriate target once SONiC Python modules are moved to their own package

#TODO description changed

qiluo-msft

Check with other reviewers.

…be copied in SNMP as a file

lguohan · 2017-12-04T10:46:48Z

@andrii-savka, please fill the commit message as it is required by every commit.

lguohan

please fill the commit message template.

andrii-savka · 2017-12-04T12:50:29Z

Added a PR description

jleveque

Reminder: please update the sonic-utilities submodule pointer, also.

lguohan · 2017-12-05T10:37:44Z

@andrii-savka , PR description has certain format.

What you did? How you did? What validation did you do? Can you use the template to fill out the description?

lguohan · 2017-12-06T16:58:32Z

@andrii-savka , I do not see sonic-utilities submodule pointer update? Do you need to do that?

lguohan · 2017-12-08T01:09:15Z

@andrii-savka , conflict on sonic-utilities, can you check?

4628a5a [vs tests] Refactor fdb_update tests to be consistent with vs test suite (#1185) 9594de0 Enable MIRRORv6 for BFN platform (#1177) a35afac [portsorch] use ingress/egress disable for LAG member enable/disable (#1166) faab3e1 remove the obsoleted platforms build links (#1169) 97abec5 [restore_neighbors.py] build arp packet with correct hwsrc and psrc (#1158) 97a22ca [vs tests] Mark VLAN and warm reboot tests as xfail (#1183) b453797 [Teamdmgrd]Fix "teamd cannot start when port channel was down before warm reboot" issue (#1171) 3732e74 [cfgmgr] clear loopback and vrf in kernel if not warmstart (#1141) 51315c1 [debugcounterorch] Add checks for supported counter types and drop reasons (#1173) 10f091d [vs tests] Disable NAT tests to unblock Jenkins (#1179) 698ca2a [portsorch] Refactor portsorch to use FlexCounterManager to setup port and queue stats (#1170) ea8b1da Merge pull request #1125 from AkhileshSamineni/natorch_changes_in_sonic bbd77fc Merge pull request #1126 from AkhileshSamineni/natsyncd_changes_in_sonic 419d9d2 Fix bug: shellquote the inner string (#1168) d507a93 [aclorch] Validate that provided IN/OUT_PORTS are physical interfaces (#1156) f24081b [AclOrch] add validation for check CRM (#1082) 85036df Merge pull request #1059 from AkhileshSamineni/nat_changes_in_sonic 1eac91e [portsorch] process PortsOrch tables in specific order (#1108) 80e01c0 Teamd :: fix for cleaning up the teamd processes correctly on teamd docker stop (#1159) 77fa5a4 Move away sairedis logrotate from signal handler (#1153) 5ad47af Merge branch 'master' into natorch_changes_in_sonic 982abd6 Fixed compilation issue after forced push. 286ca21 Fixed compilation issue. 3350b85 Added pytest testcases for NAT. 63ebb3a Added platform changes to avoid nexthop traching in OA. f9a8302 Addressed review comments. 946e53d Fixed compilation issue. 3d2b900 Addressed review comments and added 'nat miss' traps in copp json. ff2c78c Added changes to verify the NAT is supported before setting nat zone. 346f59a Addressed review comments. 8246acb Removed debug framework change to avoid compilation error f59ec87 Fixed compilation issue. e8b9dab Added aclorch changes for "do_not_nat" action. 88fe411 Orchagent changes in sonic-swss submodule to support NAT feature. 3ca530c Addressed review comments e4b7724 Addressed Review comments 72750ac Natsyncd changes in sonic-swss submodule to support NAT feature. 2086ad2 Fixed compilation issue after forced push. 3fcfea3 Fix in cleanup. 4028803 Added missed check ed405bf Addressed review comments ae6eb80 Compilation fix to pick proper ACL Table name - CFG_ACL_TABLE_TABLE_NAME d750132 Fix for string compilation issue 904e80b Corrected Makefile.am change. 5b09c41 Removed pytest related files and added those to PR #1125 fd733b8 Removed the Orchagent and Natsyncd related files as those are covered as part of PRs #1125 and #1126 b53c888 Loopback related changes for NAT. 29bb539 Added neighsync changes to support warm reboot for NAT. 4e0d0a1 Changes in sonic-swss submodule to support NAT feature. Signed-off-by: Stepan Blyschak <[email protected]>

…t and queue stats (sonic-net#1170) - Updates portsorch to use FlexCounterManager for port and queue stats instead of direct redis operations - Updates FlexCounterManager to allow clients to enable the group in the constructor, for convenience - Updates the makefile for cfgmgr to include new flex_counter dependency from portsorch Signed-off-by: Danny Allen <[email protected]>

What I did: Moved the SAI header to v1.8.1. 7cd3a7ed84db3fc9cec13496a5339b6fe1888bb7 (HEAD, tag: v1.8.1, origin/v1.8) Update SAI version to V1.8.1 (sonic-net#1218) 5913e4cdd0c9c7ae859baa2e18086327b39a94da Fix error when compiling Broadcom SAI with v1.8.0 (sonic-net#1216) 5a98bc3c7e86c01f3cf702054f9af7c7c5ca6daf (HEAD, tag: v1.8.0, origin/master, origin/HEAD, master) Update version to 1.8.0 (sonic-net#1207) b3244ceceb45184ffe37da55bb9a98ef126050ce saineighbor.h: Updated SAI_NEIGHBOR_ENTRY_ATTR_ENCAP_INDEX and deprecated SAI_NEIGHBOR_ENTRY_ATTR_ENCAP_IMPOSE_INDEX (sonic-net#1202) 8731ca6e09c7ba99b0b009e5821d80598e216756 Add source/dest/double NAPT entry available attributes (sonic-net#1194) f053d899feb9517f2db43ee462589a30572b5ed1 Add switch attributes for hash offset configuration. (sonic-net#1195) 13e5cd6940f9a0da1878d00f08e5941e09f16e7f PRBS RX State Data Type (sonic-net#1179) 9755845a06525a3c17f03e7b936a70783e8ef068 Packet header based VRF classification (sonic-net#1185) 2369ecb59fff1a5cae948d41eea06bf8b71330b2 SAI versioning (sonic-net#1183) 744279839c176e68b19734657975e3f5ec6f1a32 Replaced SAI_SWITCH_ATTR_MACSEC_OBJECT_ID with SAI_SWITCH_ATTR_MACSEC_OBJECT_LIST (sonic-net#1199) 584c724864fe565357e82d097ddcc7363bddefac [CI] Set up CI&PR with Azure Pipelines (sonic-net#1200) 08192237963174cc60edae9b4812a39c43b291fd Add attribute to query available packet DMA pool size (sonic-net#1198) f092ef1e3ce695fc3f9552721025695312b961a2 Add IPv6 flow label hash attribute. (sonic-net#1192) cbc9562bb7a8f2c3a79702b99be55f3b3afa6957 Override VRF (sonic-net#1186) 1eb35afdb2146baf40e6c2b8f2f8bfe99075eaee Add SAI_SWITCH_ATTR_SWITCH_HARDWARE_INFO format for GB MDIO sysfs access (sonic-net#1171) b2d4c9a57c7f00b2632c35ca5eb3dd6480f7916a Switch scoped tunnel attributes (sonic-net#1173) 96adc95bf8316e1905143d9ecd21f32a43e80d7f Enhancements for MPLS support (sonic-net#1181) 3dcf1f2028da4060b345ad78e8a0c87d225bf5d0 Support for ACL extensions in metadata (sonic-net#1178) 24076be95b871e8f82ecaeb908cad951dc68896c [meta] Add support for allow empty list tag (sonic-net#1190) a2b3344cdde0bf5a4f8e98e1c676a658c0c615b0 spell check fixes (sonic-net#1189) bf4271bab6e8884bd96050bcba44e3134adaaec3 Do not call sai_metadata_sai get APIs before checking if they are allocated (sonic-net#1182) 5d5784dc3dbfc713c92ae7d2c472680b837bb002 [macsec]: Separate XPN configuration attribute from read-only attribute (sonic-net#1169) 6d5a9bf5ad17cb82621cabbe2449524320930606 [macsec]: add SAI_MACSEC_ATTR_SUPPORTED_CIPHER_SUITE_LIST (sonic-net#1172) e72c8f3a0cc543cb228554be82c97a63db917740 [meta] Print each tool version in Makefile (sonic-net#1177) 8f19677da88c7494d563ef7c5acb0529ecbd0b6e [meta] Add check for START, END and RANBE_BASE enums (sonic-net#1175) 24ad7906f145930b2e25682b6248909289d39e72 [meta] Create sai_switch_pointers_t struct (sonic-net#1174) 4f5f84df3fcd0e146707df41d3e2837c48f7c760 Tunnel loopback packet action as resource (sonic-net#1163) 8a0e82c57aa0e22e696158735516904e7dc14052 [meta] Add create only oid attribute check on switch object (sonic-net#1170) 14cf50772e478551920963ecf11f4fd019a0c106 Remove obsolete stub folder (sonic-net#1168) f14f406340e4f5f1b1d674f6fdd5fd861a54c877 [meta] Use safer calloc for integer overflow check (sonic-net#1166) Also this PR include changes of this sonic-net#815 SAI commit b2d4c9a57c7f00b2632c35ca5eb3dd6480f7916a Switch scoped tunnel attributes (sonic-net#1173) needed change in sai_redis_switch.cpp and sai_vs_switch.cpp for compilation. How I verify: Verify Build is fine of libsairedis*.deb, syncd*.deb, swss*.deb Co-authored-by: Ann Pokora <[email protected]>

Added PSU util for Mellanox platforms

1da68a1

lguohan reviewed Nov 20, 2017

View reviewed changes

stcheng added the Enhancement ➕ label Nov 21, 2017

jleveque reviewed Nov 21, 2017

View reviewed changes

jleveque suggested changes Nov 21, 2017

View reviewed changes

andrii savka added 2 commits November 22, 2017 16:40

Rename PsuUtil, removed unused deb repo from snmp docker

62a0d93

PSU plugin fixes for MLNX Bulldog platform

f141965

jleveque suggested changes Nov 28, 2017

View reviewed changes

qiluo-msft reviewed Nov 28, 2017

View reviewed changes

qiluo-msft requested changes Nov 28, 2017

View reviewed changes

[PSU] Fixes of review remarks

c76d7e0

jleveque suggested changes Nov 30, 2017

View reviewed changes

qiluo-msft approved these changes Nov 30, 2017

View reviewed changes

andrii savka added 2 commits December 1, 2017 12:39

[PSU] Changed TODO descriptions for dependenciesw

114346b

[PSU] Revert usage psuutil.py link instead of file because it should …

bc2945d

…be copied in SNMP as a file

jleveque approved these changes Dec 1, 2017

View reviewed changes

lguohan requested changes Dec 4, 2017

View reviewed changes

jleveque suggested changes Dec 4, 2017

View reviewed changes

lguohan approved these changes Dec 8, 2017

View reviewed changes

[PSU] Updated sonic-utilities submodule

3751083

qiluo-msft mentioned this pull request Dec 8, 2017

Added PSU util for Mellanox platforms, updated sonic-utilities submodule #1218

Merged

lguohan merged this pull request into sonic-net:master Dec 8, 2017

stepanblyschak mentioned this pull request Jan 28, 2020

[swss] update swss pointer #4077

Merged

		@@ -1,5 +1,8 @@
		FROM docker-config-engine

		RUN echo 'deb http://debian-archive.trafficmanager.net/debian jessie-backports main' >> /etc/apt/sources.list && \

		@@ -1,5 +1,7 @@
		FROM docker-config-engine

		RUN echo 'deb [arch=amd64] http://packages.microsoft.com/repos/sonic-dev/ jessie main' >> /etc/apt/sources.list

Added PSU util for Mellanox platforms, added PSU CLI to SNMP #1170

Added PSU util for Mellanox platforms, added PSU CLI to SNMP #1170

Conversation

andrii-savka commented Nov 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jleveque Nov 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

jleveque Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrii-savka Nov 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jleveque left a comment • edited Loading

Choose a reason for hiding this comment

qiluo-msft Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jleveque Nov 29, 2017 • edited Loading

Choose a reason for hiding this comment

qiluo-msft Nov 29, 2017 • edited Loading

Choose a reason for hiding this comment

qiluo-msft Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft left a comment

Choose a reason for hiding this comment

lguohan commented Dec 4, 2017

lguohan left a comment

Choose a reason for hiding this comment

andrii-savka commented Dec 4, 2017

jleveque left a comment

Choose a reason for hiding this comment

lguohan commented Dec 5, 2017

lguohan commented Dec 6, 2017

lguohan commented Dec 8, 2017

andrii-savka commented Nov 20, 2017 •

edited

Loading

jleveque Nov 21, 2017 •

edited

Loading

qiluo-msft Nov 28, 2017 •

edited

Loading

jleveque Nov 28, 2017 •

edited

Loading

andrii-savka Nov 29, 2017 •

edited

Loading

jleveque left a comment •

edited

Loading

qiluo-msft Nov 28, 2017 •

edited

Loading

qiluo-msft Nov 28, 2017 •

edited

Loading

qiluo-msft Nov 28, 2017 •

edited

Loading

qiluo-msft Nov 28, 2017 •

edited

Loading

qiluo-msft Nov 28, 2017 •

edited

Loading

jleveque Nov 29, 2017 •

edited

Loading

qiluo-msft Nov 29, 2017 •

edited

Loading

qiluo-msft Nov 28, 2017 •

edited

Loading

qiluo-msft Nov 28, 2017 •

edited

Loading

qiluo-msft Nov 28, 2017 •

edited

Loading