Loopback Interface changes for multi ASIC devices #4825

arlakshm · 2020-06-22T18:26:14Z

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan [email protected]

- Why I did it
multi ASIC will have 2 Loopback Interfaces

Loopback0 has globally unique IP address, which is advertised by the multi ASIC device to its peers.
This way all the external devices will see this device as a single device.
Loopback4096 is assigned an IP address which has a scope is within the device. Each ASIC has a different ip address for Loopback4096. This ip address will be used as Router-Id by the bgp instance on multi ASIC devices.

This PR implements this change for multi ASIC devices

- How I did it
The following changes are done

Update the minigraph parser to update the ASIC configuration with the host/device Loopback address
Update the bgp template to use Loopback1 as router-id and advertise the Loopback4096 address only to internal neighbors if its multi_npu device.
Add the route-map HIDE_INTERNAL when advertising the Loopback4096 ip address
Update the sonic-cfggen to pass the is_multi_npu function as enviornment global while rendering this templates.
Update the sonic_device_util.py to get_num_npus both on the host and inside the containers.
Added Unit tests for generating Loopback interface configurations for multi ASIC devices

- How to verify it

Verify if the Loopback interface and bgp configuration are generated properly.
Verify the internal bgp and external bgp sessions are established.

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

prsunny · 2020-06-22T18:36:43Z

dockers/docker-fpm-frr/frr/bgpd/bgpd.main.conf.j2

@@ -8,13 +8,21 @@
 ! TSA configuration
 !
 ip prefix-list PL_LoopbackV4 permit {{ get_ipv4_loopback_address(LOOPBACK_INTERFACE, "Loopback0") | ip }}/32
+{% if multi_npu()  == true %}
+  ip prefix-list PL_LoopbackV4 permit {{ get_ipv4_loopback_address(LOOPBACK_INTERFACE, "Loopback1") | ip }}/32


There are currently discussions on adding multiple loopback interfaces to Sonic and if there is a regular "Loopback1", say got added from previous config_db etc, it could result in some conflicts. My suggestion is, can we have it renamed to something specific?

@prsunny , changed from Loopback1 to Loopback4096 for ASIC's loopback interface

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

prsunny · 2020-06-24T05:37:24Z

src/sonic-config-engine/minigraph.py

@@ -859,6 +870,7 @@ def parse_xml(filename, platform=None, port_config_file=None, asic_name=None):
        else:
            if child.tag == str(QName(ns, "DpgDec")):
                (intfs, lo_intfs, mvrf, mgmt_intf, vlans, vlan_members, pcs, pc_members, acls, vni) = parse_dpg(child, asic_name)
+                host_lo_intfs = parse_host_loopback(child, hostname)


Not sure I get this change. As far as minigraph is concerned, it is yet another loopback interface, correct? Why do we need to have a special parsing for this?

Not sure I get this change. As far as minigraph is concerned, it is yet another loopback interface, correct? Why do we need to have a special parsing for this?

In the multi ASIC minigraph, each ASIC is modeled as different device. The Loopback0 interface is in the DeviceDataPlaneInfo of the device or host, while the Loopback4096 is in the DeviceDataPlaneInfo of the ASIC. Here we are trying to get Loopback0 interface when generating the configuration of the ASIC.

lguohan · 2020-06-29T20:53:59Z

dockers/docker-fpm-frr/frr/bgpd/bgpd.main.conf.j2

+{% if multi_npu() == true %}
+{% if get_ipv6_loopback_address(LOOPBACK_INTERFACE, "Loopback4096") != 'None' %}
+  address-family ipv6
+    network {{ get_ipv6_loopback_address(LOOPBACK_INTERFACE, "Loopback4096") | ip }}/64 route-map HIDE_INTERNAL


@pavel-shirshov , do we need to add this in the bgpcfgd unit test?

@lguohan yes. when it is ready.

are we advertising this loopback 4096 to the outside world?

@lguohan, the route-map HIDE_INTERNAL will prevent advertising Loopback4096 to the external devices.

prsunny

lgtm, please check other comments.

pavel-shirshov

Can you please answer my questions?

pavel-shirshov · 2020-06-30T18:25:16Z

dockers/docker-fpm-frr/frr/bgpd/bgpd.main.conf.j2

@@ -8,13 +8,21 @@
 ! TSA configuration
 !
 ip prefix-list PL_LoopbackV4 permit {{ get_ipv4_loopback_address(LOOPBACK_INTERFACE, "Loopback0") | ip }}/32
+{% if multi_npu()  == true %}
+  ip prefix-list PL_LoopbackV4 permit {{ get_ipv4_loopback_address(LOOPBACK_INTERFACE, "Loopback4096") | ip }}/32
+{% endif %}


why do you need it here?

@pavel-shirshov, I think this might be needed to support TSA for the multi-asic platforms.
Let me know if my understanding is wrong. I will remove this change

@arlakshm I think you're not going to announce your internal loopbacks at all, right? This prefix-list is used to announce loopback only from the device in TSA mode.

Removed in the latest commit

pavel-shirshov · 2020-06-30T18:25:40Z

dockers/docker-fpm-frr/frr/bgpd/bgpd.main.conf.j2

+{% if get_ipv6_loopback_address(LOOPBACK_INTERFACE, "Loopback4096") != 'None' %}
+  ipv6 prefix-list PL_LoopbackV6 permit {{ get_ipv6_loopback_address(LOOPBACK_INTERFACE, "Loopback4096") | replace('/128', '/64') | ip_network }}/64
+{% endif %}
+{% endif %}


Why do you need it here?

@pavel-shirshov, I think this might be needed to support TSA for the multi-asic platforms.
Let me know if my understanding is wrong. I will remove this change

Removed in the latest commit

pavel-shirshov · 2020-06-30T18:42:45Z

dockers/docker-fpm-frr/frr/bgpd/bgpd.main.conf.j2

 !
 !
-{% if DEVICE_METADATA['localhost']['sub_role'] == 'FrontEnd' %}
+{% if multi_npu() == true %}


I think it is possible to write {% if multi_npu() %}.

Changed in the latest commit

pavel-shirshov · 2020-06-30T18:43:48Z

dockers/docker-fpm-frr/frr/bgpd/bgpd.main.conf.j2

+{% if multi_npu() == true %}
+{% if get_ipv6_loopback_address(LOOPBACK_INTERFACE, "Loopback4096") != 'None' %}
+  address-family ipv6
+    network {{ get_ipv6_loopback_address(LOOPBACK_INTERFACE, "Loopback4096") | ip }}/64 route-map HIDE_INTERNAL


@lguohan yes. when it is ready.

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

pavel-shirshov

LGTM

arlakshm · 2020-07-01T16:23:39Z

retest this please

dockers/docker-fpm-frr/frr/bgpd/bgpd.main.conf.j2

lguohan · 2020-07-02T07:40:51Z

retest vsimage please

…4825)" This reverts commit cae65a4.

…evices (sonic-net#4825)"" This reverts commit 5f31842.

Resubmitting the changes for (#4825) with fixes for sonic-bgpcdgd test failures Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

* [brcmsai]: Updated BRCM SAI Debina package to 3.7.5.1-2 (sonic-net#4916) Fix for Copp Rules not having Policer Rate-Limit applied. Signed-off-by: Abhishek Dosi <[email protected]> * [nephos]: upgrade Nephos SAI version to c749df (sonic-net#4814) Verified with Nephos nps8365 based platform Accton AS7116-54x. * "[config]: Multi ASIC loopback changes (sonic-net#4895) Resubmitting the changes for (sonic-net#4825) with fixes for sonic-bgpcdgd test failures Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]> * [mellanox]: Support warm reboot on MSN4700 (sonic-net#4910) * [Submodule Update] Sonic-platform-common [sfp_base] Update return value documentation of channel-specific methods (#98) [SfpBase] Fix key name typo in docstring (#99) [sfp] Tweak key names of some transceiver info fields (#97) [sfputil] Make SfpUtilHelper.get_physical_to_logical noexcept as in SfpUtilBase (#96) * [Submodule update] sonic-platform-daemons [xcvrd] Update key names in 'get_media_settings_value()' (#63) [xcvrd] Tweak some transceiver info key names (#62) * [Submodule update] sonic-utilities [sfpshow][mock_state_db] Tweak key names of some transceiver info fields (sonic-net#958) [config] Fix syntax error (sonic-net#966) [config] Fix indentation level in _get_disabled_services_list() (sonic-net#965) * [Submodule Update] sonic-swss [aclorch] Use IPv6 Next Header internally for protocol number on MLNX platform (sonic-net#1343) * [Submodule Update] Add support for attribute capability query in lua script (sonic-net#362) * Cherry-pick was not clean. Fixing it. Signed-off-by: Abhishek Dosi <[email protected]> * [telemetry] Call sonic-cfggen Once (sonic-net#4901) sonic-cfggen call is slow and this is taking place in the SONiC boot up process. The change uses templates to assemble all required vars into single template file. With this change, telemetry now calls once into sonic-cfggen. signed-off-by: Tamer Ahmed <[email protected]> * [mgmt docker] move pycryptodome installation to the end of the docker building (sonic-net#4917) * [mgmt docker] move pycryptodome installation to the end of the docker building Signed-off-by: Ying Xie <[email protected]> * pin down the version to current: 3.9.8 * comment * Add support for bcmsh and bcmcmd utlitites in multi ASIC devices (sonic-net#4926) Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]> This PR has changes to support accessing the bcmsh and bcmcmd utilities on multi ASIC devices Changes done - move the link of /var/run/sswsyncd from docker-syncd-brcm.mk to docker_image_ctl.j2 - update the bcmsh and bcmcmd scripts to take -n [ASIC_ID] as an argument on multi ASIC platforms * [caclmgrd] Improve code reuse (sonic-net#4931) Improve code reuse in `generate_block_ip2me_traffic_iptables_commands()` function. * [Submodule Update] sonic-utilities Intf table migration for APP_DB entries during warmboot (sonic-net#980) [Multi NPU] Time Improvements to the config reload/load_minigraph commands (sonic-net#917) * [Submodule Update] sonic-py-swssdk [MultiDB]: use python class composition to avoid confusion in base class (#74) * [Submodule update] sonic-snmpagent. Movent to 201911 Branch with with following PR's : Implement cbgpPeer2State in CiscoBgp4MIB (#119) Fix index nodes in LLDP tables whose access right is not-accessible. (#112) Fix quagga/FRR parser on IPv6 BGP sessions (#122) [lint] Fix some syntax errors or warnings (#127) Update README.md: Add lgtm badges (#128) [Multi-asic]: Support multi-asic platform (#126) Simplify test code (#132) [Multi-asic]: Namespace support for LLDP and Sensor tables (#131) Fix undefined variable and warning message (#134) Fix SNMP AgentX socket connection timeout when using Namespace.get_all() (#140) [Namespace] Fix interfaces counters in InterfacesMIB RFC 2863 (#141) Fix LGTM reported alert of PR#141 (#142) * [bgpcfgd] - Fix a key error during delete (sonic-net#4946) Co-authored-by: abdosi <[email protected]> Co-authored-by: gracelicd <[email protected]> Co-authored-by: arlakshm <[email protected]> Co-authored-by: Stephen Sun <[email protected]> Co-authored-by: Abhishek Dosi <[email protected]> Co-authored-by: Tamer Ahmed <[email protected]> Co-authored-by: Ying Xie <[email protected]> Co-authored-by: Joe LeVeque <[email protected]> Co-authored-by: Prince Sunny <[email protected]>

…t#4825) * Loopback IP changes for multi ASIC devices multi ASIC will have 2 Loopback Interfaces - Loopback0 has globally unique IP address, which is advertised by the multi ASIC device to its peers. This way all the external devices will see this device as a single device. - Loopback4096 is assigned an IP address which has a scope is within the device. Each ASIC has a different ip address for Loopback4096. This ip address will be used as Router-Id by the bgp instance on multi ASIC devices. This PR implements this change for multi ASIC devices Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

…onic-net#4825)" This reverts commit cae65a4.

Resubmitting the changes for (sonic-net#4825) with fixes for sonic-bgpcdgd test failures Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

* [brcmsai]: Updated BRCM SAI Debina package to 3.7.5.1-2 (sonic-net#4916) Fix for Copp Rules not having Policer Rate-Limit applied. Signed-off-by: Abhishek Dosi <[email protected]> * [nephos]: upgrade Nephos SAI version to c749df (sonic-net#4814) Verified with Nephos nps8365 based platform Accton AS7116-54x. * "[config]: Multi ASIC loopback changes (sonic-net#4895) Resubmitting the changes for (sonic-net#4825) with fixes for sonic-bgpcdgd test failures Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]> * [mellanox]: Support warm reboot on MSN4700 (sonic-net#4910) * [Submodule Update] Sonic-platform-common [sfp_base] Update return value documentation of channel-specific methods (#98) [SfpBase] Fix key name typo in docstring (#99) [sfp] Tweak key names of some transceiver info fields (#97) [sfputil] Make SfpUtilHelper.get_physical_to_logical noexcept as in SfpUtilBase (#96) * [Submodule update] sonic-platform-daemons [xcvrd] Update key names in 'get_media_settings_value()' (#63) [xcvrd] Tweak some transceiver info key names (#62) * [Submodule update] sonic-utilities [sfpshow][mock_state_db] Tweak key names of some transceiver info fields (sonic-net#958) [config] Fix syntax error (sonic-net#966) [config] Fix indentation level in _get_disabled_services_list() (sonic-net#965) * [Submodule Update] sonic-swss [aclorch] Use IPv6 Next Header internally for protocol number on MLNX platform (sonic-net#1343) * [Submodule Update] Add support for attribute capability query in lua script (sonic-net#362) * Cherry-pick was not clean. Fixing it. Signed-off-by: Abhishek Dosi <[email protected]> * [telemetry] Call sonic-cfggen Once (sonic-net#4901) sonic-cfggen call is slow and this is taking place in the SONiC boot up process. The change uses templates to assemble all required vars into single template file. With this change, telemetry now calls once into sonic-cfggen. signed-off-by: Tamer Ahmed <[email protected]> * [mgmt docker] move pycryptodome installation to the end of the docker building (sonic-net#4917) * [mgmt docker] move pycryptodome installation to the end of the docker building Signed-off-by: Ying Xie <[email protected]> * pin down the version to current: 3.9.8 * comment * Add support for bcmsh and bcmcmd utlitites in multi ASIC devices (sonic-net#4926) Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]> This PR has changes to support accessing the bcmsh and bcmcmd utilities on multi ASIC devices Changes done - move the link of /var/run/sswsyncd from docker-syncd-brcm.mk to docker_image_ctl.j2 - update the bcmsh and bcmcmd scripts to take -n [ASIC_ID] as an argument on multi ASIC platforms * [caclmgrd] Improve code reuse (sonic-net#4931) Improve code reuse in `generate_block_ip2me_traffic_iptables_commands()` function. * [Submodule Update] sonic-utilities Intf table migration for APP_DB entries during warmboot (sonic-net#980) [Multi NPU] Time Improvements to the config reload/load_minigraph commands (sonic-net#917) * [Submodule Update] sonic-py-swssdk [MultiDB]: use python class composition to avoid confusion in base class (#74) * [Submodule update] sonic-snmpagent. Movent to 201911 Branch with with following PR's : Implement cbgpPeer2State in CiscoBgp4MIB (#119) Fix index nodes in LLDP tables whose access right is not-accessible. (#112) Fix quagga/FRR parser on IPv6 BGP sessions (#122) [lint] Fix some syntax errors or warnings (#127) Update README.md: Add lgtm badges (#128) [Multi-asic]: Support multi-asic platform (#126) Simplify test code (#132) [Multi-asic]: Namespace support for LLDP and Sensor tables (#131) Fix undefined variable and warning message (#134) Fix SNMP AgentX socket connection timeout when using Namespace.get_all() (#140) [Namespace] Fix interfaces counters in InterfacesMIB RFC 2863 (#141) Fix LGTM reported alert of PR#141 (#142) * [bgpcfgd] - Fix a key error during delete (sonic-net#4946) * [Submodule Update] sonic-utilities Fix the None Type Exception when Interface Table does not exist (cold boot) as part of db migration (sonic-net#986) * Fix the below frr start.sh jija2 exception in 201911 image syslog: (sonic-net#4958) File "/usr/local/bin/sonic-cfggen", line 380, in <module> main() File "/usr/local/bin/sonic-cfggen", line 354, in main print(template.render(data)) File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 1090, in render self.environment.handle_exception() File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 832, in handle_exception reraise(*rewrite_traceback_stack(source=source)) File "<template>", line 1, in top-level template code File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 471, in getattr return getattr(obj, attribute) jinja2.exceptions.UndefinedError: 'WARM_RESTART' is undefined Signed-off-by: Abhishek Dosi <[email protected]> * [Submodule update] sonic-snmpagent [201911] Fix interface counters in RFC1213 (#144) * [docker-ptf] Add support for spytest to ptf container (sonic-net#4410) - Install apt and pip dependencies - Define traffic generator service Signed-off-by: Danny Allen <[email protected]> * [arista] update Arista drivers submodules (sonic-net#4967) Merge most of the changes that recently made it to master. This will be the last such merge operation and future commits will only cherry-pick fixes and targeted features. Major fixes and features, - reboot cause enhancement with more hardware reboot cause reporting - fix reboot cause parsing issue with 201811 release - fix get_change_event logic - fix error message on missing sysfs entry by our plugins - final piece of the platform refactors for fan and sensor reporting through the platform API * [201911][devices] Update SFP keys to align with new standard (sonic-net#4976) Align SFP key names with new standard defined in sonic-net/sonic-platform-common#97 - hardwarerev -> hardware_rev - serialnum -> serial - manufacturename -> manufacturer - modelname -> model - Connector -> connector * [201911][sonic-telemetry] Update submodule (sonic-net#4987) Point submodule to new 201911 branch of sonic-telemetry and update pointer to the current HEAD of the 201911 branch * src/sonic-telemetry aaa9188...01b5365 (1): > [testdata] Update SFP keys to align with new standard (#39) * [201911][sudoers] Add `sonic_installer list` to read-only commands (sonic-net#4997) `sonic_installer list` is a read-only command. Specify it as such in the sudoers file. This will also ensure the new `show boot` command, which calls `sudo sonic_installer list` under the hood doesn't fail due to permissions. * [caclmgrd] Filter DHCP packets based on dest port only (sonic-net#4995) Co-authored-by: abdosi <[email protected]> Co-authored-by: gracelicd <[email protected]> Co-authored-by: arlakshm <[email protected]> Co-authored-by: Stephen Sun <[email protected]> Co-authored-by: Abhishek Dosi <[email protected]> Co-authored-by: Tamer Ahmed <[email protected]> Co-authored-by: Ying Xie <[email protected]> Co-authored-by: Joe LeVeque <[email protected]> Co-authored-by: Prince Sunny <[email protected]> Co-authored-by: Danny Allen <[email protected]> Co-authored-by: Samuel Angebault <[email protected]>

Loopback IP changes for multi ASIC devices

8f22526

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

arlakshm requested review from pavel-shirshov, SuvarnaMeenakshi, abdosi and lguohan June 22, 2020 18:26

prsunny reviewed Jun 22, 2020

View reviewed changes

Change loopback interface from Loopback1 to Loopback4096

ed18962

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

prsunny reviewed Jun 24, 2020

View reviewed changes

lguohan reviewed Jun 29, 2020

View reviewed changes

prsunny previously approved these changes Jun 30, 2020

View reviewed changes

pavel-shirshov reviewed Jun 30, 2020

View reviewed changes

Addressing review comments

094e52b

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

arlakshm dismissed prsunny’s stale review via 094e52b July 1, 2020 00:03

pavel-shirshov previously approved these changes Jul 1, 2020

View reviewed changes

lguohan reviewed Jul 1, 2020

View reviewed changes

dockers/docker-fpm-frr/frr/bgpd/bgpd.main.conf.j2 Outdated Show resolved Hide resolved

change multi_npu to multi_asic

9920fff

arlakshm dismissed pavel-shirshov’s stale review via 9920fff July 1, 2020 23:20

lguohan approved these changes Jul 2, 2020

View reviewed changes

arlakshm added Multi-ASIC Multi-NPU 🔢 FRR 🚥 Request for 201911 Branch labels Jul 2, 2020

arlakshm merged commit cae65a4 into sonic-net:master Jul 2, 2020

arlakshm deleted the mnpu_loopback_address_chgs branch July 2, 2020 18:37

lguohan added a commit that referenced this pull request Jul 3, 2020

Revert "[config]: Loopback Interface changes for multi ASIC devices (#…

5f31842

…4825)" This reverts commit cae65a4.

arlakshm added a commit to arlakshm/sonic-buildimage that referenced this pull request Jul 4, 2020

Revert "Revert "[config]: Loopback Interface changes for multi ASIC d…

fc5b2f4

…evices (sonic-net#4825)"" This reverts commit 5f31842.

arlakshm mentioned this pull request Jul 4, 2020

Multi ASIC loopback changes #4895

Merged

abdosi removed the Request for 201911 Branch label Jul 5, 2020

arlakshm added a commit that referenced this pull request Jul 6, 2020

"[config]: Multi ASIC loopback changes (#4895)

31baf38

Resubmitting the changes for (#4825) with fixes for sonic-bgpcdgd test failures Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

abdosi pushed a commit that referenced this pull request Jul 8, 2020

"[config]: Multi ASIC loopback changes (#4895)

aef3f7d

Resubmitting the changes for (#4825) with fixes for sonic-bgpcdgd test failures Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <[email protected]>

qiluo-msft pushed a commit to qiluo-msft/sonic-buildimage that referenced this pull request Jul 12, 2020

Revert "[config]: Loopback Interface changes for multi ASIC devices (s…

f8da3e4

…onic-net#4825)" This reverts commit cae65a4.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Loopback Interface changes for multi ASIC devices #4825

Loopback Interface changes for multi ASIC devices #4825

arlakshm commented Jun 22, 2020 •

edited

Loading

prsunny Jun 22, 2020

arlakshm Jun 22, 2020

prsunny Jun 24, 2020

arlakshm Jun 29, 2020

lguohan Jun 29, 2020

pavel-shirshov Jun 30, 2020

lguohan Jun 30, 2020

arlakshm Jun 30, 2020

prsunny left a comment

pavel-shirshov left a comment

pavel-shirshov Jun 30, 2020

arlakshm Jun 30, 2020

pavel-shirshov Jun 30, 2020

arlakshm Jul 1, 2020

pavel-shirshov Jun 30, 2020

arlakshm Jun 30, 2020

arlakshm Jul 1, 2020

pavel-shirshov Jun 30, 2020

arlakshm Jul 1, 2020

pavel-shirshov Jun 30, 2020

pavel-shirshov left a comment

arlakshm commented Jul 1, 2020

lguohan commented Jul 2, 2020

Loopback Interface changes for multi ASIC devices #4825

Loopback Interface changes for multi ASIC devices #4825

Conversation

arlakshm commented Jun 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prsunny left a comment

Choose a reason for hiding this comment

pavel-shirshov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavel-shirshov left a comment

Choose a reason for hiding this comment

arlakshm commented Jul 1, 2020

lguohan commented Jul 2, 2020

arlakshm commented Jun 22, 2020 •

edited

Loading