[BUG] version number is incompatible with existing ES clients #693

dlvenable · 2021-05-12T19:24:27Z

Describe the bug
The API which provides version includes a field version.number. Clients parse this value to determine what version of ElasticSearch is running, and is used for checking API compatibility.

In ElasticSearch 7.10, this value is reported as 7.10.

In OpenSearch 1.0.0, the value is 1.0.0-SNAPSHOT.

Existing clients which parse this string are unable to communicate with OpenSearch when they rely on the value of the version.

To Reproduce
Steps to reproduce the behavior:

Start ElasticSearch 7.10
curl http://localhost:9200
Read version.number.
Start OpenSearch
curl http://localhost:9200
Read version.number.

Expected behavior

Because OpenSearch uses the same API as ElasticSearch 7.10.2, the value of version.number should be 7.10.2.

Expected:

  "version" : {
    "number" : "7.10.2",

Actual:

  "version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0-SNAPSHOT",

Plugins
N/A

Screenshots
N/A

Host/Environment (please complete the following information):
N/A

Additional context

We confirmed that this is the cause of #667. It also appears that this may be causing #669.

The text was updated successfully, but these errors were encountered:

dlvenable · 2021-05-12T20:51:01Z

Since the problem is only with the number property, a possible solution would be:

Retain the number property as matching the ElasticSearch API version
Add a new property representing the OpenSearch version.

For example:

"version" : {
    "distribution" : "opensearch",
    "number" : "7.10",               // This is the ES API
    "opensearchVersion" : "1.0.0"    // This is the OpenSearch version
    ...
}

There is probably a property name to use rather than "opensearchVersion." But, the advantage of this solution is that callers can know the ElasticSearch API version and the actual OpenSearch version (if that matters to the client).

dblock · 2021-05-13T17:34:04Z

Doesn't feel right that we're retaining ES number anywhere. Are we saying that we don't want clients to have to modify anything to work against OpenSearch? Won't clients want to assert compatibility with OpenSearch 1.0?

dlvenable · 2021-05-13T18:44:40Z

Yes, I believe that clients should not have to make changes to migrate from ElasticSearch 7.10.2 to OpenSearch 1.0. We've already found that Logstash with the Logstash ES plugin can work with OpenSearch if we manually change version.number to 7.10.2.

The introduction blog post also indicates this is the case.

The Amazon OpenSearch Service APIs will be backward compatible with the existing service APIs to eliminate any need for customers to update their current client code or applications.

Clients could assert compatibility with OpenSearch by using the new distribution field which has already been added.

Additionally, the solution I proposed above may serve as a stepping stone from OpenSearch 1.x to 2.x. The change allows existing clients to work with OpenSearch 1.x without modifications (assuming other APIs are still compatible). Clients and libraries could eventually move to support OpenSearch only, or attempt to support both ElasticSearch and OpenSearch by utilizing the distribution field. Perhaps OpenSearch 2.x would remove the version.number entirely in favor of version.opensearchVersion.

For example, clients could start to write code along these lines. It would be compatible with the suggestions above and would continue to work if version.number were removed in 2.x.

if version.distribution == 'opensearch':
  if version.opensearchVersion.startsWith('1.x'):
    handleOpensearch1();
else:
  if version.number.startsWith('7.x'):
    handleElasticsearch7();

dblock · 2021-05-13T20:04:05Z

I am 💯 with you on backward compatibility, but OpenSearch didn't break backward compatibility by changing the value of version.number (the API continues to return the same fields, including number, but a value has changed to 1.0-...). Even though reverting it to 7.x seems to work around the problem, someone should figure out what the actual root cause of #667 is without that. The exception produced (java.lang.IllegalArgumentException: mapper [log.level] cannot be changed from type [keyword] to [text]) looks odd to me.

Someone here will definitely get to #667 in order, but don't let them beat you to it :)

jcgraybill · 2021-05-13T20:08:20Z

@dlvenable raises a great point that this is almost certainly going to be more than a logstash issue: a lot of clients that interact with OpenSearch are going to check the version # to know the wire protocol to use. The proposed solution seems really elegant: keep version.number at 7.10.2 which reflects the API compatibility and lets current clients work without changing, and have a separate openSearchVersion identifier for new clients.

dblock · 2021-05-13T20:54:17Z

@jcgraybill I think I agree, but I do worry that this could lead to users having broken software more often as a production surprise.

Either way, we need to contribute integration tests against OpenSearch to https://github.com/logstash-plugins/logstash-output-elasticsearch. It currently runs against multiple versions of ES.

dlvenable · 2021-05-14T14:25:40Z

@dblock , Thanks for engaging on this.

I'm unsure exactly what issues you might be imagining. Are they problems with automated tools expecting specific behavior from specific versions? Or more of a human-related issue?

With the latter, I could see an issue with logging. Code like log.info("ElasticSearch version {}", version.number) would log out ElasticSearch version 7.9.2. That could lead to confusion for human operators.

It is likely a fair assumption that automated clients which parse the version mostly care about the major version, and maybe sometimes care about the minor version. The patch version is probably ignored.

Perhaps OpenSearch could have an alternate patch value? Some possible ideas are:

Format the OpenSearch version into the patch value. Something like MMmmpp (M=major, m=minor, p=patch).
Start at some large version and auto-increment from there
To support a 2-digit only patch, start at a smaller patch version and auto-increment from there

OpenSearch Version	Formatted	Auto-Increment	Auto-Increment w/ 2 Digits
1.0.0	7.9.10000	7.9.1001	7.9.51
1.0.1	7.9.10001	7.9.1002	7.9.52
1.0.2	7.9.10002	7.9.1003	7.9.53
1.1.0	7.9.10100	7.9.1004	7.9.54
2.0.0	8.0.20000	8.0.1000	8.0.51

Now that log line about could read ElasticSearch version 7.9.10000.

nknize · 2021-05-14T16:11:34Z

Let's be careful not to engineer a ton of cosmetic surgery just to treat a secondary symptom. Doing so risks causing other problems. First things first, the backwards compatibility tests are still disabled; so we absolutely need to re-enable those in order to catch any/all other areas affected by the version rebase. (/cc @harold-wang) That's being worked now.

As for this particular symptom, the root cause is similar to the "opensearch node fails to join" issue - /cc @shwetathareja. In short, since the server side transport layer is "version aware" of the client we can detect what version of the client is attempting to connect and "mask" the OpenSearch version to predecessor 7.10.2 appropriately. This is why we've been super thoughtful to have the 1.0.0 release not introduce any breaking changes from 7.10.2.

nknize · 2021-05-14T16:16:46Z

Perhaps OpenSearch could have an alternate patch value?

There is already a patch value in the semantic versioning system. (e.g., 1.0.0 is the 0 patch version of Major 1, Minor 0 release).

Are you suggesting a "feature" or "nightly" number? This has been brought up before (in the context of plugins) and I think it's worth discussing as a separate topic independent of this particular bug because I think it has merit. Would you like to open a separate discuss issue?

dlvenable · 2021-05-14T20:59:29Z

@nknize , The issue you referenced for "OpenSearch node fails to join" mentions a "backward compatibility mode" solution. Is the idea that OpenSearch would report version.number as 7.10.2 when that flag is set?

Regarding the patch idea, this was an extension of my original proposal. Rather than returning 7.9.2 as the version, we could have a different patch value so that human-operators can clearly see that it is OpenSearch. I offered it as a suggestion to @dblock 's concern:

I do worry that this could lead to users having broken software more often as a production surprise.

dlvenable · 2021-05-14T21:07:15Z

To elaborate on the purpose of this bug, we found that Logstash and Filebeats both fail to load templates when running against OpenSearch 1.0.0.

In local experiments, we modified MainResponse to return a hardcoded value:

.field("number", "7.10.2")

After this change, Logstash and Filebeats can run against OpenSearch.

It does appear that this is the cause of #706 as well. I did not create the exact setup that author included. However, both of us saw this same error in OpenSearch 1.0.0.

couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception"

Again, that manual change to MainResponse was sufficient to get past the error.

Smithx10 · 2021-05-15T02:02:58Z

@dlvenable,

Just chiming in to say thanks, and this worked in our testing. #706 (comment)

nknize · 2021-05-15T02:28:01Z

Is the idea that OpenSearch would report version.number as 7.10.2 when that flag is set?

No. I'm saying use the versioning system as it is intended (for backward compatibility) and fix MainResponse to return the predecessor version for Legacy Clients which wasn't covered in the initial patch that adds the distribution key-value to the MainResponse to identify OpenSearch builds from other forks.

In local experiments, we modified MainResponse to return a hardcoded value:

No hardcoded values. It should be based on the client version. But as I mentioned, I wouldn't treat this symptom just yet while Backwards Compatibility testing is disabled. This could be "patching" one issue and causing another. This "fix" I'm suggesting is on hold until we can smoke test w/ bwc tests enabled.

dblock · 2021-05-17T17:27:01Z

@harold-wang or @nknize, are you working on this?

nknize · 2021-05-17T17:35:44Z

@harold-wang or @nknize, are you working on this?

First PR opened

dlvenable · 2021-05-18T14:33:49Z

@nknize , As I understand PR #708, this will support the internal node-to-node communication inside an ElasticSearch cluster.

I created this ticket mostly to support external clients. Also, #667 and #706 relate to Logstash and Filebeats. How would ElasticSearch determine the client version in order to change the reported version number?

nknize · 2021-06-03T17:04:48Z

Let's not overthink this hack. It's a stopgap solution for 1.0+ forward compatibility with existing legacy clients to prevent users from having to mass upgrade.

The distribution change I suggested might not be necessary

It's not necessary but what your suggesting isn't a bad idea to spoof MainResponse exactly like a legacy cluster. This could be done in a follow up PR / "enhancement".

my instinct is to make that value consistent across all routes

/cat API uses the transport layer. Changing it breaks nodes joining the cluster so we don't want to do this.

I have mixed feelings to be allowing any random version number in opensearch.yml, but I can see how it can be convenient if we only want to change the number in the REST API.

Yes. This approach is a forward compatibility hack for MainResponse content until a compatibility module can do this "correctly". I would suggest that compatibility module provide an endpoint to return the spoofed version information instead of :9200 returning a lie.

Also, what is the plan to make all these clients understand OpenSearch distribution?

If OpenSearch sticks with Elastic clients then compatibility going forward is an unknown. Case in point: the new java client which Elastic intends to replace the existing Java High Level rest client. At some point here (preferably before 2.0) OpenSearch will be forced to fork the clients to ensure compatibility with a diverging core.

Having a REST API would be convenient as it would not require cluster restarts.

Rolling upgrades are required to deploy OpenSearch anyway.

dblock · 2021-06-03T17:42:08Z

Assigned to @mch2

mch2 · 2021-06-03T22:07:06Z

Put up a draft here for some initial comments

liangxibing · 2021-06-07T23:32:48Z

Hi, do we have any idea when we will have this released?
Just tried latest opensearch docker image (with tag 1.0.0-rc1), still return below version number:

{
  "name" : "6d33cb208961",
  "cluster_name" : "es-cluster",
  "cluster_uuid" : "Y034pkVIRxq7E4DtbPH6Kg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0-rc1",
    "build_type" : "tar",
    "build_hash" : "26d579287f50bb33e17c8fe1f05ea208d5c64d1f",
    "build_date" : "2021-05-28T18:18:49.848386Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  }
}

And when logstash started, still report below errors:

logstash            | [2021-06-07T23:08:25,184][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://es:9200/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:317:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:304:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:399:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:303:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:388:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/template_manager.rb:31:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch.rb:426:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch.rb:274:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:137:in `block in setup_after_successful_connection'"]}

Logstash image: docker.elastic.co/logstash/logstash:7.12.1

CEHENKLE · 2021-06-08T04:35:26Z

@liangxibing Hi Simon -- PR 814 wasn't ready in time to make this release, so we cut it without it. Once it's merged, we'll figure out how to get it a bundle out that includes it, preferably before GA. I'm anxious to see it out, too!

/C

dlvenable · 2021-06-08T14:05:47Z

The upcoming change should certainly help allow easy migration to using OpenSearch with existing clients.

I also want to bring up the topic of how compatibility will be expressed going forward. Once an ES administrator flips the switch off, clients will need to examine at the version.distribution and version.number to determine how to communicate with the OpenSearch cluster. This means clients must have built-in mappings from each OpenSearch version to compatible ElasticSearch versions.

For example, clients will need to know OpenSearch 1.0 is compatible with ElasticSearch 7.10, OpenSearch 2.0 is compatible with ElasticSearch 8.0 (hypothetically), OpenSearch 3.0 is compatible with ElasticSearch 8.5 (again, hypothetical), or whatever it is.

I'm hoping to open the discussion on how what support OpenSearch should provide for advertising ElasticSearch compatibility with clients. OpenSearch could make it easier for clients to know which ElasticSearch version the current OpenSearch version is compatible with. One possible solution is adding a new property named version.compatibility_number which would have the 7.10.2.

For example, curl http://localhost:9200 would have:

"version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0-SNAPSHOT",
    "compatibility_number" : "7.10"

This can make it much easier for clients to support both ElasticSearch and OpenSearch going forward.

I think one big question is whether OpenSearch will retain compatibility with ElasticSearch or not. If not, then something like this makes little sense. But if so, then this could be a good way to encourage clients to write code for both ElasticSearch and OpenSearch.

dblock · 2021-06-08T16:16:20Z

I think one big question is whether OpenSearch will retain compatibility with ElasticSearch or not. If not, then something like this makes little sense. But if so, then this could be a good way to encourage clients to write code for both ElasticSearch and OpenSearch.

OpenSearch is derived from Elasticsearch. It's a fork at 7.10.2, so it's only compatible with ES until then. I expect ES to diverge. Clients can decide whether they want to support OpenSearch or ES, or both.

We tried to answer some of these questions in the upgrading FAQ. If we didn't do a good job, maybe bring more questions up? Try to answer them in a PR and we can discuss!

dlvenable · 2021-06-08T21:27:05Z

Thank you for clarifying on the current expectations for compatibility.

shwetathareja · 2021-06-09T02:31:57Z

We tried to answer some of these questions in the upgrading FAQ. If we didn't do a good job, maybe bring more questions up? Try to answer them in a PR and we can discuss!

@dblock FAQ link is broken

dblock · 2021-06-09T11:36:02Z

~~@shwetathareja Works for me... where does it land you? https://opensearch.org/faq/#c3~~

Looks like the website had a snafu overnight, opensearch-project/project-website#126 (comment)

dlvenable · 2021-06-23T20:47:11Z

@dblock , @nknize
I want to bring up the idea of the distribution property again so that clients can know how to handle version going forward.

With the merge of #847 , if the setting is activated, then clients should get this:

"version" : {
    "distribution" : "opensearch",
    "number" : "7.10.2",

I believe that we want our clients to write code that can support all of these possibilities: ElasticSearch 7, OpenSearch 1 with version compatibility, and OpenSearch 1 without version compatibility. This helps with the migration path to OpenSearch 1 without version compatibility.

Currently, when in version compatibility mode, the client will have to add some short-term hack that also assumes OpenSearch 7 is the same as OpenSearch 1. Clients can write cleaner, more-robust code if we just remove the distribution value, or give it some other constant name, like version-compatible-opensearch. Then, OpenSearch 1 with version compatibility is treated like ElasticSearch 7 by the client.

dblock · 2021-06-24T16:01:22Z

I see your point @dlvenable, we're going to have a problem when we release OpenSearch 7.10.2. I am tempted to say that we should remove distribution=opensearch for this, but I am worried that there will be other side effects, WDYT @nknize @mch2?

nknize · 2021-06-24T17:00:44Z

I think by the time OpenSearch 7.10.2 is released there will be a long tail divergence from the Elasticsearch codebase that this "compatibility" issue will be a distant memory. But this is a good point.

The distribution key-value was added as a way of tracking compatibility and "forked" distributions. It's a new field in OpenSearch that legacy clients (forward compatibilty) will not even be aware of so there should be no issue. Nevertheless, for completeness we could simply not set this field when running in "compatibility" mode so that MainResponse looks exactly like the legacy response.

dblock · 2021-06-24T17:51:31Z

I opened #880

dlvenable · 2021-06-24T21:59:25Z

Yes, because ElasticSearch does not have the distribution property, that is why I think we can remove it in version compatibility mode. Ticket #880 is what I suggested and should help clients write code supporting different scenarios. Thanks!

newzubakhin · 2021-07-07T20:54:47Z

Are you going to release a new RC in July with the fix? My team can't use filebeat in production

dblock · 2021-07-07T21:09:14Z

A cluster setting was introduced in #847. We removed the distribution in backwards-compatible mode in #898. Set compatibility.override_main_response_version to put the cluster into a configuration that returns 7.10.2 as with ES OSS.

I am going to close this because the code is done. I added an issue to document it in opensearch-project/documentation-website#72.

dblock · 2021-07-07T21:09:59Z

Are you going to release a new RC in July with the fix? My team can't use filebeat in production

This will be in 1.0 GA, shortly.

dlvenable added Beta bug Something isn't working untriaged labels May 12, 2021

CEHENKLE mentioned this issue May 13, 2021

[BUG] Logstash cannot submit events to OpenSearch since version change #667

Closed

CEHENKLE added Priority-Immediate Severity-Critical and removed untriaged labels May 13, 2021

nknize added the backwards-compatibility label May 14, 2021

nknize assigned nknize and harold-wang May 14, 2021

This was referenced May 15, 2021

Filebeat mappings bug #706

Closed

[Performance] Potential regression in 1.0.0-Beta1 #673

Closed

nknize mentioned this issue May 17, 2021

[BUG] fix MainResponse to spoof version number for legacy clients #708

Merged

dblock assigned mch2 and unassigned nknize and harold-wang Jun 3, 2021

dblock mentioned this issue Jun 7, 2021

Please allow fake-changing the version number for the sake of compatibility. #822

Closed

mch2 mentioned this issue Jun 15, 2021

Add MainResponse version override as a cluster setting. #847

Merged

5 tasks

dblock mentioned this issue Jun 24, 2021

Remove distribution from main response in compatibility mode #880

Closed

dblock mentioned this issue Jul 7, 2021

Document override_main_response_version opensearch-project/documentation-website#72

Closed

dblock closed this as completed Jul 7, 2021

chris-brabender mentioned this issue Apr 7, 2022

2.4.4 Support (#2494) - Adding base OpenSearch compatibility checks Smile-SA/elasticsuite#2516

Merged

This was referenced Jun 17, 2022

mapper predecoder.timestamp error on 4.3.3 wazuh/wazuh-docker#674

Closed

ElasticSearch Refusing Template Mapping wazuh/wazuh#6395

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] version number is incompatible with existing ES clients #693

[BUG] version number is incompatible with existing ES clients #693

dlvenable commented May 12, 2021 •

edited

Loading

dlvenable commented May 12, 2021

dblock commented May 13, 2021 •

edited

Loading

dlvenable commented May 13, 2021 •

edited

Loading

dblock commented May 13, 2021

jcgraybill commented May 13, 2021

dblock commented May 13, 2021 •

edited

Loading

dlvenable commented May 14, 2021 •

edited

Loading

nknize commented May 14, 2021

nknize commented May 14, 2021

dlvenable commented May 14, 2021

dlvenable commented May 14, 2021

Smithx10 commented May 15, 2021

nknize commented May 15, 2021 •

edited

Loading

dblock commented May 17, 2021

nknize commented May 17, 2021

dlvenable commented May 18, 2021

nknize commented Jun 3, 2021

dblock commented Jun 3, 2021

mch2 commented Jun 3, 2021

liangxibing commented Jun 7, 2021 •

edited

Loading

CEHENKLE commented Jun 8, 2021

dlvenable commented Jun 8, 2021

dblock commented Jun 8, 2021 •

edited

Loading

dlvenable commented Jun 8, 2021

shwetathareja commented Jun 9, 2021

dblock commented Jun 9, 2021 •

edited

Loading

dlvenable commented Jun 23, 2021

dblock commented Jun 24, 2021 •

edited

Loading

nknize commented Jun 24, 2021

dblock commented Jun 24, 2021

dlvenable commented Jun 24, 2021

newzubakhin commented Jul 7, 2021

dblock commented Jul 7, 2021

dblock commented Jul 7, 2021

[BUG] version number is incompatible with existing ES clients #693

[BUG] version number is incompatible with existing ES clients #693

Comments

dlvenable commented May 12, 2021 • edited Loading

dlvenable commented May 12, 2021

dblock commented May 13, 2021 • edited Loading

dlvenable commented May 13, 2021 • edited Loading

dblock commented May 13, 2021

jcgraybill commented May 13, 2021

dblock commented May 13, 2021 • edited Loading

dlvenable commented May 14, 2021 • edited Loading

nknize commented May 14, 2021

nknize commented May 14, 2021

dlvenable commented May 14, 2021

dlvenable commented May 14, 2021

Smithx10 commented May 15, 2021

nknize commented May 15, 2021 • edited Loading

dblock commented May 17, 2021

nknize commented May 17, 2021

dlvenable commented May 18, 2021

nknize commented Jun 3, 2021

dblock commented Jun 3, 2021

mch2 commented Jun 3, 2021

liangxibing commented Jun 7, 2021 • edited Loading

CEHENKLE commented Jun 8, 2021

dlvenable commented Jun 8, 2021

dblock commented Jun 8, 2021 • edited Loading

dlvenable commented Jun 8, 2021

shwetathareja commented Jun 9, 2021

dblock commented Jun 9, 2021 • edited Loading

dlvenable commented Jun 23, 2021

dblock commented Jun 24, 2021 • edited Loading

nknize commented Jun 24, 2021

dblock commented Jun 24, 2021

dlvenable commented Jun 24, 2021

newzubakhin commented Jul 7, 2021

dblock commented Jul 7, 2021

dblock commented Jul 7, 2021

dlvenable commented May 12, 2021 •

edited

Loading

dblock commented May 13, 2021 •

edited

Loading

dlvenable commented May 13, 2021 •

edited

Loading

dblock commented May 13, 2021 •

edited

Loading

dlvenable commented May 14, 2021 •

edited

Loading

nknize commented May 15, 2021 •

edited

Loading

liangxibing commented Jun 7, 2021 •

edited

Loading

dblock commented Jun 8, 2021 •

edited

Loading

dblock commented Jun 9, 2021 •

edited

Loading

dblock commented Jun 24, 2021 •

edited

Loading