OT Collector trace exporter #405

hectorhdzg · 2020-02-11T01:27:22Z

Current implementation use OpenCensus receiver available in Collector, this will need to be updated eventually to use OT receiver when is ready, and Python compiled files for Proto are available once proto definition is final.

Fixes #343

…ector

hectorhdzg · 2020-02-11T01:27:48Z

WIP: Currently working on unit tests, docs and E2E validation

Fixing lint

codecov-io · 2020-02-12T00:12:28Z

Codecov Report

Merging #405 into master will increase coverage by 1.12%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #405      +/-   ##
==========================================
+ Coverage   88.11%   89.23%   +1.12%     
==========================================
  Files          41       41              
  Lines        2078     2090      +12     
  Branches      238      237       -1     
==========================================
+ Hits         1831     1865      +34     
+ Misses        173      156      -17     
+ Partials       74       69       -5

Impacted Files	Coverage Δ
opentelemetry-api/src/opentelemetry/util/loader.py	`81.25% <ø> (ø)`	⬆️
...try-ext-wsgi/src/opentelemetry/ext/wsgi/version.py	`100% <100%> (ø)`	⬆️
.../src/opentelemetry/ext/opentracing_shim/version.py	`100% <100%> (ø)`	⬆️
...src/opentelemetry/ext/opentracing_shim/__init__.py	`95.93% <100%> (ø)`	⬆️
...sts/src/opentelemetry/ext/http_requests/version.py	`100% <100%> (ø)`	⬆️
...y-ext-flask/src/opentelemetry/ext/flask/version.py	`100% <100%> (ø)`	⬆️
...app/src/opentelemetry_example_app/flask_example.py	`100% <100%> (ø)`	⬆️
...ts/src/opentelemetry/ext/http_requests/__init__.py	`89.47% <100%> (ø)`	⬆️
...ntelemetry-api/src/opentelemetry/trace/__init__.py	`85.62% <100%> (ø)`	⬆️
...emetry-sdk/src/opentelemetry/sdk/trace/__init__.py	`91.17% <100%> (+0.33%)`	⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9cac0c2...3df18a4. Read the comment docs.

Removing unnecessary files

mauriciovasquezbernal

Thanks for working on this!
I have some comments, most of them are nits, I think reviewing all the details would take too long.

How do you try it?, I followed the instructions in the readme and run the collector with docker-compose:

mvb@asus ~/.../docker $ docker-compose up
Starting docker_collector_1 ... done
Attaching to docker_collector_1
collector_1  | {"level":"info","ts":1582205460.2118967,"caller":"service/service.go:341","msg":"Starting OpenTelemetry Contrib Collector...","Version":"latest","GitHash":"a9f4526","NumCPU":4}
collector_1  | {"level":"info","ts":1582205460.2120395,"caller":"service/service.go:157","msg":"Setting up own telemetry..."}
collector_1  | {"level":"info","ts":1582205460.2133625,"caller":"service/telemetry.go:111","msg":"Serving Prometheus metrics","port":8888}
collector_1  | {"level":"info","ts":1582205460.213417,"caller":"service/service.go:190","msg":"Loading configuration..."}
collector_1  | {"level":"info","ts":1582205460.2144334,"caller":"service/service.go:198","msg":"Applying configuration..."}
collector_1  | {"level":"info","ts":1582205460.215752,"caller":"builder/exporters_builder.go:239","msg":"Exporter is enabled.","exporter":"logging"}
collector_1  | {"level":"info","ts":1582205460.2158055,"caller":"service/service.go:249","msg":"Starting exporters..."}
collector_1  | {"level":"info","ts":1582205460.2158175,"caller":"builder/exporters_builder.go:80","msg":"Exporter is starting...","exporter":"logging"}
collector_1  | {"level":"info","ts":1582205460.2159908,"caller":"builder/exporters_builder.go:85","msg":"Exporter started.","exporter":"logging"}
collector_1  | {"level":"info","ts":1582205460.2163672,"caller":"builder/pipelines_builder.go:177","msg":"Pipeline is enabled.","pipelines":"traces"}
collector_1  | {"level":"info","ts":1582205460.2163959,"caller":"service/service.go:262","msg":"Starting processors..."}
collector_1  | {"level":"info","ts":1582205460.216422,"caller":"builder/pipelines_builder.go:47","msg":"Pipeline is starting...","pipeline":"traces"}
collector_1  | {"level":"info","ts":1582205460.2164388,"caller":"builder/pipelines_builder.go:57","msg":"Pipeline is started.","pipeline":"traces"}
collector_1  | {"level":"info","ts":1582205460.216514,"caller":"builder/receivers_builder.go:210","msg":"Receiver is enabled.","receiver":"opencensus","datatype":"traces"}
collector_1  | {"level":"info","ts":1582205460.2165282,"caller":"service/service.go:274","msg":"Starting receivers..."}
collector_1  | {"level":"info","ts":1582205460.216537,"caller":"builder/receivers_builder.go:63","msg":"Receiver is starting...","receiver":"opencensus"}
collector_1  | {"level":"info","ts":1582205460.2166831,"caller":"builder/receivers_builder.go:68","msg":"Receiver started.","receiver":"opencensus"}
collector_1  | {"level":"info","ts":1582205460.216698,"caller":"service/service.go:167","msg":"Everything is ready. Begin running and processing data."}

But when I run the example and don't see anything. According to the configuration it should log the spans on the collector's terminals. Am I missing something?

Another point, is the name of the package the right one? opentelemetry-ext-collector is better?

examples/basic_tracer/README.md

ext/opentelemetry-ext-otcollector/README.rst

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/trace_exporter/__init__.py

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/util.py

examples/basic_tracer/docker/docker-compose.yaml

ext/opentelemetry-ext-otcollector/tests/test_otcollector_exporter.py

mauriciovasquezbernal · 2020-02-20T14:57:29Z

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/trace_exporter/__init__.py

+        collector_span = trace_pb2.Span(
+            name=trace_pb2.TruncatableString(value=span.name),
+            kind=utils.get_collector_span_kind(span.kind),
+            trace_id=span.context.trace_id.to_bytes(16, "big"),


Out of curiosity: Why "big"?

There is no particular reason, is there any issue with it?

I want to know if the collector expects the traces in a particular format, it is possible that a collection system is not able to assemble a full trace if we use the wrong trace id here.

https://github.com/census-instrumentation/opencensus-proto/blob/master/src/opencensus/proto/trace/v1/trace.proto#L41 there are no details about that, @owais s this something you know?

It shouldn't matter as long as the trace ID is consistent in all spans for a trace but I still suggest to test it out end to end. Generate a trace in python, export using this lib and check how the collector interprets it. You can use the file exporter to write the received spans to a file

exporters: file: path: ./filename.json service: pipelines: traces: receivers: [opencensusreceiver] exporters: [file]

This absolutely matters. Traces can span multiple systems, multiple OpenTelemetry implementations/languages, even systems not using OpenTelemetry at all but e.g. OpenTracing+jaeger to report to the same back end.

The "problem" here is that we use integers in Python to represent the trace ID, which is semantically a byte array. If we get an incoming trace ID like "4bf92f3577b34da6a3ce929d0e0e4736" then I think it is clear that trace_id[0] == 0x4b and trace_id[15] == 0x36 must be the case. Thus, "big" sounds correct.

I configured an http server example to use the collector exporter and a client to use Jaeger, when "big" is used the trace is correctly assembled, so "big" should be the right choice.

However I'm not sure this is true in all cases, so probably we want to keep an eye on this int <-> bytes <-> string conversions.

When we read the trace ID from the wire we assume big-endianness, so "big" is right here unless we want to reverse it on the way out.

codeboten

Nice, I tested this and got it working with your instructions. One note, it might be helpful to add jaeger to the list of exporters in the collector config to allow users to see something at the end of the example

  jaeger_grpc:
    endpoint: jaeger-all-in-one:14250

The docker-compose could be updated to do something like this:

version: "2"
services:

  # Collector
  collector:
    image: omnition/opentelemetry-collector-contrib:latest
    command: ["--config=/conf/collector-config.yaml", "--log-level=DEBUG"]
    volumes:
      - ./collector-config.yaml:/conf/collector-config.yaml
    ports:
      - "55678:55678"
    networks:
      - basic

  jaeger-all-in-one:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14268"
      - "14250"
    networks:
      - basic

ext/opentelemetry-ext-otcollector/README.rst

ext/opentelemetry-ext-otcollector/setup.cfg

ext/opentelemetry-ext-otcollector/tests/test_otcollector_exporter.py

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/trace_exporter/__init__.py

mauriciovasquezbernal · 2020-02-20T18:50:35Z

After some chat with @codeboten I was able to use it. It turns out that I was using a protobuf version less than 3.8.0. @hectorhdzg could you please check what is the minimum version required?

hectorhdzg · 2020-02-20T23:04:54Z

Thanks for reviewing @codeboten and @mauriciovasquezbernal, most comments must be resolved now please let me know if you have more feedback, @mauriciovasquezbernal regarding package name, let me know if you have strong opinions about it, collector word is pretty generic and could apply to a lot of stuff that is why I added the otcollector to avoid confusion.

…g/opentelemetry-python into hectorhdzg/otcollector

codeboten · 2020-02-20T23:34:13Z

ext/opentelemetry-ext-otcollector/README.rst

+        # optional:
+        # endpoint="myCollectorUrl:55678",
+        # service_name="test_service",
+        # host_name="http://localhost",


should this have a protocol prefixing the host name?

Updated, is actually expecting machine/container name

https://github.com/census-instrumentation/opencensus-proto/blob/master/src/opencensus/proto/agent/common/v1/common.proto#L58

codeboten · 2020-02-20T23:37:19Z

Nice, thanks for addressing my comments @hectorhdzg, just added one more question. It would be good to get clarity on @mauriciovasquezbernal's question on the byteorder, I wasn't able to dig anything regarding this in the collector repo.

Oberon00 · 2020-02-14T14:37:28Z

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/trace_exporter/__init__.py

+                8, "big"
+            )
+
+        if span.context.trace_state is not None:


I think this is not possible.

I believed I triggered this one with a manually created Span in a unit test, same as others checks in this method I added if x instead of if x is not None.

Oberon00 · 2020-02-14T14:39:42Z

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/trace_exporter/__init__.py

+            for (key, value) in span.context.trace_state.items():
+                collector_span.tracestate.entries.add(key=key, value=value)
+
+        if span.attributes is not None:


This cannot happen:

opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py

Line 173 in 72862c9

self.attributes = Span.empty_attributes

Maybe you wan to use if span.attributes instead, but I guess the check can simply be omitted.

I have multiple "if" checks without the "is not None" for all collections now, I guess is safer to check even if we expect this value to be there.

Oberon00 · 2020-02-14T14:41:46Z

ext/opentelemetry-ext-otcollector/setup.cfg

+packages=find_namespace:
+install_requires =
+    grpcio >= 1.0.0, < 2.0.0
+    opencensus-proto >= 0.1.0, < 1.0.0


This is an OpenCensus exporter? You write OT exporter in the title.

Well is in fact exporting to OT Collector through OpenCensus receiver, OT receiver will be ready in several weeks I added more details in first comment in the PR, we will need to revisit this one and add code to handle OT receiver using OT proto

FYI I spent some time trying to make it work with OT proto building the protobuf files myself then realizing the receiver is not there yet, people are interested in having this ready so decided to take same approach as JS SDK and support it through OpenCensus receiver, once OT receiver is ready hopefully changes only affect the span transformation and some other small pieces of code

Oberon00 · 2020-02-21T10:01:36Z

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/trace_exporter/__init__.py

+        collector_span = trace_pb2.Span(
+            name=trace_pb2.TruncatableString(value=span.name),
+            kind=utils.get_collector_span_kind(span.kind),
+            trace_id=span.context.trace_id.to_bytes(16, "big"),


This absolutely matters. Traces can span multiple systems, multiple OpenTelemetry implementations/languages, even systems not using OpenTelemetry at all but e.g. OpenTracing+jaeger to report to the same back end.

The "problem" here is that we use integers in Python to represent the trace ID, which is semantically a byte array. If we get an incoming trace ID like "4bf92f3577b34da6a3ce929d0e0e4736" then I think it is clear that trace_id[0] == 0x4b and trace_id[15] == 0x36 must be the case. Thus, "big" sounds correct.

mauriciovasquezbernal

Just some nits, but it looks good to me.

About the docker folder, do you think we can move it to a more generic place?, also, would it make sense to update other examples to use this exporter as well?

examples/basic_tracer/docker/docker-compose.yaml

ext/opentelemetry-ext-otcollector/tests/test_otcollector_exporter.py

hectorhdzg · 2020-02-21T19:45:17Z

Thanks for reviewing @mauriciovasquezbernal regarding examples, I believe we need to some organization in there with better folder structure and using multiple exporters like you mentioned, I don't want to focus to much about that in this PR for smoother reviews but definitely a good idea

c24t

Comments from running the examples, still need to check the exporter logic.

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/version.py

tox.ini

ext/opentelemetry-ext-otcollector/setup.cfg

examples/basic_tracer/docker/docker-compose.yaml

examples/basic_tracer/tracer.py

ext/opentelemetry-ext-otcollector/README.rst

Co-Authored-By: Chris Kleinknecht <[email protected]>

…g/opentelemetry-python into hectorhdzg/otcollector

c24t

One comment about the exporter version number, but the exporter logic and tests LGTM.

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/util.py

c24t · 2020-02-27T23:22:51Z

Nice to see this in the collector logs!

{"level":"info","ts":1582845653.7891443,"caller":"loggingexporter/logging_exporter.go:36","msg":"TraceExporter","type":"logging","name":"logging","#spans":3}

Rename TracerSource to TracerProvider (open-telemetry#441) Following discussion in open-telemetry#434, align the name with the specification. Co-authored-by: Chris Kleinknecht <[email protected]> Fix new ext READMEs (open-telemetry#444) Some of the new ext packages had ReStructuredText errors. PyPI rejected the uploads for these packages with: HTTPError: 400 Client Error: The description failed to render for 'text/x-rst'. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/ Adding attach/detach methods as per spec (open-telemetry#429) This change updates the Context API with the following: - removes the remove_value method - removes the set_current method - adds attach and detach methods Fixes open-telemetry#420 Co-authored-by: Chris Kleinknecht <[email protected]> Make Counter and MinMaxSumCount aggregators thread safe (open-telemetry#439) OT Collector trace exporter (open-telemetry#405) Based on the OpenCensus agent exporter. Fixes open-telemetry#343 Co-authored-by: Chris Kleinknecht <[email protected]> API: Renaming TraceOptions to TraceFlags (open-telemetry#450) Renaming TraceOptions to TraceFlags, which is the term used to describe the flags associated with the trace in the OpenTelemetry specification. Closes open-telemetry#434 api: Implement "named" meters + Remove "Batcher" from Meter constructor (open-telemetry#431) Implements open-telemetry#221. Also fixes open-telemetry#394. Stateful.py and stateless.py in metrics example folder are not changed to use the new loader in anticipation of open-telemetry#422 being merged first and removing them. Lastly, moves InstrumentationInfo from trace.py in the sdk to utils. Prepare to host on readthedocs.org (open-telemetry#452) sdk: Implement observer instrument (open-telemetry#425) Observer instruments are used to capture a current set of values at a point in time [1]. This commit extends the Meter interface to allow to register an observer instrument by pasing a callback that will be executed at collection time. The logic inside collection is updated to consider these instruments and a new ObserverAggregator is implemented. [1] https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/api-metrics.md#observer-instruments sdk: fix ConsoleSpanExporter (open-telemetry#455) 19d573a ("Add io and formatter options to console exporter (open-telemetry#412)") changed the way spans are printed by using write() instead of print(). In Python 3.x sys.stdout is line-buffered, so the spans were not being printed to the console at the right timing. This commit fixes that by adding an explicit flush() call at the end of the export function , it also changes the default formatter to include a line break. To be precise, only one of the changes was needed to solve the problem, but as a matter of completness both are included, i.e, to handle the case where the formatter chosen by the user doesn't append a line break. jaeger: Usage README Update for opentelemetry-ext-jaeger (open-telemetry#459) Usage docs for opentelemetry-ext-jaeger need to be updated after the change to `TracerSource` with v0.4. Looks like it was partially updated already. Users following the usage docs will currently run into this error: `AttributeError: 'Tracer' object has no attribute 'add_span_processor'` api: Implementing Propagators API to use Context (open-telemetry#446) Implementing Propagators API to use Context. Moving tracecontexthttptextformat to trace/propagation, as TraceContext is specific to trace rather that broader context propagation. Using attach/detach for wsgi and flask extensions, enabling activation of the full context rather that activating of a sub component such as traces. Adding a composite propagator. Co-authored-by: Mauricio Vásquez <[email protected]>

hectorhdzg added 5 commits January 7, 2020 14:31

Initial commit

d6aa116

Blocked on proto python compiled files

83267f6

WIP

1c79ae6

Merge remote-tracking branch 'upstream/master' into hectorhdzg/otcoll…

444334c

…ector

WIP

2bab1cc

hectorhdzg added WIP Work in progress tracing exporters labels Feb 11, 2020

hectorhdzg requested a review from a team February 11, 2020 01:27

hectorhdzg added 4 commits February 11, 2020 10:52

Adding protobuf dependency

bd7a021

Adding more checks in tests

e36cf3e

Fixing lint

Removing pypy3 because if is not supported for GRPC

809a27c

Removing pypy3 Collector env from tox

3c769e1

Adding readme details

f90526d

Removing unnecessary files

hectorhdzg changed the title ~~OT Collector exporter~~ OT Collector trace exporter Feb 13, 2020

Oberon00 changed the title ~~OT Collector trace exporter~~ [WIP] OT Collector trace exporter Feb 14, 2020

Adding sample

242704a

hectorhdzg added needs reviewers PRs with this label are ready for review and needs people to review to move forward. and removed WIP Work in progress labels Feb 19, 2020

hectorhdzg changed the title ~~[WIP] OT Collector trace exporter~~ OT Collector trace exporter Feb 19, 2020

Fix lint

3c794df

mauriciovasquezbernal reviewed Feb 20, 2020

View reviewed changes

codeboten approved these changes Feb 20, 2020

View reviewed changes

Addressing comments

3a66e47

hectorhdzg and others added 2 commits February 20, 2020 15:06

Merge branch 'master' into hectorhdzg/otcollector

744b372

Fix lint

402e6d6

Merge branch 'hectorhdzg/otcollector' of https://github.com/hectorhdz…

4c48178

…g/opentelemetry-python into hectorhdzg/otcollector

codeboten reviewed Feb 20, 2020

View reviewed changes

Oberon00 reviewed Feb 21, 2020

View reviewed changes

mauriciovasquezbernal approved these changes Feb 21, 2020

View reviewed changes

Addressing comments

a4d93b9

Merge branch 'master' into hectorhdzg/otcollector

9cac0c2

hectorhdzg added PR:please merge This PR is ready to be merged by a Maintainer (has enough valid approvals, successful build, etc.) and removed needs reviewers PRs with this label are ready for review and needs people to review to move forward. labels Feb 24, 2020

c24t added 3 commits February 26, 2020 15:06

Merge branch 'master' into hectorhdzg/otcollector

bb43c70

Add missing init file

2c9c018

Use TracerSource in OTC example

dc1274e

c24t reviewed Feb 26, 2020

View reviewed changes

hectorhdzg and others added 3 commits February 26, 2020 16:20

Update tox.ini

8f5bbe9

Co-Authored-By: Chris Kleinknecht <[email protected]>

Addressing comments

f7feccb

Merge branch 'hectorhdzg/otcollector' of https://github.com/hectorhdz…

64250e0

…g/opentelemetry-python into hectorhdzg/otcollector

hectorhdzg removed the PR:please merge This PR is ready to be merged by a Maintainer (has enough valid approvals, successful build, etc.) label Feb 27, 2020

c24t approved these changes Feb 27, 2020

View reviewed changes

ext/opentelemetry-ext-otcollector/src/opentelemetry/ext/otcollector/util.py Outdated Show resolved Hide resolved

Adding exporter version

3df18a4

c24t merged commit a756492 into open-telemetry:master Feb 28, 2020

hectorhdzg deleted the hectorhdzg/otcollector branch March 24, 2020 22:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OT Collector trace exporter #405

OT Collector trace exporter #405

hectorhdzg commented Feb 11, 2020

hectorhdzg commented Feb 11, 2020

codecov-io commented Feb 12, 2020 •

edited

Loading

mauriciovasquezbernal left a comment

mauriciovasquezbernal Feb 20, 2020

hectorhdzg Feb 20, 2020 •

edited

Loading

mauriciovasquezbernal Feb 20, 2020

hectorhdzg Feb 20, 2020

owais Feb 21, 2020

Oberon00 Feb 21, 2020 •

edited

Loading

mauriciovasquezbernal Feb 21, 2020

mauriciovasquezbernal Feb 21, 2020

c24t Feb 27, 2020

codeboten left a comment

mauriciovasquezbernal commented Feb 20, 2020

hectorhdzg commented Feb 20, 2020

codeboten Feb 20, 2020

hectorhdzg Feb 21, 2020

codeboten commented Feb 20, 2020

Oberon00 Feb 14, 2020

hectorhdzg Feb 21, 2020

Oberon00 Feb 14, 2020

hectorhdzg Feb 21, 2020

Oberon00 Feb 14, 2020

hectorhdzg Feb 21, 2020 •

edited

Loading

hectorhdzg Feb 21, 2020

Oberon00 Feb 21, 2020 •

edited

Loading

mauriciovasquezbernal left a comment

hectorhdzg commented Feb 21, 2020

c24t left a comment

c24t left a comment

c24t commented Feb 27, 2020

OT Collector trace exporter #405

OT Collector trace exporter #405

Conversation

hectorhdzg commented Feb 11, 2020

hectorhdzg commented Feb 11, 2020

codecov-io commented Feb 12, 2020 • edited Loading

Codecov Report

mauriciovasquezbernal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hectorhdzg Feb 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Oberon00 Feb 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codeboten left a comment

Choose a reason for hiding this comment

mauriciovasquezbernal commented Feb 20, 2020

hectorhdzg commented Feb 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codeboten commented Feb 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hectorhdzg Feb 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Oberon00 Feb 21, 2020 • edited Loading

Choose a reason for hiding this comment

mauriciovasquezbernal left a comment

Choose a reason for hiding this comment

hectorhdzg commented Feb 21, 2020

c24t left a comment

Choose a reason for hiding this comment

c24t left a comment

Choose a reason for hiding this comment

c24t commented Feb 27, 2020

codecov-io commented Feb 12, 2020 •

edited

Loading

hectorhdzg Feb 20, 2020 •

edited

Loading

Oberon00 Feb 21, 2020 •

edited

Loading

hectorhdzg Feb 21, 2020 •

edited

Loading

Oberon00 Feb 21, 2020 •

edited

Loading