Add documentation how to build and use custom/vendor auto-instrumenta…

…tion (open-telemetry#762)

* doc: customized auto-instrumentation

* doc: update based on comments

* doc: wording

* doc: update based on comments

* doc: update based on comments

README.md

@@ -215,6 +215,28 @@ The possible values for the annotation can be
 * `"my-instrumentation"` - name of `Instrumentation` CR instance.
 * `"false"` - do not inject
 
+#### Use customized or vendor instrumentation
+
+By default, the operator uses upstream auto-instrumentation libraries. Custom auto-instrumentation can be configured by
+overriding the image fields in a CR.
+
+```yaml
+apiVersion: opentelemetry.io/v1alpha1
+kind: Instrumentation
+metadata:
+  name: my-instrumentation
+spec:
+  java:
+    image: your-customized-auto-instrumentation-image:java
+  nodejs:
+    image: your-customized-auto-instrumentation-image:nodejs
+  python:
+    image: your-customized-auto-instrumentation-image:python
+```
+
+The Dockerfiles for auto-instrumentation can be found in [autoinstrumentation directory](./autoinstrumentation). 
+Follow the instructions in the Dockerfiles on how to build a custom container image.
+
 ## Compatibility matrix
 
 ### OpenTelemetry Operator vs. OpenTelemetry Collector

autoinstrumentation/java/Dockerfile

@@ -1,3 +1,7 @@
+# To build one auto-instrumentation image for Java, please:
+#  - Download your customized `javaagent.jar` to `/javaagent.jar`. This is required as when instrumenting the pod,
+#    one init container will be created to copy the jar to your app's container.
+#  - Grant the necessary access to the jar. `chmod -R go+r /javaagent.jar`
 FROM busybox
 
 ARG version

autoinstrumentation/nodejs/Dockerfile

@@ -1,3 +1,12 @@
+# To build one auto-instrumentation image for Node.js, please:
+# - Ensure the packages are installed in the `/autoinstrumentation` directory. This is required as when instrumenting the pod,
+#   one init container will be created to copy all the content in `/autoinstrumentation` directory to your app's container. Then
+#   update the `NODE_OPTIONS` environment variable accordingly. To achieve this, you can mimic the one in `autoinstrumentation/nodejs/Dockerfile`
+#   by using multi-stage builds. In the first stage, install all the required packages in one custom directory.
+#   Then in the second stage, copy the directory to `/autoinstrumentation`.
+# - Ensure you have `@opentelemetry/api`, `@opentelemetry/auto-instrumentations-node`, and `@opentelemetry/sdk-node` or your customized
+#   alternatives installed.
+# - Grant the necessary access to `/autoinstrumentation` directory. `chmod -R go+r /autoinstrumentation`
 FROM node:16 AS build
 
 WORKDIR /operator-build

autoinstrumentation/python/Dockerfile

@@ -1,3 +1,13 @@
+# To build one auto-instrumentation image for Python, please:
+# - Ensure the packages are installed in the `/autoinstrumentation` directory. This is required as when instrumenting the pod,
+#   one init container will be created to copy all the content in `/autoinstrumentation` directory to your app's container. Then
+#   update the `PYTHONPATH` environment variable accordingly. To achieve this, you can mimic the one in `autoinstrumentation/python/Dockerfile`
+#   by using multi-stage builds. In the first stage, install all the required packages in one custom directory with `pip install --target`.
+#   Then in the second stage, copy the directory to `/autoinstrumentation`.
+# - Ensure you have `opentelemetry-distro` and `opentelemetry-instrumentation` or your customized alternatives installed.
+#   Those two packages are essential to Python auto-instrumentation.
+# - Grant the necessary access to `/autoinstrumentation` directory. `chmod -R go+r /autoinstrumentation`
+
 FROM python:3.10-alpine AS build
 
 WORKDIR /operator-build