Unsupported Media Type (HTTP 415) when patching custom resources

[nolar]

Actual behaviour

v10.0.0 was released this night. Since then, custom resource patching is broken:

HTTP 415 "Unsupported Media Type" when patching the custom resources.

Traceback (most recent call last):
  File "kopf-134.py", line 15, in <module>
    rsp = api.patch_namespaced_custom_object(**request_kwargs)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/apis/custom_objects_api.py", line 1951, in patch_namespaced_custom_object
    (data) = self.patch_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, body, **kwargs)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/apis/custom_objects_api.py", line 2057, in patch_namespaced_custom_object_with_http_info
    collection_formats=collection_formats)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 393, in request
    body=body)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/rest.py", line 286, in PATCH
    body=body)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (415)
Reason: Unsupported Media Type
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Thu, 04 Jul 2019 11:53:38 GMT', 'Content-Length': '263'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json","reason":"UnsupportedMediaType","code":415}

When tracing step by step, the actual Content-Type used is application/json-patch+json, which should match the supported types from the error message.

There is no way to control Content-Type from the user side, i.e. when calling the patch-method.

Expected behaviour

Patching works smoothly.

It all works fine if installed as

pip install 'kubernetes<10.0.0'

Steps to reproduce:

minikube start --kubernetes-version=v1.15.0
kubectl apply -f https://raw.githubusercontent.com/zalando-incubator/kopf/master/examples/crd.yaml
kubectl apply -f https://raw.githubusercontent.com/zalando-incubator/kopf/master/examples/obj.yaml

Any other CRD should show the same behaviour (probably); this one is just for a quick example.

pip install kubernetes==10.0.0

Then a Python script:

import kubernetes

kubernetes.config.load_kube_config()  # developer's config files

request_kwargs = {
    'group': 'zalando.org',
    'version': 'v1',
    'plural': 'kopfexamples',
    'name': 'kopf-example-1',
    'namespace': 'default',
    'body': {},
}

api = kubernetes.client.CustomObjectsApi()
rsp = api.patch_namespaced_custom_object(**request_kwargs)
print(rsp)

Any body content cause the same error, event the empty body — so, I guess, it is not about the patch-content itself.

Versions

Kubernetes: 1.15.0
kubernetes==10.0.0
Python 3.7

Notes

Might be related:

• patch_namespaced_custom_object only accepts merge-patch+json patch type #862
• feat: custom objects can be merged by json-patch gen#119

[nolar]

Found the root cause:

With 9.0.0, the type was application/merge-patch+json.
With 10.0.0, it is application/json-patch+json.

The select_header_content_type() method choses the first content type from a list of available. The content is not checked:

            return content_types[0]

The CustomObjectsApi feeds it with these 2 content-types:

        header_params['Content-Type'] = self.api_client.\
            select_header_content_type(['application/json-patch+json', 'application/merge-patch+json'])

Previously (in 9.0.0), it was one:

        header_params['Content-Type'] = self.api_client.\
            select_header_content_type(['application/merge-patch+json'])

This happened because of 9c8bd4a.

And so, the default content type has suddenly changed since 9.0 to 10.0.0.

The Kubernetes API doesn't like it, because application/json-patch+json is a list, not dict (see Wikipedia):

[
    { "op": "add", "path": "/myPath", "value": ["myValue"] }
]

However, the content data sent by the users all over the world was not changed, and most likely remains a dict as per application/merge-patch+json content-type (since it worked before).

PS: With body changed from {} to [], the example from the issue description works.

[roycaihw]

/assign

[roycaihw]

Thanks for the detailed bug report. I see the select_header_content_type is not intelligent as I would expected.

kube-apiserver supports both JSON patch and JSON merge patch as content types for patching custom resources. The problem is select_header_content_type unconditionally puts the first type from the list into the HTTP header. So it used to be people using the python client can only use JSON patch as body (which was bad), but upgrading the python client suddenly changes the HTTP header to use JSON merge patch, which breaks existing users. What's more, JSON patch is no longer an option given how select_header_content_type works.

I think we should do two things:

• For short-term, we should restore the old behavior (setting JSON patch as content type) with a patch release v10.0.1. This can be achieved by excluding feat: custom objects can be merged by json-patch gen#119 and doing a release

• For long-term, we should give users the option to specify the content-type. It can be defaulted as JSON patch, but currently it's really hard for users to hijack the HTTP request and change the header. I think removing select_header_content_type is part of migrate from swagger-codegen to openapi-generator gen#93. Ref [WIP] feat: client generated by openapi-generator #738 (comment)

[roycaihw]

fixed in 10.0.1

/close

[k8s-ci-robot]

@roycaihw: Closing this issue.

In response to this:

fixed in 10.0.1

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[abdulsalama]

which version is this fix in?

I still get the same problem when patching profiles.
kubectl patch profile foo --patch ....

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3"
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-502bfb",

Any idea why I'm still seeing it?

[ableuler]

• For long-term, we should give users the option to specify the content-type. It can be defaulted as JSON patch, but currently it's really hard for users to hijack the HTTP request and change the header. I think removing select_header_content_type is part of kubernetes-client/gen#93. Ref #738 (comment)

This would be great!

Until this is implemented and for everybody coming here in search for a way to apply a JSON patch to a custom resource with content_type="application/json-patch+json", the dynamic client supports this option already. See for example here.

[retr0h]

Reading this issue, I still don't understand what needs done to actually use this ;)

I'm calling the following code with version 19.15.0 of the python bindings.

        body = [{"op": "remove", "path": "/spec/remoteWrite"}]
        response = api.patch_namespaced_custom_object(
            group="monitoring.coreos.com",                                                                                                                  version="v1",
            name="kube-prometheus-stack-prometheus",
            namespace="monitoring",                                                                                                                         plural="prometheuses",
            body=body,

However, I receive the following error:

Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '95e7d775-0b04-4674-bcba-f35bf7a9cac8', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'aa6ddeef-d18c-429c-a54e-d2be86b818b6', 'Date': 'Wed, 17 Nov 2021 01:33:09 GMT', 'Content-Length': '812'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":" \"\" is invalid: patch: Invalid value: \"[{\\\"op\\\":\\\"remove\\\",\\\"path\\\":\\\"/spec/remoteWrite\\\"}]\": couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" }","reason":"Invalid","details":{"causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"[{\\\"op\\\":\\\"remove\\\",\\\"path\\\":\\\"/spec/remoteWrite\\\"}]\": couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" }","field":"patch"}]},"code":422}

/cc @roycaihw @nolar

[jmgilman]

For future readers, the body is a JSON patch which is described here. In particular, it took me a while to figure out how to add an annotation with a / in it:

        body = [
            {
                "op": "add",
                # '~1' is an escaped '/'
                # See https://jsonpatch.com/#json-pointer for escaping rules
                "path": "/metadata/annotations/myproject.io~1archive-name",
                "value": archive.metadata.name,
            }
        ]