Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

URL encoding is performed when passing URL parameters under the rest protocol #13349

Closed
wiscloud opened this issue Nov 11, 2023 · 10 comments
Closed
Assignees
Labels
type/bug Bugs to being fixed

Comments

@wiscloud
Copy link

wiscloud commented Nov 11, 2023

image

When requesting, the sayHello method uses URL parameters and performs URL encoding, and the server directly returns a String that is also URL encoded. The second method say does not have such a problem.


请求时,sayHello方法采用了 URL 传参,进行了 URL 编码,且服务端直接返回 String 也进行了 URL 编码,第二个方法say并没有这样的问题。

@wiscloud wiscloud added the type/bug Bugs to being fixed label Nov 11, 2023
@AlbumenJ AlbumenJ changed the title rest 协议下url 传参时进行了url 编码 URL encoding is performed when passing URL parameters under the rest protocol Nov 14, 2023
@AlbumenJ
Copy link
Member

@suncairong163 PTAL

@kazetyan
Copy link
Contributor

image

image

ParamProviderParamParser 获取到参数时是否应该进行url decode 处理

image

或者在创建NettyRequestFacade 的时候进行?

image

排查的时候顺便看了下BodyProviderParamParser的实现,解码时编码要么是取系统默认的,要么是写死的utf-8(或许是个隐藏bug?)
image
image

@wiscloud
Copy link
Author

demo-bug.zip
这是写的一个示例压缩包,采用了三个子模块的形式,然后也测试过将所有的代码放到一个模块中,在一个模块中并没有出现乱码问题,但在三个子模块中就出现了乱码问题

@kazetyan
Copy link
Contributor

demo-bug.zip 这是写的一个示例压缩包,采用了三个子模块的形式,然后也测试过将所有的代码放到一个模块中,在一个模块中并没有出现乱码问题,但在三个子模块中就出现了乱码问题

image

@wiscloud 这个代码我试过了,问题还是这个,至于放到一个模块的情况,dubbo直接走了injvmInvoker,相当于本地调用了

image

image

@MajaChen
Copy link
Contributor

我也遇到了同样的问题,这个issue我先认领了,我尝试解决一下

@CrazyHZM
Copy link
Member

@MajaChen
Is there any progress?

@MajaChen
Copy link
Contributor

MajaChen commented Dec 26, 2023

@CrazyHZM 我正在努力修复,麻烦再给我两周时间,看到2024年1月7号能否解决,或者考虑放弃。谢谢!

@MajaChen
Copy link
Contributor

MajaChen commented Jan 6, 2024

In English, I've read the code very carefully, I think I've come up a proper way to fix the problem, see the pull request, it also contains necessary unit test.

cause:

  • In consumer side, parameters in RequestTemplate will be encoded with UTF-8 and will be appended to http request uri. Chinese character like '李强' in parameters will be transformed to '%E6%9D%8E%E5%BC%BA'
    (The problem occur when using subclass of RestClient like OkhttpClient or HttpClientRestClient but not URLConnectionRestClient as demonstrated in unit test in RestClientTest)

  • Then the request is sent to provider.

  • In provider side, ReuqeFascade will parse the http request and extract the parameters from the uri, but forget to decode paramters with UTF-8, so Chinese character remains undecoded state, like '%E6%9D%8E%E5%BC%BA'

solution:

We need to decode the parameters with UTF-8 in RequestFascade.initParameters. For simplicity, I decode the whole uri and I'm pretty sure is safe even the uri is not encoded with UTF-8 at all in consumer side. You can refer the unit test in NettyRequestFacadeTest.

demonstration:

tGBBeWues4
img_v2_2f58fbac-41b0-42d8-87aa-fb32282425al
BX5Jf8ALdE

@MajaChen
Copy link
Contributor

MajaChen commented Jan 6, 2024

supplement: if you use @QueryParam to annotate the parameter,the parameter will be trated as request parameter in RequestTemplate, and it can cause problmes; But if you don't, the parameter will be trated as request body, the problem won't occur.
Besides, if you use URLConnectionRestClient instead of OkhttpClient, the problem won't occur eigther.

@MajaChen
Copy link
Contributor

@CrazyHZM I've launch a pull request, can you review the code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Bugs to being fixed
Projects
Archived in project
Development

No branches or pull requests

5 participants