WebClient/MockMVC issue with jquery ajax post #25768

guisimon28 · 2020-09-14T17:05:22Z

Hello,

i am using htmlunit with spring test in order to test all ihm interface from my web application. It works fine with html form (post and get) and with ajax get but i have a problem with ajax post request.

The controller don't received the request. If i replace post by get the junit test case works fine.

this the html view

<!DOCTYPE html>
<html lang="fr" xmlns="http://www.w3.org/1999/xhtml" 
	xmlns:th="http://www.thymeleaf.org">
	<head>
		<title>Spring Html Unit</title>
        <meta charset="utf-8" />
		<meta name="format-detection" content="telephone=no">
		<meta name="viewport" content="width=device-width, initial-scale=1.0">
		<script src="https://code.jquery.com/jquery-3.5.1.min.js" integrity="sha256-9/aliU8dGd2tb6OSsuzixeV4y/faTqgFtohetphbbj0=" crossorigin="anonymous"></script>
		<script>
			function postTest() {
				$.ajax({
			        type:"post",
					data: {
						'subject' : 'subject test',
						'message' : 'message test'
					},
			        url:"[[@{/test/post}]]",
			        success: function(data, textStatus, jqXHR){
			        	$("#result").text(data);
			        }
			    });
			}		
		</script>
	</head>
	<body>
		<h4 th:text="'Hello to Thymeleaf '+${myParam}">Hello from Thymeleaf</h4>
		
		<button type="button" onClick="postTest()">Test post</button>
		<div>
			<span>result :</span><span id="result"></span>
		</div>
		
	</body>
</html>

and the controller

@Controller
public class WelcomeController {

	@GetMapping("/")
	public String init(Model model) {
		model.addAttribute("myParam", "Guillaume");
		return "welcome";
	}

	@PostMapping("/test/post")
	public @ResponseBody String post(@RequestParam String subject, @RequestParam String message, Model model) {
		return subject + " " + message;
	}
}

you can also find the complete code on my github https://github.com/guisimon28/spring-test-htmlunit

Can you help me to find if there is some missing configuration or if its a htmlunig bug or pull request ?

Thanks for All

Guillaume

bclozel · 2020-09-14T20:37:41Z

Thanks for getting in touch, but it feels like this is a question that would be better suited to StackOverflow. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements. Feel free to update this issue with a link to the re-posted question (so that other people can find it) or add some more details if you feel this is a genuine bug.

The sample you've provided doesn't use Spring Boot - I'd suggest to use Spring Boot instead to dramatically reduce the amount of boilerplate.

Also there seems to be a confusion around your usage of jQuery. Right now it's sending the form as a POST, but with an urlencoded request body. You should either switch to GET or send the body as "multipart/form-data" and use a JavaScript FormData object.

Even better, you could look into the getting started guides for Spring Boot. See https://spring.io/guides/gs/handling-form-submission/

Thanks!

guisimon28 · 2020-09-15T05:04:42Z

Hi bclozel,

i thinked it was a bug because my code works fine in a tomcat server. This code was an example to reproduce the probem i have with my real application.
With my real application jquery post works fine when my application is deployed in tomcat but not with MockMVc

it works great on chrome.
I put the application on debug in eclipse with a tomcat server.
And when i click on Test post button the result label is refresh with the content

That the reason why i think there is a difference between a real browser and the MockMvc b
So i am not agree, maybe you can try in a browser and with MockMvc to see the difference.

regards

guisimon28 · 2020-09-24T09:57:01Z

Hello,

after some investigations, we think there is a problem in HtmlUnitRequestBuilder.content() as you can see in this analyse
HtmlUnit/htmlunit#223 (comment)

mor info on this github to reproduce : https://github.com/twendelmuth/spring-test-htmlunit

can you reopen the ticket in order to correct spring test ?

regards

Guillaume

rstoyanchev · 2020-09-24T11:25:47Z

after some investigations, we think there is a problem in HtmlUnitRequestBuilder.content()

If I understand correctly, the issue is that we have a POST with "application/x-www-form-urlencoded" data, but the form data is not parsed and used to populate request parameters as Servlet containers do. Wouldn't it make more sense for HtmlUnit to do that so that anything using it (besides Spring's HtmlUnit integration) can also benefit from this improvement?

rstoyanchev · 2020-09-24T11:32:39Z

Or I guess in HtmlUnit requestParameters actually means URL query parameters rather than what the Servlet API calles "request pameters", in which case I to think it make sense for HtmlUnitRequestBuilder to do that.

Note that we already have similar logic when using MockMvc (without HtmlUnit).

rstoyanchev · 2020-09-24T11:47:27Z

It works fine with html form (post and get) and with ajax get but i have a problem with ajax post request.

@guisimon28 I'm wondering how does it work with HTML form post? I would expect that to run into the same issue. Is the server side controller in that case different?

guisimon28 · 2020-09-24T16:29:09Z

thanks for analyse my issue rstoyanchev, i can't debug today.
But the same test with html form post works fine.
All works fine with Htmlunit when i put my application on a server and start htmunit test on a remote server.

regards

guisimon28 · 2020-09-30T11:42:08Z

Still dont have time to debug do you think you can improve htmlunitrequestbuilder like mokckmvc ?

fredarene · 2020-10-02T13:30:50Z

Hi @rstoyanchev ,

I work with @guisimon28.

When we debug the passage in HtmlUnitRequestBuilder.buildRequestfor both cases, we see that :

html FORM POST : the webRequest.requestParameters_ is populated with the form's params; they're then put in MockHttpServletRequest.parameters and used "correctly"
AJAX POST : the webRequest.requestBody_ is populated with the form's params and dumped in MockHttpServletRequest.content, so it seems the params are not user afterwards, at least not as parameters of the request

rstoyanchev · 2020-10-02T14:38:46Z

Thanks @fredarene for the extra findings. It sounds like HtmlUnit is populating WebRequest.requestParameters in one case but not the other. I would think that there is a case to argue it should do the same in both cases.

Any chance you can update the sample project with both scenarios?

twendelmuth · 2020-10-03T12:18:26Z

Hi,

quoting my answer here from the HTMLUnit side. I've transformed the Spring Boot project to illustrate the issues as well

Okay I finally found some time to look into this.
You're indeed correct that the error seems to stem from the MockMvcWebClientBuilder from Spring. To be honest I'm not to familiar with the MockMvcWebClientBuilder. The problem seems to be what we're seeing here:
The FormData / RequestBody is not correctly put in to the @RequestParam parameters in the Controller.
Therefore the request fails with 400 Bad Request.
I'm having some difficulties debugging the Spring test code ... however I think the issue comes from HtmlUnitRequestBuilder.content():

This one takes the request body without any escaping and passes it to the controller:
	private void content(MockHttpServletRequest request, Charset charset) {
		String requestBody = this.webRequest.getRequestBody();
		if (requestBody == null) {
			return;
		}
		request.setContent(requestBody.getBytes(charset)); //requestBody = 'subject test message test'
		//requestBody should be: 'subject=subject+test&message=message+test' since it's form encoded.
	}
To get to the results I've transformed the project to use SpringBoot since it was easier for me to find out what's actually happening and where the error potentially is. I hope that it's running with gradle ... since I needed to convert it to a maven project because of my gradle issue 👼

The project is here: https://github.com/twendelmuth/spring-test-htmlunit
Added tests that should illustrate what's wrong: https://github.com/twendelmuth/spring-test-htmlunit/tree/master/src/test/java/spring/test/htmlunit/controller
And more specific: https://github.com/twendelmuth/spring-test-htmlunit/blob/master/src/test/java/spring/test/htmlunit/controller/WelcomeControllerMockMvc.java#L37 - this is what's happening in the end in your test.

I hope this helps :-)

rstoyanchev · 2020-10-07T17:24:58Z

@twendelmuth I wasn't looking for the example with "AJAX POST" but rather for the example with "html FORM POST" in order to understand why they work differently even though they are the same type of request. In any case I investigated and updated the HtmlUnit issue. I'll wait for the response there.

guisimon28 · 2020-10-21T17:40:19Z

@rstoyanchev

in your post on 24/09/2020, you say "Note that we already have similar logic when using MockMvc (without HtmlUnit)."

can you apply the same logic with HtmlUnit ?

regards

rstoyanchev · 2020-10-22T08:22:35Z

MockMvc simulates a client and builds the request from the ground up. By contrast HtmlUnit is a client and it is a legitimate question what HtmlUnit should do in this case. From the comment in HtmlUnit/htmlunit#223 (comment) it's clear the main question there is what does a real browser do? Therefore the best way to move this forward is to provide the answers to the questions there first so we can figure out where the fix belongs. I have not gotten to it yet.

spring-projects/spring-framework#25768

bclozel · 2020-10-22T10:22:09Z

@guisimon28 please take a look at HtmlUnit/htmlunit#223 (comment)

bclozel · 2020-10-23T11:53:10Z

This issue is superseded, since this seems to be an inconsistency in HtmlUnit's behavior (see issue linked in previous comment). I'm closing this issue for now, we can reopen it if work is required in the Spring Framework codebase.

…ameters implement WebRequest.getRequestParameters from scratch to take care of the different ways parameters are packed into requests This break the backward compatibility in some way but solves #223 and spring-projects/spring-framework#25768 (hopefully)

…amework#25768)

guisimon28 mentioned this issue Sep 14, 2020

Jquery ajax post doesn't work HtmlUnit/htmlunit#223

Closed

spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged or decided on label Sep 14, 2020

bclozel closed this as completed Sep 14, 2020

bclozel added status: invalid An issue that we don't feel is valid and removed status: waiting-for-triage An issue we've not yet triaged or decided on labels Sep 14, 2020

bclozel reopened this Sep 24, 2020

bclozel self-assigned this Sep 24, 2020

bclozel added status: waiting-for-triage An issue we've not yet triaged or decided on and removed status: invalid An issue that we don't feel is valid labels Sep 24, 2020

rstoyanchev added the status: waiting-for-feedback We need additional information before we can continue label Sep 24, 2020

rstoyanchev added this to the 5.3 RC2 milestone Sep 24, 2020

rstoyanchev assigned rstoyanchev and unassigned bclozel Sep 24, 2020

bclozel self-assigned this Sep 25, 2020

rstoyanchev modified the milestones: 5.3 RC2, 5.3 GA Oct 12, 2020

bclozel added a commit to bclozel/htmlunitrepro that referenced this issue Oct 22, 2020

Create repro project for spring-framework#25768

4b226ab

spring-projects/spring-framework#25768

bclozel removed this from the 5.3 GA milestone Oct 23, 2020

bclozel added status: invalid An issue that we don't feel is valid and removed type: enhancement A general enhancement labels Oct 23, 2020

bclozel closed this as completed Oct 23, 2020

bclozel mentioned this issue Dec 17, 2021

Working POST ajax requests sent by HtmlUnit #27832

Closed

rbri added a commit to HtmlUnit/htmlunit that referenced this issue Mar 25, 2022

better naming and document change (#223 and spring-projects/spring-fr…

602433e

…amework#25768)

rbri mentioned this issue Mar 27, 2022

Make use of new HtmlUnit API for access request parameters #28240

Closed

bclozel added status: superseded An issue that has been superseded by another for: external-project Needs a fix in external project and removed status: invalid An issue that we don't feel is valid labels Sep 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WebClient/MockMVC issue with jquery ajax post #25768

WebClient/MockMVC issue with jquery ajax post #25768

guisimon28 commented Sep 14, 2020 •

edited by sbrannen

Loading

bclozel commented Sep 14, 2020

guisimon28 commented Sep 15, 2020 •

edited

Loading

guisimon28 commented Sep 24, 2020

rstoyanchev commented Sep 24, 2020 •

edited

Loading

rstoyanchev commented Sep 24, 2020

rstoyanchev commented Sep 24, 2020

guisimon28 commented Sep 24, 2020 •

edited

Loading

guisimon28 commented Sep 30, 2020

fredarene commented Oct 2, 2020

rstoyanchev commented Oct 2, 2020 •

edited

Loading

twendelmuth commented Oct 3, 2020

rstoyanchev commented Oct 7, 2020 •

edited

Loading

guisimon28 commented Oct 21, 2020

rstoyanchev commented Oct 22, 2020 •

edited

Loading

bclozel commented Oct 22, 2020

bclozel commented Oct 23, 2020

WebClient/MockMVC issue with jquery ajax post #25768

WebClient/MockMVC issue with jquery ajax post #25768

Comments

guisimon28 commented Sep 14, 2020 • edited by sbrannen Loading

bclozel commented Sep 14, 2020

guisimon28 commented Sep 15, 2020 • edited Loading

guisimon28 commented Sep 24, 2020

rstoyanchev commented Sep 24, 2020 • edited Loading

rstoyanchev commented Sep 24, 2020

rstoyanchev commented Sep 24, 2020

guisimon28 commented Sep 24, 2020 • edited Loading

guisimon28 commented Sep 30, 2020

fredarene commented Oct 2, 2020

rstoyanchev commented Oct 2, 2020 • edited Loading

twendelmuth commented Oct 3, 2020

rstoyanchev commented Oct 7, 2020 • edited Loading

guisimon28 commented Oct 21, 2020

rstoyanchev commented Oct 22, 2020 • edited Loading

bclozel commented Oct 22, 2020

bclozel commented Oct 23, 2020

guisimon28 commented Sep 14, 2020 •

edited by sbrannen

Loading

guisimon28 commented Sep 15, 2020 •

edited

Loading

rstoyanchev commented Sep 24, 2020 •

edited

Loading

guisimon28 commented Sep 24, 2020 •

edited

Loading

rstoyanchev commented Oct 2, 2020 •

edited

Loading

rstoyanchev commented Oct 7, 2020 •

edited

Loading

rstoyanchev commented Oct 22, 2020 •

edited

Loading