Jquery ajax post doesn't work #223

guisimon28 · 2020-09-03T15:05:47Z

Hello,

i am using htmlunit with spring test in order to test all ihm interface from my web application. It works fine with html form (post and get) and with ajax get but i have a problem with ajax post request.

The controller don't received the request. If i replace post by get the junit test case works fine.

this the html view

<html lang="fr" xmlns="http://www.w3.org/1999/xhtml" 
	xmlns:th="http://www.thymeleaf.org">
	<head>
		<title>Spring Html Unit</title>
        <meta charset="utf-8" />
		<meta name="format-detection" content="telephone=no">
		<meta name="viewport" content="width=device-width, initial-scale=1.0">
		<script src="https://code.jquery.com/jquery-3.5.1.min.js" integrity="sha256-9/aliU8dGd2tb6OSsuzixeV4y/faTqgFtohetphbbj0=" crossorigin="anonymous"></script>
		<script>
			function postTest() {
				$.ajax({
			        type:"post",
					data: {
						'subject' : 'subject test',
						'message' : 'message test'
					},
			        url:"[[@{/test/post}]]",
			        success: function(data, textStatus, jqXHR){
			        	$("#result").text(data);
			        }
			    });
			}		
		</script>
	</head>
	<body>
		<h4 th:text="'Hello to Thymeleaf '+${myParam}">Hello from Thymeleaf</h4>
		
		<button type="button" onClick="postTest()">Test post</button>
		<div>
			<span>result :</span><span id="result"></span>
		</div>
		
	</body>
</html>

and the controller

@Controller
public class WelcomeController {

	@GetMapping("/")
	public String init(Model model) {
		model.addAttribute("myParam", "Guillaume");
		return "welcome";
	}

	@PostMapping("/test/post")
	public @ResponseBody String post(@RequestParam String subject, @RequestParam String message, Model model) {
		return subject + " " + message;
	}
}

you can also find the complete code on my github https://github.com/guisimon28/spring-test-htmlunit

Can you help me to find if there is some missing configuration or if its a htmlunig bug or pull request ?

Thanks for All

Guillaume

The text was updated successfully, but these errors were encountered:

twendelmuth · 2020-09-11T21:30:32Z

This issue is jQuery related / Spring related.

What happens:

jQuery.post('somewhere', type: post, data ...) will pass the parameters in the requestBody. get will add th data as query parameters (since GETs usually don't come with bodies).

guisimon28 · 2020-09-14T17:06:34Z

Hi twendelmuth,

i create right now and new issue on spring test github
spring-projects/spring-framework#25768

twendelmuth · 2020-09-14T20:07:51Z

Okay that’s not what I meant
All frameworks work correctly. You got get your code to run by using @RequestBody in Spring for example
Or you pass the parameters as queryParameters in jQuery.

The issue is that your code expects query parameters but a POST like your a sending it with jQuery will not contain those. In the end the call fails because of validation.

guisimon28 · 2020-09-15T05:37:45Z

Ok twendelmuth, can you suggest what can i change to work with HtmlUnit ?
I am surprised because this code works well when the application is deployed on a tomcat server

twendelmuth · 2020-09-15T10:45:38Z

So you do have a different setup that you're actually testing than the one you provided in the test project? I don't think that the code you provided works in a real browser 👼

The easiest way to fix it would be to change the url in the ajax() method and append your params as query params. Then it's working for GET & POST requests.

			        url:"[[@{/test/post}]]?subject=subject+test&message=message+test",

guisimon28 · 2020-09-15T12:31:08Z

twendelmuth, it works great on chrome. you can test it ;)
I put the application on debug in eclipse with a tomcat server.
And when i click on Test post button the result label is refresh with the content

That the reason why i think there is a difference between a real browser and the MockMvc but spring test team dont think

twendelmuth · 2020-09-19T00:42:46Z

Okay I finally found some time to look into this.
You're indeed correct that the error seems to stem from the MockMvcWebClientBuilder from Spring. To be honest I'm not to familiar with the MockMvcWebClientBuilder. The problem seems to be what we're seeing here:
The FormData / RequestBody is not correctly put in to the @RequestParam parameters in the Controller.
Therefore the request fails with 400 Bad Request.
I'm having some difficulties debugging the Spring test code ... however I think the issue comes from HtmlUnitRequestBuilder.content():

This one takes the request body without any escaping and passes it to the controller:

	private void content(MockHttpServletRequest request, Charset charset) {
		String requestBody = this.webRequest.getRequestBody();
		if (requestBody == null) {
			return;
		}
		request.setContent(requestBody.getBytes(charset)); //requestBody = 'subject test message test'
		//requestBody should be: 'subject=subject+test&message=message+test' since it's form encoded.
	}

To get to the results I've transformed the project to use SpringBoot since it was easier for me to find out what's actually happening and where the error potentially is. I hope that it's running with gradle ... since I needed to convert it to a maven project because of my gradle issue 👼

The project is here: https://github.com/twendelmuth/spring-test-htmlunit
Added tests that should illustrate what's wrong: https://github.com/twendelmuth/spring-test-htmlunit/tree/master/src/test/java/spring/test/htmlunit/controller
And more specific: https://github.com/twendelmuth/spring-test-htmlunit/blob/master/src/test/java/spring/test/htmlunit/controller/WelcomeControllerMockMvc.java#L37 - this is what's happening in the end in your test.

I hope this helps :-)

vkandrotas · 2020-09-23T08:59:55Z

@guisimon28 as workaround we are using request rewriting in AjaxController, which reads queryparams from requestbody and sets it to requestparams list.

guisimon28 · 2020-09-24T16:43:46Z

hi vkandrotas,

can you explain this workaround because it's not working in my case ?

fredarene · 2020-10-02T13:19:19Z

Hi,

I work with @guisimon28.

@twendelmuth, by testing the post request with html form POST submission ("FORM") and by ajax POST submission ("AJAX") I see this in HtmlUnitRequestBuiler.buildRequest() in debug :
"FORM" : this.webRequest contains the parameters in its requestParameters_ property, and nothing in requestBody_.
"AJAX" : this.webRequest contains nothing in requestParameters_ and contains the parameters in requestBody_ (which will be copied, like you said, in the MockHttpServletRequest's content property, and I think won't be used ever as parameters of the request afterwards)

So it seems that we could do something in content() as our parameters are in requestBody_, but maybe the difference in processing happens before that, when in case of "AJAX" the webRequest has the parameters in requestBody_, while they're in requestParameters_ when "FORM".

Unfortunately, I know nothing of spring-test or HtmlUnit architecture...

rbri · 2020-10-03T12:03:16Z

@fredarene not sure i got your point but from my point of view we should use the behavior of real browsers as reference. If HtmlUnit sends different requests its a bug.

twendelmuth · 2020-10-03T12:20:07Z

@rbri the issue is on the Spring Testing side from my pov.

I've seen that the spring issue is open again & provided my answer from here there as well.
As previously stated: the issue is only in the Mocking part of the Spring application - would the application run in an non mocked enviroment everything would be fine.

rstoyanchev · 2020-10-07T17:18:13Z

@rbri if I understand correctly the requestParameters property of WebRequest is meant to contain name-value pairs for a form submission. So for example when the content is FormData, requestParameters are populated accordingly.

However when the form is posted like this:

function postTest() {
	$.ajax({
        type:"post",
		data: {
			'subject' : 'subject test',
			'message' : 'message test'
		},
        url:"[[@{/test/post}]]",
        success: function(data, textStatus, jqXHR){
        	$("#result").text(data);
        }
    });
}

it ends up in the request body which seems inconsistent since both are POST "application/x-www-form-urlencoded" requests with form data as content but one comes through with requestParameters while the other with a requestBody.

The reason it works when running with Tomcat is because Servlet containers parse the request body and populate HttpServletRequest request parameters and we can also do the same in the MockMvc integration for HtmlUnit but again it seems a little inconsistent that HtmlUnit treats those differently.

I just want to double check is this intentional or a possible issue?

rbri · 2020-10-11T13:13:13Z

@rstoyanchev

HtmlUnit mimics the browser behavior as close as possible - if the current behavior is different from real browsers then it is a bug.
Request parameters are processed differently for GET (as part of the url) and POST (encoded in the body) requests. Based on this the current behavior might be correct.
Your sample is a plain ajax (XMLHttpRequest) request - and the jQuery docu states for the data parameter: 'Data to be sent to the server. If the HTTP method is one that cannot have an entity body, such as GET, the data is appended to the URL.'

From this i think the current implementation in HtmlUnit might be correct - but as always i'm not 100% sure.

It will be great if you can run your sample

using Charles Web Proxy or Fiddler
once with a real browser and save the request
once with HtmlUnit and again save the request

If there is a difference in the way the data are sent it will be helpful to strip down the jQuery sample to a plain javascript sample code.
In any case - if there is a difference we will try to eliminate this.

rbri · 2020-10-16T21:54:58Z

Any news here?

guisimon28 · 2020-10-17T19:24:01Z

still waiting an update from spring-test

rstoyanchev · 2020-10-18T17:52:43Z

@guisimon28 what needs to happen is to answer the questions above. I haven't been able to get to it but anyone can do that.

bclozel · 2020-10-22T10:18:03Z

@rbri @guisimon28

I don't think the problem is about a behavior difference with browsers, as the request seems to be sent correctly.
This is more about an inconsistent behavior when parsing it as a WebRequest.

Spring Framework, for its @RequestParam support, looks at the WebRequest#getRequestParameters. Depending on how a form/request is sent, inputs can be parsed as request parameters, or not:

with an HTML form, inputs are present in WebRequest#getRequestParameters
with an multipart/form-data ajax post, inputs are present in WebRequest#getRequestParameters
with an form url encoded ajax get inputs are present in WebRequest#getRequestParameters
with an form url encoded ajax post, inputs are not present in WebRequest#getRequestParameters

The following repro project shows the difference of behavior: https://github.com/bclozel/htmlunitrepro

We'd like to get the opinion from the HtmlUnit team about this.
Is this the expected behavior? What's the expected content of WebRequest#getRequestParameters and in which case?

Thanks!

rbri · 2020-10-22T11:18:36Z

Many thanks for the details, will have a look at this.

rbri · 2020-10-22T18:17:01Z

Looks like this inconsistency makes no real sense. I like to write some HtmlUnit internal test for the various cases.
Will inform about the progress here.

rbri · 2020-10-28T19:07:44Z

@bclozel, @rstoyanchev, @guisimon28, @twendelmuth

Hi all,

have done some debugging. My findings so far:

The HtmlUnit WebRequest is a bit different from the WebRequest from the servlet API. There is no magic to detect POST request parameters from the body and make them available as request parameters; it is more low level. But the other way around works, you can set the request parameters of an WebRequest and if the request is a post request, the parameters are encoded in the body

For the sample code (jQuery) the call itself is already not symetric.

if you switch the jQuery ajax request in the ajax.html sample to GET the processing is different. jQuery will call the open method with '/submit?subject=subject%20test&message=message%20test' as url param. The later send() call gets no content
if you switch the jQuery ajax request in the ajax.html sample to POST jQuery will call the open method with '/submit' as url param. The later send() call gets 'subject=subject+test&message=message+test' as content.

Still not sure how to proceed.
What do you think? Any ideas?

bclozel · 2020-11-04T09:58:12Z

This issue is not about making HtmlUnit consistent with the Servlet API or Servlet container behavior. The repro project is indeed using different HTTP calls and they're not equivalent.

I guess we're asking the HtmlUnit team to clarify the meaning of WebRequest#getRequestParameters.

Are those query parameters only? In this case, they should not parse the request body (in the case of HTML POST forms nor multipart/form-data ajax calls).
Are they meant to be like Servlet request parameters? In this case, the "form url encoded ajax post" is inconsistent as containers do parse this type of request body.
Is it some concept specific to HtmlUnit, as a way to make things easier for users and not parse the request body themselves? In this case, this should be explained more clearly. Also, the "form url encoded ajax post" case is inconsistent still, since users need to parse the request body there, unlike other cases. If it's an expected behavior, It should be called out in the documentation as well.

At the Spring level, we'll adapt to the team's decision.
We just thought that we should make that inconsistency point as the current API all lead us to think (for a very long time!) that this case was properly covered.

rbri · 2020-11-04T19:42:09Z

You are right, WebRequest#getRequestParameters or even the whole WebRequest class requires clarification.

At the moment this is the (internal) way to construct a web request. From this point of view the interesting part are the setter methods. You can call setRequestParameter or setBody. There is some protection to make sure you are not overwriting data. Later on the internal machine is able to construct a valid Http request out of this. The machine uses the getter methods to determine what to do. Or to sum it up, there is some convenience in using the setters but not in the getters.

The thinking behind the interface is more like a bean - getRequestParametes returns the value you have set before using setRequestParameter. This has nothing to do with the way the parameters are later encoded in the Http request.

I think the major difference here is the different point of thinking. The WebRequest in HtmlUnit is the client point of view. As a client i like to create a WebRequest - usually i only like to say something like this: create a web request using this method and this parameters / this body.
There is later no need to ask the request for something because i have created the request - i know the request already.

Your point of view is more like a server - you are called by the request and like to get the details out of the request. This is more or less the same work done by the translation process from the WebRequest into an HttpRequest.
Should i provide some links to the code that does this?

Will try to improve the javadoc also during the next days.

If you like to have a decision:
I think all your points are valid - at least the inconvenience and the inconsistent of the api in this place. But because of my limited time i like to stay with the current api.
If is is not possible for you to live with the current situation i will try to do some changes here.

rbri · 2020-11-04T19:53:42Z

And by the way PR's are always welcome.

guisimon28 · 2020-12-11T21:05:54Z

Hi Rbri, how can we help you ?

guisimon28 · 2021-02-18T17:04:24Z

Hi Rbri, some developpers on my team can help you to find a fix. So how can we help you ?

rbri · 2021-02-18T17:36:43Z

Hi @guisimon28

If you like to have a decision:
I think all your points are valid - at least the inconvenience and the inconsistent of the api in this place. But because of my limited time i like to stay with the current api.
If is is not possible for you to live with the current situation i will try to do some changes here.

So far my understanding was that i do not have to change it in HtmlUnit and the fix will by done in spring.
But i'm open for any PR's here.
And if your really have some resources to contribute i think adding the support for the Fetch API will be a great step forward.

thuri · 2022-01-02T10:12:55Z

Happy new year everyone :-)

I have a similar issue with htmlunit wenn posting a FormData object to a MockMvc Client.
I still think this is an issue with htmlunit as the jquery docs say

Data to be sent to the server. If the HTTP method is one that cannot have an entity body, such as GET, the data is appended to the URL.

see jquery docs

Therefore the content of the "data" attribute in the $.ajax call in the example MUST be added to the request body if the method used is "POST". If the method is "GET" the data must be appended to the URL. This is what I would expect from any browser.

But we have to accept that, the url in the ajax call could still contain other url params. I think that is, way the jquery docs state that in case of "GET" the "data" contant is APPENDED to the URL.

So in htmlunit (in my opinion in com.gargoylesoftware.htmlunit.javascript.host.xml.XMLHttpRequest) we need to evaluate the http method and set WebRequest.setRequestBody or WebRequest.setRequestParameters or both. Whereas in the last case we need to decide which values have to be set where.

Also we need to determine the enctype to send, as there is multipart/form-data (only for POST requests i think) or x-www-form-urlencoded

I'll try to have a look into this today but I'm not sure how far I'll get.

Just in case it's not already known to the readers of this:
@RequestParam in Spring can be used for all parameters either in the URL (aka query Paramerters) or the Body

In Spring MVC, "request parameters" map to query parameters, form data, and parts in multipart requests. This is because the Servlet API combines query parameters and form data into a single map called "parameters", and that includes automatic parsing of the request body.

see java doc of RequestParam

But that is on the server side, htmlunit and it's MockMvc "bridge" org.springframework.test.web.servlet.htmlunit.HtmlUnitRequestBuilder must not have a notion of "request parameters". On the client side we need to have a clear distinction between parameters in the query part/search-path of a http-URL and parameters in the body whose format is specifically defined by the content-type/enctype sent to the server

added another "default" case into the prepareRequestContext method for POST, PATH and PUT requests. If no other condition holds and before putting the content into the request body we now check if the encoding of the request is "application/x-www-form-urlencoded", parse the content and put the values into the request parameters. Those request parameters are later used by Springs RequestParamMethodArgumentResolver if htmlunit is running with MockMvc under the hood. This is necessary as there is no parsing of the body involved in Spring MockMvc. The implementation now behaves as it would if the javascript code passed a FormData instance as content. This should also work when htmlunit runs on apaches http client as HttpWebConnection will use the requestParameters for POST requests and recreate the urlencoded form body. The HttpWebConnection was also changed to do this for PATCH and PUT also if the encoding is application/x-www-form-urlencoded WIP: as I wasn't able to run all the tests yet

thuri · 2022-01-02T22:44:12Z

After looking deeper into this I changed my mind a bit.
When using htmlunit on top of Springs MockMvc there is no parsing of the body. Actually the RequestParamMethodArgumentResolver is using the Request object that was created by HtmlUnitRequestBuilder to access the request params.
So i changed the XmlHttpRequest object of htmlunit to fill the requestParameters object as it would do it in case that the javascript code passed a FormData object in the ajax call. (see the commits in my fork)

The fork is still work in progress as I wasn't sure which tests to run and which tests to update.
If someone could give me a hint to this I will add a test and den open a Merge Request.

rbri · 2022-01-03T07:39:06Z

@thuri can you please add a separate issue for your case; if you like we can have a talk about it later on today. Maybe you can suggest a timeslot (my email is in the pom).

thuri · 2022-01-03T18:17:52Z

Hi @rbri I'll send you a mail but just for all others: There is no issue with posting FormData objects via mockMvc.
I had another issue with my own code and somehow thought it had to do with the post request. Not sure what happened but than i found myself searching the web and found this issue and thought it looked like my problem 😉 So I eventually made it my problem 🤣

jcaillabet · 2022-01-05T13:50:03Z

Hi @thuri and happy new year every one.

I've tried using HtmlUnit sources with your correction ( JQueryAJAXFormURLEncoded branch ), and on my side my issue is corrected. Now I can send a POST form request with Ajax ( and without my Spring PR )

If finally the correction must be done in HtmlUnit, I think I will not try to develop one as yours is correct for my needs, and I will wait that you do a PR, so thank you :)

thuri · 2022-01-05T16:47:48Z

Hi, I had a nice little chat with @rbri yesterday. He'll have a deeper look into this and come back to me with some help. I'll create a PR then if still necessary but probably it won't be necessary anymore Am 5. Januar 2022 14:50:15 MEZ schrieb jcaillabet ***@***.***>:

…

Hi @thuri and happy new year every one. I've tried using HtmlUnit sources with your correction ( JQueryAJAXFormURLEncoded branch ), and on my side my issue is corrected. Now I can send a POST form request with Ajax ( and without my Spring PR ) If finally the correction must be done in HtmlUnit, I think I will not try to develop one as yours is correct for my needs, and I will wait that you do a PR, so thank you :) -- Reply to this email directly or view it on GitHub: #223 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

rbri · 2022-03-25T09:06:26Z

Again two months later... Sorry
Your fix breaks some other tests, i have a different solution in mind now .- lets cross our fingers.

…ameters implement WebRequest.getRequestParameters from scratch to take care of the different ways parameters are packed into requests This break the backward compatibility in some way but solves #223 and spring-projects/spring-framework#25768 (hopefully)

rbri · 2022-03-25T11:59:05Z

@bclozel @thuri - there is a new snapshot 2.61.0-SNAPSHOT. Please try....

…amework#25768)

thuri · 2022-03-26T09:43:13Z

Hi @rbri , glad you found the time to look into this.
I updated my Test Project to 2.61.0-SNAPSHOT
and the issue seems to be resolved there 🥳

BUT it looks like sending the data application/x-www-form-urlencoded using the GET method is broken now.
(See Test AJAXControllerTest#submissionsucceeds)

I didn't had the time to look into the implementation but it seems like the value of the parameter is added twice to RequestParam parameters in the controller methods.
(see screenshot of debugging the issue)

rbri · 2022-03-26T13:10:32Z

@thuri thanks for testing, from my point of view i have nothing changed in the processing of xmlhttp requests. will have a look at your finding - i really like to get this closed ;-)

rbri · 2022-03-26T18:31:35Z

ok, think i found it - but it looks like spring needs something special here....

rbri · 2022-03-27T10:17:37Z

Have created an Spring issue because i think it makes no sense to solve this in HtmlUnit only.
Hope we can agree on something here soon.

…tUrlParameters and introduce WebRequest.getParameters() #223

…or get requests also (#223)

rbri · 2022-09-08T17:26:10Z

As spring-projects/spring-framework#28240 is closed i will close this also.

This was referenced Sep 24, 2020

Okay I finally found some time to look into this. #235

Closed

WebClient/MockMVC issue with jquery ajax post spring-projects/spring-framework#25768

Closed

rbri self-assigned this Oct 22, 2020

bclozel mentioned this issue Dec 17, 2021

Working POST ajax requests sent by HtmlUnit spring-projects/spring-framework#27832

Closed

thuri added a commit to thuri/htmlunit-test-project that referenced this issue Jan 2, 2022

tests for fix of HtmlUnit/htmlunit#223

7f3a089

rbri added a commit that referenced this issue Mar 25, 2022

better naming and document change (#223 and spring-projects/spring-fr…

602433e

…amework#25768)

rbri mentioned this issue Mar 27, 2022

Make use of new HtmlUnit API for access request parameters spring-projects/spring-framework#28240

Closed

rbri added a commit that referenced this issue Mar 29, 2022

revert rename WebRequest.getRequestParameters to WebRequest.getReques…

2771366

…tUrlParameters and introduce WebRequest.getParameters() #223

rbri added a commit that referenced this issue May 20, 2022

WebRequest.getParameters() now takes care of explicit set parametes f…

ebf69ae

…or get requests also (#223)

rbri closed this as completed Sep 8, 2022

Jquery ajax post doesn't work #223

Jquery ajax post doesn't work #223

Comments

guisimon28 commented Sep 3, 2020 • edited Loading

twendelmuth commented Sep 11, 2020

guisimon28 commented Sep 14, 2020

twendelmuth commented Sep 14, 2020

guisimon28 commented Sep 15, 2020 • edited Loading

twendelmuth commented Sep 15, 2020

guisimon28 commented Sep 15, 2020 • edited Loading

twendelmuth commented Sep 19, 2020 • edited Loading

vkandrotas commented Sep 23, 2020

guisimon28 commented Sep 24, 2020

fredarene commented Oct 2, 2020 • edited Loading

rbri commented Oct 3, 2020

twendelmuth commented Oct 3, 2020

rstoyanchev commented Oct 7, 2020 • edited Loading

rbri commented Oct 11, 2020

rbri commented Oct 16, 2020

guisimon28 commented Oct 17, 2020

rstoyanchev commented Oct 18, 2020

bclozel commented Oct 22, 2020 • edited Loading

rbri commented Oct 22, 2020

rbri commented Oct 22, 2020

rbri commented Oct 28, 2020 • edited Loading

bclozel commented Nov 4, 2020

rbri commented Nov 4, 2020

rbri commented Nov 4, 2020

guisimon28 commented Dec 11, 2020

guisimon28 commented Feb 18, 2021

rbri commented Feb 18, 2021

thuri commented Jan 2, 2022

thuri commented Jan 2, 2022

rbri commented Jan 3, 2022

thuri commented Jan 3, 2022

jcaillabet commented Jan 5, 2022

thuri commented Jan 5, 2022 via email

rbri commented Mar 25, 2022

rbri commented Mar 25, 2022

thuri commented Mar 26, 2022

rbri commented Mar 26, 2022

rbri commented Mar 26, 2022

rbri commented Mar 27, 2022

rbri commented Sep 8, 2022

guisimon28 commented Sep 3, 2020 •

edited

Loading

guisimon28 commented Sep 15, 2020 •

edited

Loading

guisimon28 commented Sep 15, 2020 •

edited

Loading

twendelmuth commented Sep 19, 2020 •

edited

Loading

fredarene commented Oct 2, 2020 •

edited

Loading

rstoyanchev commented Oct 7, 2020 •

edited

Loading

bclozel commented Oct 22, 2020 •

edited

Loading

rbri commented Oct 28, 2020 •

edited

Loading