REST interface put method failing #2186
Comments
|
Hi Eric, just to make sure: are these machines behind a reverse-proxy, could that limit the max size allowed for file transfers (i know nginx does that by default)? |
|
@duncdrum Maybe, I'll check. But it worked before, so... sounds unlikely. And anyway, the error message doesn't make sense if that's the cause. Another problem could be in the http client module I'm using. The native hc module is no longer recommended I think? The standard conf.xml is a bit schizophrenic about it: It flags it as deprecated and the entry is commented out, but it also has a entry that enables it... Maybe that's something to improve in the standard distribution? Anyway, I'm going to try the expath http client also. |
|
Using the expath module makes no difference there is no NGINX or other proxy in the way |
|
I'm no longer sure about the "since 4.4.0" statement I made: it was used recently only on a 4.4.0 installation, so it might be in older versions as well. |
|
@eriksiegel So to be clear. Do you have exactly the same problem with the legacy HTTP module and the EXPath HTTP module? Could you share your simple example that uses the EXPath HTTP module? |
|
@adamretter Ah, good you ask. I thought I used the legacy http client but it turned out to be the EXPath client after all, right from the beginning. Confusion on my side and sorry I didn't amend this issue once I realized it. What about the example in the first comment? That's code that pretty consistently runs into errors on my machine(s). It uses the EXPath client. |
|
@eriksiegel, okay as I suspected. Can you make it super simple for me to reproduce so I can send my time on fixing the issue instead of setting up for it. Would you be able to send me a zip file with the collection structure (and just a very few XML files!) and the XQuery script in it. Before sending, can you just test that zip file against a clean eXist-db to make sure it reproduces? If you can get me that, I can probably fix it quickly... |
|
@adamretter I'll try to do this somewhere this week. You need at least two machines I suspect. We've also encountered something similar when using eXist's ANT extensions to store data on a remote machine. Scripts worked before, now lots of errors. It might be linked. I'll investigate. |
|
Ok, I've set up a test situation, here is what I've done:
Results: For small files (transferring For transferring bigger files (transferring ~550Kb @adamretter Is this enough information? |
|
@eriksiegel looks good to me. I will try and take a look tomorrow... |
|
@eriksiegel When running with |
|
So my working theory so far is this -
The reason this works for a small file is that the client completes sending the data by time (4) happens, and so (6) never occurs. For larger files the client can't send all of the data before the server closes the connection. The question is, who is at fault, the client or the server? |
|
Okay I have been able to isolate the HTTP Client code from eXist-db and reproduce the issue simply with the following Java code acting as the Client: public class Issue2186 {
private static final Path file = Paths.get("/tmp/mediumfile.xml");
private static HttpEntity getRequestBody() {
final ContentProducer producer = os -> {
// copyFast(file, os);
// NOTE: shows the broken pipe error
copySlow(file, os);
};
final EntityTemplate entityTemplate = new EntityTemplate(producer);
entityTemplate.setContentType("application/xml");
entityTemplate.setChunked(true);
return entityTemplate;
}
private static void copyFast(final Path file, final OutputStream os) throws IOException {
Files.copy(file, os);
}
private static void copySlow(final Path file, final OutputStream os) throws IOException {
try(final LineNumberReader reader = new LineNumberReader(Files.newBufferedReader(file));
final OutputStreamWriter writer = new OutputStreamWriter(new CloseShieldOutputStream(os), UTF_8)) {
String line;
while((line = reader.readLine()) != null) {
writer.write(line + "\n");
}
}
}
public static void main(final String args[]) throws IOException {
final CloseableHttpClient httpClient = HttpClients.createDefault();
try {
final HttpPut httpPut = new HttpPut("http://localhost:8080/exist/rest/db/mediumfile.xml");
httpPut.setEntity(getRequestBody());
final CloseableHttpResponse httpResponse = httpClient.execute(httpPut);
try {
System.out.println(httpResponse.getStatusLine());
} finally {
httpResponse.close();
}
} finally {
httpClient.close();
}
}
}Running the above code results in the output and then errors: So whilst I can reproduce it, I am still not sure who is at fault here. The client or the server? I think this will require me getting some clarification from the Apache HTTP Component project , and/or possibly the Jetty project? |
|
@adamretter Thanks for investigating so quickly. And: Oops. Sorry for omitting
But the testscript sends authorization... (at least its specified). So where does it gets missing then? And the EXPath http client worked fine in the past. So what's changed then in the newest release(s) of eXist that broke it? |
You mentioned 4.3.1 and 4.4.0. I can see no changes made to either the EXPath HTTP Client or the Jetty Server during that time. So I really can't see that anything could have broken it... Interestingly I can see that the authentication is not being send pre-emptively, which I think it should be as you set So we might have two different issues here:
Regards (2), I did get a reply from my enquiry to the Apache HTTP Client mainly list about how to fix this - |
This was referenced Oct 11, 2018
|
@eriksiegel Okay I sent fixes for the Preemptive Authentication stuff. It looks to me like that can't have ever worked in the EXPath HTTP Client. So now that Preemptive works, you should not see the problem, and that workaround will work for you. Unfortunately the underlying issue is still there when not using preemptive auth. Fixing the other stuff might be much more difficult. If it involves fixes to the HTTP Client itself, I would be inclined to leave that until we deliver and implement HTTP Client 2.0. |
adamretter
added
bug
|
Okay so fixing the EXPath HTTP Client 1.0 implementation to correctly support the issue we have would be a lot of work. Instead, I think we will say that we won't fix it. Instead my plan is to implement the EXPath HTTP Client 2.0, and when I do that, I will make sure to use the Asynchronous mode of the Apache HTTP Client. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is the problem
We have code that copies files from one server to the other using the standard REST interface functionality eXist provides. Here is a dressed down code example:
The environment is all ok as far as I can see. The source file exists, is well-formed, the destination collection exists and is writable.
Since using 4.4.0 there are problems with this:
What did you expect
File copied to remote system
Describe how to reproduce or add a test
Have two separate servers running eXist, preferably the source on Windows and the destination on Unix
Adapt the script above with the right stuff (address, password, etc).
Try it to run on the source machine
Context information
Please always add the following information
The text was updated successfully, but these errors were encountered: