Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get with redirect leads to ~30sec hang #435

Open
stephematician opened this issue Jun 29, 2024 · 7 comments
Open

get with redirect leads to ~30sec hang #435

stephematician opened this issue Jun 29, 2024 · 7 comments

Comments

@stephematician
Copy link

stephematician commented Jun 29, 2024

I'm trying to download a file (or other 'get' requests) which have redirects, but they seem to hang for some time. E.g. let's say I use a promise to download a file:

function get_file(url: string, file_path: string) : Promise<void> {

    return new Promise<void>((resolve, reject) => {
        needle.get(
            url, { follow_max: 1, output: file_path },
        (error) => {
            if (error) { reject(error); }
            resolve();
        })
    }); 

}

async function foo() : Promise<void> {
    await get_file(
        "https://github.com/tomas/needle/archive/refs/tags/v3.2.0.zip",
        "foo.zip"
    );
    console.log("So we should be done here.");
}

foo();
// > So we should be done here.
// then a 30 second delay (roughly) before the program exits

The only way around this that I've had luck with, so far, is to attach a signal to the request and abort it after the file has downloaded.

It seems to be an issue when following? Because if I download a file that doesn't require a redirect, there isn't any delay before the program ends.

@stephematician
Copy link
Author

stephematician commented Jun 29, 2024

Can get the same behaviour with a simple change to example/download-to-fiile.js:

var fs = require('fs'),
    needle = require('needle'),
    path = require('path');

var url  = process.argv[2] || "https://github.com/tomas/needle/archive/refs/tags/v3.2.0.zip";
var file = path.basename(url);

console.log('Downloading ' + file);

needle.get(url, { output: file, follow: 3 }, function(err, resp, data){
  console.log('File saved: ' + process.cwd() + '/' + file);

  var size = fs.statSync(file).size;
  if (size == resp.bytes)
    console.log(resp.bytes + ' bytes written to file.');
  else
    throw new Error('File size mismatch: ' + size + ' != ' + resp.bytes);
});
// > Downloading v3.2.0.zip
// > File saved: /home/username/v3.2.0.zip
// > 79318 bytes written to file.
// then ~30sec wait til exit

@stephematician
Copy link
Author

@Afam007 You could not reproduce? What system/node are you using?

@stephematician
Copy link
Author

Thanks, but I'll wait and see if others can reproduce given the MWE I provided.

@stephematician
Copy link
Author

stephematician commented Jul 3, 2024

I tested with a non-github file that also requires a redirect, the results are the same, e.g. substituting the following for url http://www.vim.org/scripts/download_script.php?src_id=9750. The file downloads, and then the program waits for about 30 seconds before exiting.

I do not observe the same behaviour with ky, e.g. with a simple test module:

import ky from "ky";

var file_url  = process.argv[2] || "http://www.vim.org/scripts/download_script.php?src_id=9750"

async function foo(file_url) { const response = await ky(file_url); }

foo(file_url);
// Exits promptly - although note I have not written the output to file, but I suspect
// that would not make a difference.

@stephematician
Copy link
Author

stephematician commented Jul 3, 2024

Also works without 30sec hang using fetch:

const fs = require('fs');
const { Readable } = require('stream');
const { finished } = require('stream/promises');

async function foo() {
const stream = fs.createWriteStream('v3.2.0.zip');
const { body } = await fetch('https://github.com/tomas/needle/archive/refs/tags/v3.2.0.zip');
await finished(Readable.fromWeb(body).pipe(stream));
}

foo();

As it stands, I don't think it makes sense that the ISP would be at fault and that a proxy/VPN is needed to download. All the other approaches work from my machine on the ISP. No other application hangs after a file is downloaded like needle does.

It is also not only limited to files (content type). Other content types that require a redirect have the same behaviour.

@stephematician
Copy link
Author

stephematician commented Jul 4, 2024

Lastly: here's some output that shows one of the requests (the original one) doesn't close even though I think request.end() should have been called within needle's send_request.

var fs = require('fs'),
    needle = require('needle'),
    path = require('path');

var url  = process.argv[2] || "https://github.com/tomas/needle/archive/refs/tags/v3.2.0.zip";
var file = path.basename(url);

console.log('Downloading ' + file);

x = needle.get(url, { output: file, follow: 3 }, function(err, resp, data){
  console.log('File saved: ' + process.cwd() + '/' + file);

  var size = fs.statSync(file).size;
  if (size == resp.bytes)
    console.log(resp.bytes + ' bytes written to file.');
  else
    throw new Error('File size mismatch: ' + size + ' != ' + resp.bytes);
});
let start = new Date();

x.on("response", () => {
    console.log(`response (x) at ${(new Date()).getTime() - start.getTime()}`);
});
x.on("close", () => {
    console.log(`close (x) at ${(new Date()).getTime() - start.getTime()}`);
});
x.request.on("response", () => {
    console.log(`response (x.req) at ${(new Date()).getTime() - start.getTime()}`);
});
x.request.on("timeout", () => {
    console.log(`timeout (x.req) at ${(new Date()).getTime() - start.getTime()}`);
});
x.request.on("close", () => {
    console.log(`close (x.req) at ${(new Date()).getTime() - start.getTime()}`);
});

With output:

Downloading v3.2.0.zip
response (x) at 385
response (x.req) at 389
response (x) at 790
close (x) at 823
File saved: /home/stephematician/Documents/temp/test_needle/v3.2.0.zip
79318 bytes written to file.
timeout (x.req) at 5384
close (x.req) at 30384

@stephematician
Copy link
Author

Potentially related: nodejs/node#47228

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant