Output to stdout

[seanfisk]

aria2 is awesome! However, one thing I am missing is output to stdout, mainly so things like this can work:

# download and untar the latest version of Boost
wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.gz -O - | tar -xz

I am comfortable with C++, so I have no problem making a pull request to add it. But I wanted to gauge how involved this feature might be. It seems simple, but it might be more complex than I imagine, since all visible output now has to go to stderr or directly to the tty. So I figured I'd ask about it first.

[tatsuhiro-t]

The big issue is that aria2 can download multiple files simultaneously and out-of-order in its original file position. In that condition, output to stdout is somewhat hard to use.

[seanfisk]

OK, so it sounds like it would be pretty difficult and possibly in conflict with typical use of the tool. A couple other things I noticed that would make it difficult to implement:

• Progress is printed to stdout by default. Obviously this wouldn't be possible if the file is being output
  to stdout, so that would have to change (presumably to stderr or /dev/tty). That might be a breaking change for others' scripts if they expected progress to be output to stdout.
• The -o option isn't considered a path, it's the basename for the downloaded file.

I tried a named pipe (FIFO) approach, but the issue is that aria2 has no way of writing to a file descriptor that is known before the command is executed. Yes, you can make sure that foo.1 doesn't happen and predict the file name, but you can't create a file that aria2 then writes to. That makes it "impossible" to do the simultaneous download/untar with the current aria2 and standard tools (it's probably possible, but I'm no UNIX whiz).

Maybe this isn't the right fit for this particular tool.

[AnselmD]

I am interested in it. It is possible to determine outside from aria, how much from the "head" of the file is downloaded? Is is possible to to determine it with the controle file of aria2?
http://aria2.sourceforge.net/manual/en/html/technical-notes.html#control-file-aria2-format

[seanfisk]

@AnselmD Possibly, but I think that if this feature was implemented, it should be implemented internally rather than externally.

[yarikoptic]

I would like to chime another 1c for the ability to specify output file name. ATM, I see no ability to download from multiple URLs into a differently named file. Usecase -- use by git-anex see http://git-annex.branchable.com/todo/extensible_addurl/#comment-40a1d58630f56dd744d56dc56a68770e

[tatsuhiro-t]

To specify different file name for each downloads, currently one has to use -i file option.
http://aria2.sourceforge.net/manual/en/html/aria2c.html#cmdoption-i
Inside it you can set options per download item: http://aria2.sourceforge.net/manual/en/html/aria2c.html#input-file

[yarikoptic]

On Fri, 05 Dec 2014, Tatsuhiro Tsujikawa wrote:

To specify different file name for each downloads, currently one has to
use -i file option.
http://aria2.sourceforge.net/manual/en/html/aria2c.html#cmdoption-i
Inside it you can set options per download item:
http://aria2.sourceforge.net/manual/en/html/aria2c.html#input-file

sorry if I am blind but I don't see an option for specifying the output
filename among those options ... could you please point me further or
just provide an example? Thanks in advance!

[tatsuhiro-t]

For example, suppose you created the following file:

http://localhost/foo/bar
    out=buzz
http://localhost/index.html
    out=file.html

and save it to list.txt.

Then aria2c -ilist.txt makes aria2 download 2 files and file from http://localhost/foo/bar is saved as buzz and file from http://localhost/index.html is saved as file.html.
In list.txt, you can write URI per line for each download. Options for each download can be specified after the URI line and preceded by one or more spaces. In the above example, out option is specified, which is the same as --out option in command-line. You can also write multiple URIs per line delimited by TAB to download a resource from multiple URIs.

[ghost]

That's the major showstopper for me as well. I think it would be rather easy to determine internally if we're currently downloading single file or multiple and enable "--output2stdout" (or whatever you might call it) only for single-file downloads. There're already options which work differently for single- and multi- file downloads (--download-result for example).
I really hope one day aria2c will be able to catch up with wget's features.

[icodeforlove]

I needed this as well because we were pulling 500GB, up to 1TB snapshot files from cloud storage to a machine which did not have double the physical space. Resumability is not required as if the pipe breaks the whole operation is broken, this does not mean that the underlying process cant keep retrying on parts (and it does this).

I threw together https://github.com/icodeforlove/npm_fatpipe.

It's only for piping output, so if you are looking for resumable features I'd stick to aria2. As for how the concurrency is handled It downloads multiple parts, and pipes to stdout in order. One of the big issues you will run into is handling back pressure, so having the ability to fine tune this is very useful.

Maybe something like this can be added into aria2c, if not you can use fatpipe for this edgecase.

[jalr]

I want to regularyly restore compressed database backups. A drop-in replacement for the curl $dumpurl | lzop -dc | mysql would solve my problems I currently have with aborted downloads.

[sebma]

Do you plan to implement this feature so that we can pipe the output to html parsers such as pup ?

[mikepurvis]

Just adding another voice here; I'd like to be able to specify arbitrary streams for my downloads, especially when talking to the daemonized aria2 client. As a quick trial of attemping to set it to write directly to stdout, we end up having issues when it tries to do things like seek/flush the file:

$ aria2c https://github.com/aria2/aria2/raw/master/AUTHORS --dir / -o dev/stdout --allow-overwrite

09/11 16:41:16 [NOTICE] Downloading 1 item(s)

09/11 16:41:16 [NOTICE] CUID#7 - Redirecting to https://raw.githubusercontent.com/aria2/aria2/master/AUTHORS

09/11 16:41:16 [ERROR] Exception caught
Exception: [DefaultBtProgressInfoFile.cc:211] errorCode=1 Failed to write into the segment file /dev/stdout.aria2

09/11 16:41:16 [ERROR] Error when trying to flush write cache
Exception: [AbstractDiskWriter.cc:335] errorCode=17 Failed to seek the file /dev/stdout, cause: Illegal seek

09/11 16:41:16 [ERROR] CUID#7 - Download aborted. URI=https://github.com/aria2/aria2/raw/master/AUTHORS
Exception: [AbstractCommand.cc:403] errorCode=17 URI=https://raw.githubusercontent.com/aria2/aria2/master/AUTHORS
  -> [DownloadCommand.cc:124] errorCode=17 Write disk cache flush failure index=0

09/11 16:41:16 [NOTICE] Download GID#ee8c0e9b5e8a9d7a not complete: /dev/stdout

So the first order of business here is probably a stream-friendly output mode which avoids doing these things.

[exebook]

I came here because I also wanted this feature, but reading the comments I just realized it is just not possible at least for my usage scenarios. I mostly use Aria for torrents, and torrents simply do not download the files sequentially, they instead download random chunks of a file or many files actually. Aria could cache the contents of a file until it's fully downloaded and then print it to stdout, but for large videos for example it will make little sense. So not only such feature should abort when multiple files output was detected but also if torrent protocol was used. Basically aria is not wget because wget/curl/rsync will not support torrents.

[IngwiePhoenix]

Came here to see if this was an option since -o - failed (on Windows). Hope this eventually becomes an option for single-file HTTP downloads. :)

[lury]

While it's not a perfect answer to the original question/request, i't might be important to add to man/docs way to compress files after downloading, because it's possible.

This is not an in-place compression - you have to have disk space to store uncompressed files at first, but after compressing them, space is freed.

To do this, you have to use --on-download-complete ./hook.sh option with compression defined in hook.sh as follows:

#!/bin/sh
echo "Compressing [$3]"
zstd $3
rm -f $3

and now even with input file it will:

1. download each file from input file
2. compress the downloaded file
3. remove original file

and when you have an input file, it basically comes to:
aria2c --on-download-complete ./hook.sh -i list_of_files_to_download.txt

You don't have to change file name because it's passed as an argument.

[jnorthrup]

I have proposed a seemingly realistic streamed queuing feature with a synergy to this and other issues, #2060 wherein this enhancement has synergy with issues below also

#911
#169
#1364
#1092
#339
#281
webtorrent/webtorrent#2536

[CoderYellow]

any progress on this , at least provide a callback function in the api to receive buffer data, so that we can write to custom place

[xiekunwhy]

also request for standard output