The Download Helper

Download helper

FOF comes with the FOF30\Download\Download helper class for performing downloads using cURL or the ULR fopen() wrappers. The helper decides which of the two connectors is more suitable for the current site and allows you to perform full downloads (the entire file in one go), partial downloads (a range of bytes in one go) and staggered, or chunked, downloads (the entire file, a certain amount of bytes at a time). The first two should be used for relatively small files. The latter can be used for downloading files bigger than a couple of Megabytes which would cause a timeout, memory outage or CPU usage limit error if you tried downloading them all at once.

Getting and configuring an object instance

Create a new object instance, passing your component's container as a parameter to the constructor:

$download = new FOF30\Download\Download($container)

You can get the adapter name using

$adapter = $download->getAdapterName();

This will return one of curl or fopen. You can then configure the adapter with the setAdapterOptions method. For example you can configure the curl adapter with something like:

$download->setAdapterOptions([
    CURLOPT_FOLLOWLOCATION => 0,
    CURLOPT_USERAGENT => 'AcmeExample/1.0',
]);

Full download

You can download the entire response of a remote URL into a string using the getFromURL method:

$contents = $download->getFromURL('http://www.example.com/example.txt');

Partial download

You can also download a part of the response of a remote URL into a string using the getFromURL method:

$from = 500;
$to = 999;
$contents = $download->getFromURL('http://www.example.com/example.txt', $from, $to);

This will download the second 500 bytes chunk of the file.

A few notes are in order:

• Partial downloads are only supported on servers which understand HTTP Range headers. This is rarely the case with the output of scripts or even sources such as Amazon S3. If the server doesn't understand the HTTP Range header it will usually return the entire file. Some servers may detect the Range header but since they don't support it they will return an HTTP 416 Requested Range Not Satisfiable error which causes the getFromURL method to return false.
• The first byte is 0, not 1. The first 500 bytes of a file are from 0 to 499.
• If you ask for a range which doesn't exist you will get boolean false in return. If, however, you ask for a range which has a valid start but not enough bytes to satisfy the end you will get the bytes from the $from byte mark to the actual end of content. This means that your return will be less bytes than you requested. Therefore you should NOT assume that the response is exactly as many bytes as you requested, you MUST check the length yourself.

Chunked download

You can download remote URLs of a big content size into local files using importFromURL. Typically, this used to download large remote files into your site over several page loads. You should call the importFromURL method with an array containing its operating variables:

$options = [
   // Setup
   'url'           => 'http://www.example.com/big.zip',
   'localFilename' => JPATH_SITE . '/tmp/big.zip',
   'maxExecTime'   => 5,
   'runTimeBias'   => 75,
   'length'        => 1048576,
   // Runtime
   'frag'          => -1
   'totalSize'     => -1,
   'doneSize'      => -1,
];

$response = $download->importFromURL($options);

The setup variables MUST be unchanged throughout all steps of the process and are used to tell the download helper how to perform the chunked download:

• url. The URL to download from.
• localFilename. The absolute filesystem path where the file will be stored. This is written to with fopen so make sure that the file or containing folder have the correct permissions.
• length. The size, in bytes, of each download chunk. The default is 1048576 bytes (1Mb). You should keep this low. The time it takes for each chunk to download depends on its size and the bandwidth available for the connection your server does to the remote server. If you're connecting to congested servers or Internet-facing NAS over domestic Internet connections the default might be too big, causing timeouts. Values as low as 130048 bytes (128Kb) may be required in such cases.
• maxExecTime. The maximum time, in seconds, to spend downloading chunks. Please note that a single chunk may take longer than this setting, depending on the chunk size and the bandwidth between your server and the remote server. We recommend using a small chunk size (length) to avoid this situation. Keep the maxExecTime lower than the PHP max_execution_time to avoid timeouts. As a rule of thumb the maxExecTime should be about half of PHP's max_execution_time. The default value (5) is a mostly safe choice if you're unsure.
• runTimeBias. If the download of a chunk has finished between (maxExecTime * runTimeBias / 100) seconds and maxExecTime seconds then the download process is paused and importFromURL returns. This is an added precaution against timeouts. This setting is given in percentile points. Values lower than 10 and higher than 100 are ignored. You are advised to leave this to the default value of 75.

The next three options (Runtime) must all be -1 the first time you call importFromURL. Every next time they must come from importFromURL's response.

The response of importFromURL is also an array, containing the following keys:

• status. Boolean false if an error occurred, true otherwise.
• error. The error message when status === false.
• frag. Next chunk number. Pass it back to importFromURL in the next call.
• totalSize. Total size of the download file in bytes (if the server reports sizes). Pass it back to importFromURL in the next call.
• doneSize. Total number of bytes already downloaded. Pass it back to importFromURL in the next call.
• percent. Percent completed (0-100). It's 100 * doneSize / totalSize when totalSize ≠ 0, zero otherwise. You can use it to display progress bars.
• localfile. The absolute path to the local file we are downloading to.

If the status is Boolean true and frag is -1 the download has completed. In any other case you need a further call to importFromURL in the next page load to continue the download process. You should first do:

$options['frag'] = $response['frag'];
$options['totalSize'] = $response['totalSize'];
$options['doneSize'] = $response['doneSize'];

Then store the $options array in the session and retrieve it from there in the subsequent page load.

Tip: this is done in Akeeba CMS Update when downloading the Joomla! update package.

Important notes:

• Chunked downloads are only possible on servers which understand the HTTP Range header. If your server does not understand it you'll end up downloading the entire file in one go.
• FOF performs intelligent analysis of the download rate. If it detects that downloading the next chunk is likely to get us past maxExecTime it will pause the download and return. Therefore each call to importFromURL may take substantially less time than maxExecTime depending on the chunk length and network conditions.
• As explained, the time to download a single chunk depends on its length and the network conditions. Therefore each call to importFromURL may take substantially more time than maxExecTime depending on the chunk length and network conditions. This could lead to timeout errors.
• The percent response variable can be persistently zero if the remote server does not return the content size in the HTTP headers or when you're using the fopen adapter (which doesn't support getting the content size of the request). You are advised to display the doneSize to your users to provide feedback on the download process.