Using cURL to Download Remote Files from the Command Line osxdailynews.blogspot.com

osxdailynews.blogspot.com ® Using cURL to Download Remote Files from the Command Line

The powerful curl command line tool can be used to download files from just about any remote server. Longtime command line users know this can be useful for a wide variety of situations, but to keep things simple, many will find that downloading a file with curl can often be a quicker alternative to using a web browser or FTP client from the GUI side of OS X (or linux). This is is helpful for local situations, but there’s particular value if you’re in a situation where you need to download something to a remote Mac when connected through SSH.

For the purposes of this walkthrough, we’ll focus primarily on downloading files from the two commonly encountered HTTP and SFTP protocols, though it should be noted that cURL supports many more protocols. Though curl is easy to use, having some knowledge of the command line is recommended.

Download Exact Match Files with curl -O

Using the uppercase -O flag with curl downloads the file from the remote server while maintaining the exact file name, the basic syntax for this is the following:

curl -O [url]

This means if the specified URL file is named “sample.zip” it will download with the filename “sample.zip”, and if the file is named something enormous and complicated like “LongExampleFileNameForOSXDaily-v-1-3-51-revision-515b12-readme.txt” on the remote server, it will save with that exact name on the local machine. Longer file names are often better handled with the -o flag rather than -O, which we’ll address shortly.

Regular readers may recall that we used the curl -O command when explaining how to extract the actual audio content from an m3u streaming file.

Beginning any download with curl shows the percent transferred, time spent downloading and time remaining, and the transfer speed.

The screenshot offers a better representation than the pasted example below, but it looks something like this:

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 10505 100 10505 0 0 79741 0 --:--:-- --:--:-- --:--:-- 142k

With transfer speed showing you could redirect the output of curl to /dev/null and use it to test internet connection speed, but the wget command has an easier to read and follow transfer bar thus wget is better suited for that task.

Saving a Remote File with a Different Name with curl -o

Using a lowercase -o flag will allow you to specify a different file name for the downloaded file than how it is named on the remote server. This can be helpful to reduce lengthly file names or just label something so that it’s easier to find on your own. The general syntax would be:

curl -o [shortname] [url]

For example, if you want to save an iOS IPSW file that you found listed on Apple servers without having the lengthy full name, you could use the following:

curl -o iPhone5C-704.ipsw http://ift.tt/1cYGoFU

This will download the “iPhone5,3_7.0.4_11B554a_Restore.ipsw” file but named shorter as the more meaningful “iPhone5C-704.ipsw”.

If you’d rather not save the file to the present working directory, specify a path as part of the file name like so:

curl -o ~/Desktop/localexample.dmg http://url-to-file/example.dmg

Downloading Multiple Files Concurrently with curl

cURL can easily download multiple files at the same time, all you need to do is specify more than one URL like so:

curl -O [URL 1] [URL 2] [URL 3]

For files with different names, or hosted on different servers, or within different directory paths, use the complete URL, for example:

curl -O http://ift.tt/1ewzTnX http://ift.tt/1ewzVfn http://ift.tt/1meaQjr

On the other hand, if the file names to be downloaded are using incremental naming, you can use brackets to specify a download range, like so:

curl -O http://ift.tt/1meaOrM

This would grab the files fdl-1.1.txt, fd1-1.2.txt, and fd1-1.3.txt all at the same time without having to specify each unique URL. Of course this only works if the files are in the same directory together and at the same domain.

Authenticating with curl

You can also pass authentication with cURL by using the -u flag:

curl -u user:pass -O http://ftpremote_url/file-to-download.zip

Keep in mind that bash history will store the password in plain text when using -u with a username and password specified, thus this is not recommended for most situations. You can get around that by placing a space in front of ‘curl’. If you don’t use the spacebar to prefix the command, you’ll probably want to empty the command history afterwards to be safe.

Supported Protocols & Usage Beyond HTTP & FTP

As mentioned earlier, cURL usage goes far beyond HTTP and FTP, as the curl manual page entry mentions additional protocols in the description:

curl is a tool to transfer data from or to a server, using one of the

supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP,

IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS,

TELNET and TFTP).

Additionally, you’ll find that curl can also be used for PUT and POST requests, cookies, proxies, tunnels, resume downloads, and even grabbing HTTP header information or changing a user agent (effectively spoofing) without the need to use a dedicated web browser.

Like most command line utilities, you can learn much more about curl by summoning the appropriate man page with the ‘man curl’ command.

Email this Using cURL to Download Remote Files from the Command Line to a Friend! Receive Articles like this one direct to your email box! Subscribe for free today!