cURL is a useful tool that allows you to easily transfer data and files across systems over a network. It can be used as a tool as well as library. cURL supports all major protocols such as HTTP, HTTPS, FTP, SFTP and more. In this article, we will look at how to use curl to download files in Linux.
How to Use Curl to Download Files in Linux
Here are the steps to use curl to download files in Linux.
How to Install cURL
By default, curl is already installed on most Linux systems. However, if it is not present on your system, you can install it using the following command.
## Debian/Ubuntu Linux $ sudo apt install curl ## Fedora/CentOS/RHEL $ sudo dnf install curl ## OpenSUSE Linux users $ sudo zypper install curl
You can check if cURL is installed on your system, with the following command
$ curl --version
Also read : How to Set or Change Time Zone in Ubuntu
We will look at some common example of using cURL command.
Download Files using cURL
The basic syntax of cURL is
curl [options] url_with_protocol
You need to specify the URL with protocol, from where you want to download the file.
Here is an example to download a file from a URL.
$ sudo curl --output your_file.pdf https://www.example.com/my_file.pdf OR $ sudo curl -o your_file.pdf https://www.example.com/my_file.pdf
If you don’t specify the -o or –output option curl will save the downloaded file with a random name.
If you want to save the downloaded file with the same name, use -O (capital O) option
$ sudo curl -O https://www.example.com/my_file.pdf
Also read : How to Implement SSL/TLS in Apache Tomcat
Resume failed or paused downloads
If for some reason, your download was interrupted, then you can resume file download with the -C command
$ sudo curl -C - -O https://www.example.com/my_file.pdf
Download files with URL redirect
By default, curl does not download a file from a URL which gets redirected. You can do so using -L option.
$ sudo curl -L -O https://www.example.com/my_file.pdf
Also read : How to Extract data from JSON file using Python
Download Multiple Files
If you want to download multiple files from different URLs then you need to add each URL separately with its own -O option.
$ sudo curl -O url1 -O url2
$ sudo curl -O https://www.example.com/my_file.pdf \
-O https://www.example.com/my_file2.pdf
If you have these URLs stored in a file (e.g. URLs.txt) then you can ask curl to pick those URLs from the file and download them, using -n command. Make sure you list 1 URL on each line of the file.
URLs.txt ------ https://www.example.com/my_file.pdf https://www.example.com/my_file2.pdf https://www.example.com/my_file3.pdf
Here is the command to download a list of URLs. We also use xargs command along with curl for this purpose.
xargs -n 1 curl -O < "urls.txt"
Also read : How to Install Laravel with NGINX
Download password protected file
Sometimes the file you want to download may be password protected. Here is the command to download a password protected file. You need to mention the username & password with -u option. Replace test_user & test_password below with your username & password.
$ curl -u test_user:test_password https://www.example.com/my_file.pdf
Download Files Using Proxy Server
Here is the syntax to download files using proxy server.
curl -x proxy-server-ip:PORT -O url
If it requires authentication, then you need to also provide username & password.
curl -u username:password -x proxy-server-ip:PORT -O url
Here is an example,
curl -x 'https://proxy.example.com' -v -O https://www.example.com/my_file.pdf
Also read : How to Check Cron Log with Linux
Rate Limits in Download
If you want to limit the download speed of your files, you can do so with the –limit-rate option.
curl --limit-rate {speed} url
Here is an example to limit download rate to 200kb/sec.
$ sudo curl --limit-rate 200k -O https://www.example.com/my_file.pdf
There are many more things you can do with cURL, but we have covered the most common ones. You can also import cURL as a library in Python/Java, which we will cover in another article.
Also read : How to Install CouchDB in Ubuntu