Table of Contents
URL: https://www.progressiverobot.com/workflow-downloading-files-curl/
Client URL, or cURL, is a library and command-line utility for transferring data between systems. It supports many protocols and tends to be installed by default on many Unix-like operating systems. Because of its general availability, it is a great choice for downloading a file to your local system, especially in a server environment.
In this tutorial, you'll use the curl command to download a text file from a web server. You'll view its contents, save it locally, and tell curl to follow redirects if files have moved. This knowledge is particularly useful when working with REST APIs or setting up Node.js applications.
Downloading files from the Internet can be dangerous, so be sure you are downloading from reputable sources. In this tutorial, you'll download files from the cloud provider, and you won't be executing any files you download.
Step 1 — Fetching remote files
Out of the box, without any command-line arguments, the curl command will fetch a file and display its contents to the standard output.
Let's give it a try by downloading the robots.txt file from www.progressiverobot.com:
curl https://www.progressiverobot.com/
You'll see the file's contents displayed on the screen:
[secondary_label Output]
User-agent: *
Disallow:
sitemap: https://www.progressiverobot.com/
sitemap: images/workflow-downloading-files-curl-section-1.png
sitemap: images/workflow-downloading-files-curl-section-1.png
sitemap: images/workflow-downloading-files-curl-section-1.png
Give curl a URL and it will fetch the resource and display its contents.
Saving Remote Files
Fetching a file and displaying its contents is all well and good, but what if you want to actually save the file to your system?
To save the remote file to your local system, with the same filename as the server you're downloading from, add the --remote-name argument, or use the -O option:
curl -O https://www.progressiverobot.com/
Your file will download:
[secondary_label Output]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 286 0 286 0 0 5296 0 --:--:-- --:--:-- --:--:-- 5296
Instead of displaying the contents of the file, curl displays a text-based progress meter and saves the file to the same name as the remote file's name. You can check on things with the cat command:
cat robots.txt
The file contains the same contents you saw previously:
[secondary_label Output]
User-agent: *
Disallow:
sitemap: https://www.progressiverobot.com/
sitemap: images/workflow-downloading-files-curl-section-1.png
sitemap: images/workflow-downloading-files-curl-section-1.png
sitemap: images/workflow-downloading-files-curl-section-1.png
Now let's look at specifying a filename for the downloaded file.
Step 2 — Saving Remote Files with a Specific File Name
You may already have a local file with the same name as the file on the remote server.
To avoid overwriting your local file of the same name, use the -o or --output argument, followed by the name of the local file you'd like to save the contents to.
Execute the following command to download the remote robots.txt file to the locally named do-bots.txt file:
curl -o do-bots.txt https://www.progressiverobot.com/
Once again, you'll see the progress bar:
[secondary_label Output]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 286 0 286 0 0 6975 0 --:--:-- --:--:-- --:--:-- 7150
Now use the cat command to display the contents of do-bots.txt to verify it's the file you downloaded:
cat do-bots.txt
The contents are the same:
[secondary_label Output]
User-agent: *
Disallow:
sitemap: https://www.progressiverobot.com/
sitemap: images/workflow-downloading-files-curl-section-1.png
sitemap: images/workflow-downloading-files-curl-section-1.png
sitemap: images/workflow-downloading-files-curl-section-1.png
By default, curl doesn't follow redirects, so when files move, you might not get what you expect. Let's look at how to fix that.
Step 3 — Following Redirects
Thus far, all of the examples have included fully qualified URLs that include the https:// protocol. If you happened to try to fetch the robots.txt file and only specified www.progressiverobot.com, you would not see any output, because the cloud provider redirects requests from http:// to https://:
You can verify this by using the -I flag, which displays the request headers rather than the contents of the file:
curl -I www.progressiverobot.com/robots.txt
The output shows that the URL was redirected. The first line of the output tells you that it was moved, and the Location line tells you where:
[secondary_label Output]
<^>HTTP/1.1 301 Moved Permanently<^>
Cache-Control: max-age=3600
Cf-Ray: 65dd51678fd93ff7-YYZ
Cf-Request-Id: 0a9e3134b500003ff72b9d0000000001
Connection: keep-alive
Date: Fri, 11 Jun 2021 19:41:37 GMT
Expires: Fri, 11 Jun 2021 20:41:37 GMT
<^>Location: https://www.progressiverobot.com/;
Server: cloudflare
. . .
You could use curl to make another request manually, or you can use the --location or -L argument which tells curl to redo the request to the new location whenever it encounters a redirect. Give it a try:
curl -L www.progressiverobot.com/robots.txt
This time, you see the output as curl followed by the redirect:
[secondary_label Output]
User-agent: *
Disallow:
sitemap: https://www.progressiverobot.com/
sitemap: images/workflow-downloading-files-curl-section-1.png
sitemap: images/workflow-downloading-files-curl-section-1.png
sitemap: images/workflow-downloading-files-curl-section-1.png
You can combine the -L argument with some of the aforementioned arguments to download the file to your local system:
curl -L -o do-bots.txt www.progressiverobot.com/robots.txt
[warning] Warning: Many resources online will ask you to use curl to download scripts and execute them. Before you run any scripts you have downloaded, it's good practice to check their contents before making them executable and running them. Use the less command to review the code to ensure it's something you want to run.
Step 4 — Downloading Files with Authentication
Some web files are protected and require authentication. curl allows you to handle these cases easily. This is particularly useful when working with proxy servers or secure API endpoints.
Basic Authentication (Username & Password)
To access a file that requires login credentials, use the -u flag:
curl -u username:password -O https://example.com/securefile.zip
Token-Based Authentication
You can also use headers to pass API tokens:
curl -H "Authorization: Bearer YOUR_TOKEN" -O https://api.example.com/protected/data.json
For better security, avoid hardcoding sensitive data. Instead, use environment variables or configuration files.
Step 5 — Handling Timeouts, Retries, and Resuming Downloads
Robust scripts need to account for network interruptions and delays.
Resume Interrupted Downloads
Use the -C - option to resume:
curl -C - -O https://example.com/largefile.iso
Set Timeouts
Prevent curl from hanging indefinitely:
curl --max-time 30 -O https://example.com/file.txt
Retry Failed Downloads
Automatically retry failed downloads up to 3 times:
curl --retry 3 -O https://example.com/file.txt
Step 6 — Automating Downloads with Shell Scripts
Automating downloads can be useful in CI/CD pipelines or regular backups. This is especially relevant when working with Node.js applications or REST APIs that require regular data updates.
Sample Script
#!/bin/bash
URL="https://example.com/file.zip"
DEST="/home/user/downloads/file.zip"
curl -L -o "$DEST" "$URL"
Make the script executable with chmod +x script.sh, then schedule it with cron or use it in a deployment pipeline.
Step 7 — Troubleshooting Common Download Issues
Sometimes downloads might fail or behave unexpectedly. Here are some common issues and their solutions:
File Not Downloading
If curl isn't downloading your file, try these troubleshooting steps:
curl -I https://example.com/file.zip
- Verify if the server requires a specific user agent:
curl -A "Mozilla/5.0" -O https://example.com/file.zip
- Check for SSL/TLS issues:
curl -v -O https://example.com/file.zip
- Try with different protocols if available:
# Try HTTP if HTTPS fails
curl -O http://example.com/file.zip
- Check if the file exists and you have permissions:
curl -u username:password -O https://example.com/file.zip
If you're still having issues, the verbose output (-v) will help identify the problem:
curl -v -O https://example.com/file.zip
Step 8 — Using wget as an Alternative
While curl is powerful, sometimes wget might be a better choice for certain download scenarios. wget is specifically designed for downloading files and has some features that make it particularly useful:
Basic wget Usage
To download a file with wget:
wget https://example.com/file.zip
Key wget Features
- Automatic retry on failure:
wget -t 3 https://example.com/file.zip
- Download in the background:
wget -b https://example.com/file.zip
- Limit download speed:
wget --limit-rate=200k https://example.com/file.zip
- Download entire websites:
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent https://example.com
When to Choose wget over curl
- When you need recursive downloading
- For mirroring entire websites
- When you want automatic retry functionality
- For simpler download commands (wget has fewer options to remember)
When to Stick with curl
- When you need to interact with APIs
- For more complex HTTP requests
- When you need to send data to servers
- For better script integration
Frequently Asked Questions (FAQ)
1. What is the difference between -O and -o in cURL?
The -O (uppercase "o") option in curl saves the downloaded file using the original filename as provided by the server in the URL or HTTP headers. This is particularly useful when you want to preserve the server's naming convention, or when downloading multiple files in a batch without having to specify each filename manually.
On the other hand, the -o (lowercase "o") option allows you to specify a custom filename for the downloaded file. This is helpful for organizing your downloads, preventing filename collisions, or when you want to give the file a more meaningful name locally.
Example:
curl -O https://example.com/file.zip # Saves as file.zip
curl -o custom-name.zip https://example.com/file.zip # Saves as custom-name.zip
2. How do I resume an interrupted download with cURL?
If your download was interrupted due to a network error or system reboot, you can resume it using the -C - option. This tells curl to continue the download from where it left off, assuming the server supports HTTP range requests.
Example:
curl -C - -O https://example.com/largefile.iso
This is particularly useful for large ISO files or multi-gigabyte datasets, especially when working with unreliable internet connections or automating downloads in scripts.
3. Can I download files behind authentication with cURL?
Yes, curl supports both basic and token-based authentication methods.
For basic authentication using a username and password:
curl -u username:password -O https://secure.example.com/file.zip
For token-based authentication, use the Authorization header:
curl -H "Authorization: Bearer YOUR_TOKEN" -O https://api.example.com/file.zip
Always avoid hardcoding sensitive credentials. Use environment variables or configuration files when scripting to enhance security.
4. What should I do if the download URL redirects?
Some URLs redirect from http to https, or from an old endpoint to a new one. By default, curl does not follow these redirects.
To handle this, add the -L or --location option:
curl -L -O http://example.com/download
This is especially important when accessing public APIs, shortened URLs, or migrating download links.
5. Is cURL available on Windows?
Yes, cURL is included by default in Windows 10 and later. You can use it directly from Command Prompt or PowerShell.
For older versions, or for enhanced Unix-like behaviour, you can install it via:
- Git Bash
- Cygwin
- Chocolatey with:
choco install curl
Once installed, you can run cURL commands just like on Linux or macOS. This enables cross-platform scripting and consistent development workflows.
6. How can I download multiple files at once with cURL?
You can download multiple files in a single command using either a list of URLs or a pattern. Here are two common approaches:
Using multiple URLs:
curl -O https://example.com/file1.zip -O https://example.com/file2.zip
Using a pattern with brace expansion:
curl -O https://example.com/file{1..5}.zip
For more complex scenarios, you can also use a text file containing URLs and the -K option:
# urls.txt contains one URL per line
curl -K urls.txt
7. How do I handle SSL/TLS certificate issues with cURL?
Sometimes, you might encounter SSL certificate errors when downloading from servers with expired or self-signed certificates (SSL). While not recommended for production use, you can bypass certificate verification using the -k or --insecure option:
curl -k -O https://example.com/file.zip
For a more secure approach, you can specify a custom certificate:
curl --cacert /path/to/certificate.pem -O https://example.com/file.zip
Remember that bypassing certificate verification can expose you to security risks, so use these options cautiously.
8. How can I monitor download progress and speed with cURL?
By default, cURL shows a progress bar, but you can customise the output using various options:
For a simple progress bar:
curl -# -O https://example.com/file.zip
For detailed progress information:
curl -w "\nDownloaded: %{size_download} bytes\nSpeed: %{speed_download} bytes/sec\nTime: %{time_total} seconds\n" -O https://example.com/file.zip
You can also create a custom progress format using the -w option with various variables like %{speed_download}, %{time_total}, and %{size_download}.
Conclusion
curl lets you quickly download files from a remote system. It supports a wide range of protocols like HTTP, HTTPS, FTP, and more—making it a reliable and script-friendly choice for file transfers. But it doesn’t stop there.
From simple downloads to complex API interactions, curl can handle everything from setting custom headers and authentication to managing redirects and resumable downloads. It’s a staple tool for developers, sysadmins, and DevOps engineers who need precise control over network communication without relying on heavyweight tools.
Whether you're automating tasks in a CI/CD pipeline, integrating data from external sources, or testing endpoints in REST APIs or Node.js applications, curl fits naturally into modern development workflows.
To dive deeper into all its capabilities, view the manual page by running:
man curl
Or explore online examples to sharpen your command-line skills even further.
