How To Run Multiple cURL Requests in Parallel

cURL is a useful utility to make requests and download files. However, it can make only 1 request at a time. Sometimes you may need to run multiple cURL requests in parallel. Here is how to execute multiple cURL commands in parallel using xargs utility in Linux.

How To Run Multiple cURL Requests in Parallel

By default, you can run only 1 cURL command at a time. We will use xargs command to make multiple cURL requests in parallel.

xargs command accepts one or more strings as argument, from standard input, and generates separate command for each string. Here is an example.

echo "test1 test2 test3" | xargs mkdir

In the above command, xargs will split the echo string “test1 test2 test3” into individual strings test1, test2, test3 using space (‘ ‘) delimiter , loop through them and generate the following separate commands.

mkdir test1
mkdir test2
mkdir test3

Let us say you want to run the following cURL command 5 times in parallel.

$ curl -I "https://www.google.com"

Also read : Git Stash Save Local Changes without Commit

Here is how to execute cURL command in parallel using xargs.

$ xargs -I % -P 5 curl "https://www.google.com" \ < <(printf '%s\n' {1..5})
OR
$ seq 1 5 | xargs -n1 -P 5 curl -I "https://www.google.com"

In the above commands, -P 5 option tells xargs to run 5 commands in parallel. If you do not include it, then xargs will run cURL requests sequentially.

In the first command, we use printf to generate command statement using loop with 5 iterations {1..5}. In the second command, we use seq command to create the loop 1-5.

If you want to make requests to 3 separate domains in parallel, then here is the command to do it.

$ echo "https://www.google.com https://www.facebook.com https://www.yahoo.com" | xargs -P 5 -n 1 curl -I

If you want to download a set of remote files from different locations just add them to a file (e.g list.txt), one URL on each line

$ sudo vi list.txt

Add the following URLs

https://website1.com/file1.txt
https://website2.com/file2.txt
https://website3.com/file3.txt

Then simply run the following command. xargs will pick up the URLs from your file, one by one, and generate cURL statements for each URL, and then run them in parallel.

$ xargs -P 5 -n 1 curl -O < list.txt

Using xargs you can not only generate multiple curl statements but also run them in parallel.

Also read : How to Uninstall Slack in Ubuntu