"curl" Command

Learn all about curl command

HOME

Curl is a command line tool for doing all sorts of URL manipulations and transfers. cURL supports various protocols like, DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP.

root:~# man curl

Curl is not written to do everything for you. It makes the requests, it gets the data, it sends data and it retrieves the information.

Using curl's option --verbose (-v as a short option) will display that what kind of commands curl sends to the server, as well as a few additional informational texts. This will help in debugging.

root:~# curl -v http://candidinformation.com/
* Hostname was NOT found in DNS cache
*   Trying 166.62.6.49...
* Connected to candidinformation.com (166.62.6.49) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: candidinformation.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 20 Sep 2017 18:12:08 GMT
* Server Apache/2.4.25 is not blacklisted
< Server: Apache/2.4.25
< Last-Modified: Mon, 12 Jun 2017 07:18:14 GMT
< ETag: "49a059d-13b5-551be1f4c2293"
< Accept-Ranges: bytes
< Content-Length: 5045
< Vary: Accept-Encoding,User-Agent
< Content-Type: text/html
<
<!DOCTYPE html>
<html lang="en">
.
.
.
</html>

Sometimes even --verbose is not enough then --trace and --trace-ascii offer even more details as they show EVERYTHING curl sends and receives.

--trace [filename] will save a full trace in the given file name. User can also give '-' instead of a file name to get it passed to stdout.

root:~# curl --trace dump http://www.candidinformation.com/

When completed, there is a 'dump' file that generated and first 20 lines of the dump looks like:

root:~# head -n 20 dump
== Info: Hostname was NOT found in DNS cache
== Info:   Trying 166.62.6.49...
== Info: Connected to www.candidinformation.com (166.62.6.49) port 80 (#0)
=> Send header, 89 bytes (0x59)
0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
0010: 55 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c User-Agent: curl
0020: 2f 37 2e 33 35 2e 30 0d 0a 48 6f 73 74 3a 20 77 /7.35.0..Host: w
0030: 77 77 2e 63 61 6e 64 69 64 69 6e 66 6f 72 6d 61 ww.candidinforma
0040: 74 69 6f 6e 2e 63 6f 6d 0d 0a 41 63 63 65 70 74 tion.com..Accept
0050: 3a 20 2a 2f 2a 0d 0a 0d 0a                      : */*....
<= Recv header, 17 bytes (0x11)
0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a                                              .
<= Recv header, 37 bytes (0x25)
0000: 44 61 74 65 3a 20 53 75 6e 2c 20 31 37 20 53 65 Date: Sun, 17 Se
0010: 70 20 32 30 31 37 20 31 34 3a 30 36 3a 35 38 20 p 2017 14:06:58
0020: 47 4d 54 0d 0a                                  GMT..
== Info: Server Apache/2.4.25 is not blacklisted
<= Recv header, 23 bytes (0x17)
0000: 53 65 72 76 65 72 3a 20 41 70 61 63 68 65 2f 32 Server: Apache/2

Every single sent and received byte gets displayed individually in hexadecimal numbers so if use thinks that hexadecimal isn't helping then --trace-ascii can be used.

--trace-ascii [filename] will save a full trace in the given file name. User can also give '-' instead of a file name to get it passed to stdout.

root:~# curl --trace-ascii dump http://www.candidinformation.com/

When completed, there is a 'dump' file that generated and first 20 lines of the dump looks like:

root:~# head -n 20 dump
== Info: Hostname was NOT found in DNS cache
== Info:   Trying 166.62.6.49...
== Info: Connected to www.candidinformation.com (166.62.6.49) port 80 (#0)
=> Send header, 89 bytes (0x59)
0000: GET / HTTP/1.1
0010: User-Agent: curl/7.35.0
0029: Host: www.candidinformation.com
004a: Accept: */*
0057:
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
<= Recv header, 37 bytes (0x25)
0000: Date: Sun, 17 Sep 2017 14:12:40 GMT
== Info: Server Apache/2.4.25 is not blacklisted
<= Recv header, 23 bytes (0x17)
0000: Server: Apache/2.4.25
<= Recv header, 46 bytes (0x2e)
0000: Last-Modified: Mon, 12 Jun 2017 07:18:14 GMT
<= Recv header, 36 bytes (0x24)
0000: ETag: "49a059d-13b5-551be1f4c2293"

--trace-time options will prefixes all verbose/trace outputs with a high resolution timer for when the line is printed. It works with regular -v/--verbose option as well as with --trace/--trace-ascii

root:~# curl -v --trace-time http://www.candidinformation.com/
14:24:36.467692 * Hostname was NOT found in DNS cache
14:24:36.496278 *   Trying 166.62.6.49...
14:24:36.690067 * Connected to www.candidinformation.com (166.62.6.49) port 80 (#0)
14:24:36.690122 > GET / HTTP/1.1
14:24:36.690122 > User-Agent: curl/7.35.0
14:24:36.690122 > Host: www.candidinformation.com
14:24:36.690122 > Accept: */*
14:24:36.690122 >
14:24:36.891372 < HTTP/1.1 200 OK
14:24:36.891416 < Date: Sun, 17 Sep 2017 14:24:36 GMT
14:24:36.891442 * Server Apache/2.4.25 is not blacklisted
14:24:36.891464 < Server: Apache/2.4.25
14:24:36.891488 < Last-Modified: Mon, 12 Jun 2017 07:18:14 GMT
14:24:36.891511 < ETag: "49a059d-13b5-551be1f4c2293"
14:24:36.891534 < Accept-Ranges: bytes
14:24:36.891557 < Content-Length: 5045
14:24:36.891580 < Vary: Accept-Encoding,User-Agent
14:24:36.891603 < Content-Type: text/html

Example-1

1.1 Download Content of Single URL

Following Command will get the content of the URL and display on standard terminal.

root:~#curl http://www.candidinformation.com/

1.2 Save the output to a file (-o or -O)

-o (lowercase) the content will be saved in given filename
-O (uppercase) the filename in the URL will be taken and used as file to store its content

root:~# curl http://www.candidinformation.com/index.html -o temp.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5045  100  5045    0     0  11769      0 --:--:-- --:--:-- --:--:-- 11787
root:~# curl http://www.candidinformation.com/index.html -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5045  100  5045    0     0  11735      0 --:--:-- --:--:-- --:--:-- 11732
root:~# ls
index.html  temp.html

Here, as you see
In first(-o lowercase) temp.html file is used to save content. In second(-O uppercase) index.html from URL itself is used to save content.

1.3 Fetch Multiple Files at a time

curl -O URL_1 -O URL_2 -O URL_3

Following command will download both cat.html and awk.html in the same directory.

root:~# curl -O http://www.candidinformation.com/linux/linux_cmds/cat.html -O http://www.candidinformation.com/linux/linux_cmds/awk.html

1.4 Multiple HTTP methods in a single command line (--next) Sometimes user needs to operate on several URL with different HTTP methods on each. For this, following command can be used:

#This will send HEAD and then a GET:
curl -l http://www.candidinformation.com --next http://www.candidinformation.com

Following command will download both cat.html and awk.html in the same directory.

root:~# curl -O http://www.candidinformation.com/linux/linux_cmds/cat.html -O http://www.candidinformation.com/linux/linux_cmds/awk.html

Example-2

2.1 Follow HTTP Redirection/Location Header (-L)

By Default, curl doesn't follow HTTP location headers that is termed as Redirects. When a requested web page is moved to another place, then an HTTP location header will be sent as a response where actual web page is located.

root:~# curl http://www.yahoo.com/

<TITLE>Document Has Moved</TITLE>
<H1>Document Has Moved</H1>
<FONT FACE="Helvetica,Arial">
Description: The document you requested has moved to a new location.  The new location is "https://www.yahoo.com/".
</FONT>

Now, when -L is given, curl will download web page html code.

root:~# curl -L http://www.yahoo.com/

2.2 Continue/Resume Previous Download (-C)

Curl supports resume of download large file which was stopped already due to some reason. When '-C -' is given then curl will find from where to start resuming the download. User can also give an offset '-C OFFSET'. when offset is given then curl will skip those bytes from the begining of the source file. Here '-#' has been used which shows the transfer as progress bar.

root:~# curl -# -O http://speedtest.ftp.otenet.gr/files/test10Mb.db
###################################                                       49.2%^C
root:~# ls -l test10Mb.db
-rw-r--r-- 1 root root 5660672 Sep 17 13:39 test10Mb.db
root:~# curl -# -C - -O http://speedtest.ftp.otenet.gr/files/test10Mb.db
#####################################                                    51.4%

root:~# rm test10Mb.db
root:~# curl -# -O http://speedtest.ftp.otenet.gr/files/test10Mb.db
##############                                                            20.6%^C
root:~# ls -l test10Mb.db
-rw-r--r-- 1 root root 2400256 Sep 17 13:43 test10Mb.db
root:~# curl -# -C 2400256 -O http://speedtest.ftp.otenet.gr/files/test10Mb.db
######################################################################## 100.0%
root:~# mv test10Mb.db temp.db
root:~# curl -# -O http://speedtest.ftp.otenet.gr/files/test10Mb.db
######################################################################## 100.0%
root:~# md5sum test10Mb.db temp.db
f1c9645dbc14efddc7d8a322685f26eb  test10Mb.db
f1c9645dbc14efddc7d8a322685f26eb  temp.db

Example-3

3.1 Host

The host name is usually resolved using DNS or /etc/hosts file to an IP address and that's what curl will communicate with. Alternatively user can specify the IP address directly in the URL instead of a name. For development and other trying out situtions,user can point to a different IP address for a host name than what would have otherwise be used, by using curl's --resolve option:

root:~# curl --resolve www.candidinformation.com:80:127.0.0.1 -v http://www.candidinformation.com/

3.2 Port Number

Each protocol curl supports operates on a default port number, be it over TCP or in some cases UDP.Normally user don't have to take that into consideration, but at some times user run test servers on other ports or similar; then you can specify the port number in the URL with a colon and a number immediately following the host name.

root:~# curl -v http://www.candidinformation.com:80/

3.3 Proxy

Using a HTTP proxy on port 4321:

root:~# curl -v --proxy http://proxy.abc.com:4321 http://www.candidinformation.com/

3.4 Username and Password

Some services are setup to require HTTP authentication and then you need to provide name and password which is then transferred to the remote site in various ways depending on the exact authentication protocol used.

root:~# curl http://user:password@candidinformation.com/

or

root:~# curl -u user:password http://candidinformation.com/

Other Authentication
Site might require a different authentication method; check the headers returned by the server and then use --ntlm, --digest, --negotiate or even --anyauth might be options that suit you.
Proxy Authenitcation

root:~# curl --proxy-user proxyuser:proxypassword http://www.candidinformation.com/

Example-4

4.1 Limit Data Transfer Rate (--limit-rate)

User can limit the amount at which the data gets transferred by using --limit-rate. Here user can specify the maximum transfer rate as argument.

root:~# curl --limit-rate 10000B -# -O http://speedtest.ftp.otenet.gr/files/test10Mb.db

4.2 Download a File if its modified before/after given time (-z, --time-cond)
Following command will download file if its modified after 1st Dec 2015.

root:~# curl -z 01-Dec-15 -# -O http://speedtest.ftp.otenet.gr/files/test10Mb.db

Following command will download file if its modified before 1st Dec 2015.

root:~# curl -z -01-Dec-15 -# -O http://speedtest.ftp.otenet.gr/files/test10Mb.db

Note: Kindly see '-' sign before date.

Example-5

5.1 HTTPS (--cert and --key)

There are a few ways to do secure HTTP transfers. By far the most common protocol for doing this is what is generally known as HTTPS, HTTP over SSL. SSL encrypts all the data that is sent and received over the network and thus makes it harder for attackers to spy on sensitive information. To get a page from a HTTPS server, simply run curl like:

root:~# curl https://www.yahoo.co.in/

In the HTTPs World, you use certificates to validate that you are the one you claim to be,as an addition to normal passwords. Curl supports client-side certificates. All certificates can be locked with a pass phrase, which user needs to enter before the certificates can be used by curl. The pass phrase can be specified on the command line or if not, entered interactively when curl queries for it.

root:~# curl --cert mycert.cer --key mykey.pem https://secure.xyzcompany.com/

5.2 --insecure (-k)

Default, curl will verify the server who it claim to be, by verifying the server's certificate against a locally stored CA cert bundle; Failing the verification will cause curl to deny the connection. In this case, user must use --insecure (-k) option to tell curl to ignore server verification. This will be helpful to download bundles from company's internal website which can't be verified by standard CA cert bundles.

root:~# curl --cert mycert.pem -k https://bundles.xyzcompany.com/packages/a.tar.gz

5.3 SSL version ( --sslv2 (-2) or --sslv3(-3))

Few websites only supports SSLv2 or SSLv3 and that can be given by following option:

root:~# curl -2 https://bundle.xyzcompany.com/

Above command will use ssl v2 version.

root:~# curl -3 https://bundle.xyzcompany.com/

Above command will use ssl v3 version.

Example-6

6.1 HTTP Methods

It should be noted that curl selects which methods to use on its own depending on what action to ask for. -d will do POST, -I will do HEAD and so on. If you use the --request / -X option you can change the method keyword curl selects, but you will not modify curl's behavior. It means that, for example use -d "data" to do a POST, you can modify the method to a PROPFIND with -X and curl will still think it sends a POST. You can change the normal GET to a POST method by simply adding -X POST in a command line like:

root:~# curl -X POST http://www.candidinformation.com/

6.2 Send File along with Request

File can be sent along with the request.

root:~# curl -s -v -H 'YOUR-CUSTOM-HEADER: VALUE' -X POST 'http://www.candidinformation.com/' --data-binary @/root/pashah/your_file

-H option will be used to send header.

--data-binary option will be used to send file. Here requested file will be sent to the server.

6.3 Send Mail using SMTP protocol

Curl can also be used to send mail using SMTP protocol.

root:~# curl --mail-from blah@test.com --mail-rcpt foo@test.com smtp://mailserver.com

Once the above command is entered,it will wait for the user to provide the data to mail.Once use has composed the message, type . (period) as the last line,which will send the mail immediately.

Subject : Curl Testing
This is a test mail please dont reply
.