Curl is a command-line utility used as an HTTP client. It is popular for making HTTP requests from the command line. cURL can be used to write scripts for web scraping, check a site’s health, and download files from the command line.
It is incredibly simple and can be used in many programming languages. This article guides what cURL is and how to use it within Python.
What is cURL?
According to the website, cURL stands for “client URL”. It is a command-line tool and a library for transferring data using different application-layer network protocols such as HTTP, HTTPS, FTP, and IMAP.
It is incredibly popular and used in over 10 billion installations across devices such as radios, TVs, routers, printers, and computers. cURL is completely free and open-source. Its source code is available on GitHub.
cURL Use Cases
cURL is very useful and versatile. Listed below are the most popular use cases for cURL. While the list is not exhaustive, the following are just some of the most popular cases:
- Testing APIs: It can check if the API functions correctly, returning the correct data for a given request. In addition, it can also be used to check the API speed, that is, how quickly it responds to requests. You can write a script to check the API health periodically and send alerts when something goes wrong.
- Web Scraping: It can also automatically extract data from websites. cURL can be used in conjunction with many programming languages and as a Bash command. With cURL, you can fetch data from websites dynamically. From the response HTML, you can parse and extract the data you need. If you are interested in web scraping, you might want to check out the Geekflare Web Scraping API that makes it easier to scrape data.
- Downloading Data: With cURL, you can save the response of requests to a file. These responses could be data from API requests or files from a server. Effectively, by writing the response to a file, you have downloaded the file. Since cURL is a command-line tool, you can automate this process to download lots of files at once.
How to use cURL in Python (PycURL)
Installing PycURL
To use cURL in Python, we use the PycURL library. PycURL is a Python interface to the cURL library. It creates a thin wrapper over the already fast cURL library. This makes PycURL faster than other libraries for making requests such as urllib and requests. To use PycURL, you first have to install it. There are detailed instructions here, but an easy way to install it is using pip.
pip install PycURLOn Ubuntu 22.04, I had to install additional tools before installing PycURL. Use this command before trying to install PycURL with pip:
sudo apt install libcurl4-openssl-dev libssl-devMaking a Simple GET Request
To make a request, begin by creating a Python script to write the code in. Open the file with a text editor. I am going to use Vim, but you can use any you like. To open the file with Vim, you use the command given below:
vim pycurl.pyHere, pycurl.py is the name of the file I am going to be writing my code in, but you can name your file anything.
Next, we import the cURL class from the PycURL module
from pycurl import CurlAfter importing PycURL, we import BytesIO from io. We will need this to create a buffer to write the response of PycURL.
from io import BytesIOThen we instantiate a new Curl instance.
c = Curl()Next, we instantiate BytesIO to create a new buffer. PycURL does not have a built-in storage mechanism to store responses. So we have to create a buffer and tell it where to write the data.
buffer = BytesIO()With the buffer created, we can set options on our client object. In this case, we want to set two options; the first is the URL we request. The second is where we want to write the response body. Here is the code to do it:
c.setopt(c.URL, 'http://pycurl.io/')
c.setopt(c.WRITEDATA, buffer)Once we do this, we can request by calling the perform method of the client object and then close the request by calling the close method.
c.perform()
c.close()To get the response, we call the getvalue() method of the buffer object and decode it. We can then print it to the console.
body = buffer.getvalue()
print(body.decode('iso-8859-1'))Your file should look like this:
from pycurl import Curl
from io import BytesIO
# Create a pycUrl instance
c = Curl()
buffer = BytesIO()
c.setopt(c.URL, 'http://pycurl.io/')
c.setopt(c.WRITEDATA, buffer)
# Make the request
c.perform()
# Close the connection
c.close()
body = buffer.getvalue()
print(body.decode('iso-8859-1'))After executing the script, you should get the following output:
Making a POST Request
To make a POST request, you will need to set the POSTFIELDS options of the cURL client object. For example, here is a request making a POST request to the JSON Placeholder API.
from io import BytesIO
from json import dumps
from pycurl import Curl
# Create a pycUrl instance
c = Curl()
buffer = BytesIO()
# Create a data dictionary
data = {
'userId': 1,
'title': 'Lorem Ipsum',
'body': 'Dolor sit amet'
}
# Encode the data to json
encoded_data = dumps(data)
# Set request options
c.setopt(c.URL, 'https://jsonplaceholder.typicode.com/posts')
c.setopt(c.HTTPHEADER, ['Accept: application/json', 'Content-Type: application/json'])
c.setopt(c.POSTFIELDS, encoded_data)
c.setopt(c.WRITEDATA, buffer)
# Make the request
c.perform()
# Close the connection
c.close()
body = buffer.getvalue()
print(body.decode('iso-8859-1'))
In the code above, I created a dictionary object with the data I will be sending as part of the request. Next, I encoded the data to JSON and sent it attached as payload to the request by setting the POSTFIELDS option to the encoded data. I also specified headers to specify the request content type and accepted response data type. When you run the code, you should get a response like this.
Writing Responses to Files
You can also pass in a file buffer to the cURL WRITEDATA option. This will write the response data to the file. The following example illustrates the concept:
from pycurl import Curl
file_name = 'output.json'
# Opening the file in write mode
with open(file_name, 'wb') as f:
# Creating a Curl instance
c = Curl()
# Set request options
c.setopt(c.URL, 'https://jsonplaceholder.typicode.com/users/1')
c.setopt(c.HTTPHEADER, ['Accept: application/json'])
c.setopt(c.WRITEDATA, f)
# Make the request
c.perform()
# Close the connection
c.close()
print(f'Wrote output to {file_name}')
Alternatives to PycURL
PycURL only provides a thin layer over the cURL library. This makes it highly customizable as you have more control over lower-level features.
However, it makes it harder to use and is, therefore, more targeted toward the advanced developer. Oftentimes, you may want a simpler alternative to PycURL. In this section, we will discuss the alternatives.
#1. Other libraries
Other than PycURL, Python has other libraries that can be used for making requests. These include the requests library ad the urllib library. Both of which are popular ad easier alternatives to pycURL.
#2. Other Languages
cURL has interfaces implemented in other languages. A popular website for converting cURL websites is Curl Converter. With the cURL Converter, you write a cURL command for the request you want to make, and it automatically converts your command to any chosen programming language. You can also just call the cURL command directly in your terminal or write a Bash script.
Final Words
In this article, I introduced cURL and explained how to use it in Python using the PycURL module. We also discussed alternatives to PycURL, such as the requests module and using different languages altogether for your programs.
Next, check out cURL command usage with real-time examples.
