How to use Proxy with cURL and wget?

Are you tired of being blocked from accessing certain websites or services?

Do you want to bypass these restrictions and access the content you want?

If your answer is yes, Then you can consider using a proxy server through the command line interface using cURL or wget tools.

First, Let’s see what exactly is proxy and how it works.

What is a Proxy Server?

A proxy server acts as a middleman between your device (like your computer or phone) and the internet.

When you want to access a website or service, your request goes to the proxy server first instead of directly to the destination.

The proxy server then sends your request to the destination server on your behalf and receives the response back from the destination server. Finally, the proxy server sends that response back to your device.

It’s kind of like having a personal assistant who takes care of all the communication between you and the destination server.

But why use a proxy server in the first place?

Well, there are a number of reasons.

Using a proxy server can help protect your privacy and anonymity. When you connect to a website or service via a proxy server, your IP address is hidden from the destination server. Instead, the server only sees the proxy IP address. This can make it more difficult for websites to track your online activities.

And…

Another reason to use a proxy server is to filter content. This can be particularly useful for organizations that want to control or restrict access to certain types of content.

Proxy servers can be configured to block access to certain websites based on specific criteria such as keywords or URL categories. This can help ensure that employees aren’t accessing inappropriate or distracting websites during work hours.

If you want to know the key differences between proxy and VPN, Feel free to visit this article.

How to Use a Proxy with cURL?

cURL is an open-source command line tool designed to transfer data between servers which makes it a go-to choice for tech wizards worldwide. Thanks to its impressive compatibility with an extensive range of protocols such as HTTP, FTP, and SMTP.

It can handle even the most complex data transfer tasks with ease. Additionally, this innovative tool is equipped with an array of secure encryption methods that help to keep data safe and secure throughout the transfer process.

cURL is readily available for use on most operating systems. If you find that it’s not installed on your system, don’t worry – you can easily download it manually. Simply follow the steps below to install cURL and unlock its full potential.

Windows

• Download the curl executable from the official website.
• Extract the zip file to a folder on your computer.
• Add the folder path to the system’s PATH environment variable.

Mac OS

• Open the terminal application.
• Install Homebrew (if you haven’t already) by running the following command.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

• Now Install cURL by running the following command.

brew install curl

Linux

On most Linux distributions, cURL is installed by default. In case it’s missing, you have to install it using the package manager specific to your distribution.

For example, on Ubuntu or Debian, you can install cURL by running the following command.

sudo apt-get install curl

On CentOS or Fedora, you can use the below command.

sudo yum install curl

Once you have installed cURL, you can test it by running the “curl –version” command in your terminal/command prompt.

Let’s move on to our main topic.

You can use a proxy with cURL by specifying the proxy server and port number in the cURL command using the -x or –proxy option.

Here’s a basic syntax of a cURL command that uses a proxy.

curl -x <proxy-server>:<port-number> <url>

Replace <proxy-server> with the hostname or IP address of the proxy server, <port-number> with the port number used by the proxy, and <url> with the URL of the website or API endpoint you want to access.

Here’s a sample example.

curl -x 5.189.184.6:80  ipinfo.io

This command sends a request to the ipinfo.io website through the proxy server at 5.189.184.6 on port 80. This sample proxy address I have taken from the publicly available proxy list. Feel free to visit this website, if you are looking for a temporary proxy address. Try at your own risk!☢️

Note that sometimes you may also need to provide authentication credentials to the proxy server, in which case you can use the “--proxy-user” and the “--proxy-password” options to include your username and password. For example:

curl -x  5.189.184.6:80 --proxy-user username:password  <target-url>

Replace the “username” and “password” with your actual authentication credentials for the proxy server.

Testing the proxy connection with cURL

Before using proxy

Here, You can see cURL is showing my original IP address and location. For security reasons, I had to hide my IP address.

After using a proxy

Here, the cURL is showing the proxy address instead of my original IP address.

How to use proxy with wget?

wget is a command-line tool used for downloading files from the internet. It comes pre-installed on most Linux, Unix, and macOS systems and is also available for Windows.

To install wget on a Linux system, you can use the package manager of your distribution.

For example, on Debian or Ubuntu, you can use the following command:

sudo apt-get install wget

On CentOS or Fedora

sudo yum install wget

On macOS, you can install wget using Homebrew.

brew install wget

On Windows, you can install wget by downloading the binary executable from the official GNU website and adding it to your system path.

Once you have installed wget, you can use it to download files from the command line by specifying the URL of the file you want to download. For example, To download a file named “example.zip” from a website, you can use the following command:

wget http://example.com/example.zip

You can also use additional options to customize the download such as specifying the output filename or directory, limiting the download speed,  and resuming interrupted downloads. To learn more about the available options, Feel free to use the wget manual by typing “man wget” in your terminal.

Let’s take a dive into the configuration of proxy using this wget.

First, Set the environment variables for the proxy server by running the following command in the terminal.

export http_proxy=http://<proxy_server>:<port>
export https_proxy=https://<proxy_server>:<port>

Test if the proxy server is properly configured by running the following command:

wget <target-url>

Replace <target-url> with the URL of the file or webpage you want to download using the proxy server. If the command succeeds in downloading the file or webpage, then the proxy server is properly configured. If it fails, then you may need to check your proxy server settings and credentials.

(Optional) To make the proxy server settings permanent, you can add the environment variables to the shell profile file such as ~/.bashrc or ~/.bash_profile. You can add the following lines at the end of the file:

export http_proxy=http://<proxy_server>:<port>
export https_proxy=https://<proxy_server>:<port>

If your proxy server requires authentication, you can include your username and password in the “http_proxy” and “https_proxy environment” variables.

export http_proxy=http://<username>:<password>@<proxy_server>:<port>
export https_proxy=https://<username>:<password>@<proxy_server>:<port>

Note: Some proxy servers may use a different protocol scheme such as socks instead of “http” or “https“. In that case,  you need to adjust the scheme accordingly in the environment variables and the URLs.

Testing proxy connection with wget

Before using proxy

wget is showing my original IP address before using a proxy address. For security reasons, I had to hide my IP address.

After using proxy

Here you can see, wget is trying to connect to a proxy server before accessing the website.

Troubleshooting Common Issues

Using a proxy server with cURL and wget can sometimes result in issues. Fear not, we have some common troubleshooting tips to get you back on track!

#1. Proxy Server Not Responding

If your proxy server is not responding, it may be down or experiencing issues. To figure out what’s going on, try the following.

• Check if the proxy server is up by pinging the server IP address.
• Make sure that you have entered the correct proxy server address and port number.
• Try using a different proxy server to see if the issue persists.

#2. Connection Refused

Oh, connection refused?

That’s not good.

This error typically indicates that the proxy server is not accepting connections. Here are some steps to troubleshoot this issue:

• First, Check if the proxy server is configured to accept connections from your IP address or subnet.
• Ensure that the proxy server is not being blocked by a firewall or antivirus software.
• And next, check if the proxy server requires authentication credentials and ensure that you have entered the correct credentials.

Don’t forget to try a different proxy server if none of these tips work.

#3. SSL/TLS Certificate Verification Failed

certificate verification failed?

Sounds like a problem with the SSL/TLS certificate of the proxy server.

To troubleshoot this issue, try the following:

• Check the certificate of the proxy server to ensure that it is valid and has not expired.
• If the certificate is invalid, contact the proxy server administrator to obtain a valid SSL/TLS certificate.
• If the certificate is valid and still it’s not working. You can disable SSL/TLS certificate verification, but be careful that doing so can put your security at risk.

#4. Authentication Failed

If the proxy server requires authentication and you’re unable to authenticate, it’s probably because you’ve entered the wrong username and password, or the proxy server is using a different authentication method.

Try adjusting your authentication credentials accordingly or contact the proxy server administrator for help.

Author’s note✍️

There are many free and paid proxy providers available on the internet, but one recommended option for commercial use is IPRoyal. IPRoyal offers a wide range of proxy solutions for businesses, including residential and data center proxies with multiple locations & IP types to choose from.

It provides reliable and secure proxy services for various use cases, such as web scraping, data mining, and automation.

I hope you found this article helpful in learning about using proxy with cURL and wget command line tools. You may also be interested in learning how to anonymize Linux traffic using proxy chains and the Tor network.