Web scraping using Python is a common way to collect information from websites for research, automation, content analysis, price tracking, and AI applications. Python offers a simple and flexible way to retrieve website content. Its rich library ecosystem and ease of use have made it a popular choice for web scraping.

In this tutorial, I will show you how to scrape website content with Python, understand the API response, choose the right output format, and use AI to extract only the information you need from a webpage. I’ll also explain how to test your requests before integrating them into your application.

Prerequisites

Before you begin, make sure you have:

• A Geekflare API key. Sign in to your Geekflare account and copy your API key from the dashboard.
• Python 3.8 or later installed.
• The Geekflare Python SDK.

Install the SDK using:

pip install geekflare-api

Create Your First Web Scraper Using Python

The Geekflare Python SDK lets you scrape website content with a few lines of Python code. In this example, we’ll scrape the Hacker News homepage and return the response in JSON format.

Send a Web Scraping Request

Create a new Python file, such as main.py, and add the following code:

from geekflare_api.client import GeekflareClient
from geekflare_api.models import WebScrapeDto

with GeekflareClient(
    api_key="<api-key>"  # Replace with your Geekflare API key.
) as client:
    result = client.web_scrape(
        WebScrapeDto(
            device="desktop",
            format=["json"],  # Change the output format based on your requirements.
            render_j_s=True,
            block_ads=True,
            stealth=False,
            url="https://news.ycombinator.com/",
            file_output=False
        )
    )

    print(result)

Replace <api-key> with your Geekflare API key. You can also replace the URL with any public website that you want to scrape.

Understand the Request Parameters

When performing web scraping using Python, you can customize how the API retrieves and returns website content by changing the following request parameters.

Run the Script

Open a terminal in your project directory and run:

python main.py

(If you saved the file with a different name, replace main.py with the name of your Python file)

If the request is successful, the API returns the scraped website content in the selected format along with metadata about the request.

Understand the Web Scraping Response

After you run the script, the API returns a JSON response that contains the scraped website content and additional metadata about the request.

The response includes the following fields:

• success: Indicates whether the request completed successfully.
• timestamp: Shows when the API processed the request.
• apiStatus: Returns the status of the API request.
• apiCode: Returns the HTTP status code. A value of 200 indicates that the request completed successfully.
• meta: Contains information about the request, including the website URL and the selected output format.
• data: Contains the scraped website content. In this example, the response is returned in JSON format and includes the extracted content from the Hacker News homepage.

Extract Specific Information with AI

Usually, web scraping returns the entire webpage. But often, you only need specific information from the page.

To solve this, the Geekflare Web Scraping API includes a feature called AI Extraction. It extracts only the information you request from the webpage, reducing the amount of post-processing required in your application.

AI Extraction supports the following extraction types:

For a complete explanation of each extraction type and additional examples, refer to our guide on AI Extraction with the Geekflare Web Scraping API.

Example: Ask Questions About a Webpage

The following example uses the Open-ended Question extraction type to answer a question from a Wikipedia article.

from geekflare_api.client import GeekflareClient
from geekflare_api.models import WebScrapeDto

with GeekflareClient(
    api_key="<api-key>"  # Replace with your Geekflare API key.
) as client:
    result = client.web_scrape(
        WebScrapeDto(
            device="desktop",
            format=["json"],
            render_j_s=True,
            block_ads=True,
            stealth=False,
            ai_prompt={
                "type": "prompt",  # AI extraction type.
                "query": "What are the key features of Claude Fable?"  # Question to answer from the webpage.
            },
            url="https://www.anthropic.com/claude/fable",
            file_output=False
        )
    )

    print(result)

Understand the ai_prompt Parameter

The ai_prompt parameter controls how AI extraction processes the webpage.

After running the script, the API analyzes the webpage and returns the answer based on the selected AI extraction type.

The structure of the ai_prompt parameter changes based on the selected AI extraction type. Each extraction type accepts different inputs depending on the information you want to extract.

Refer to the AI Extraction documentation to see how the ai_prompt parameter changes for each supported extraction type.

Use the Geekflare Playground: A No-Code Platform

If you don’t want to write code, you can use the Geekflare Playground, a no-code platform for testing the Web Scraping API. It lets you configure the same request parameters available in the Python SDK through a simple interface.

Enter the website URL, choose the output format, select the device type, and configure options such as Render JS, Block Ads, Stealth Mode, File Output, Proxy Country, and AI Extraction Type. After configuring the request, click Send Request.

For this example, select Product under AI Extraction Type and use the following product page:

https://www.apple.com/shop/buy-iphone/iphone-16

The Playground extracts the product details and displays the response instantly. You can also click Get Code to generate ready-to-use code snippets in Python, cURL, Node.js, PHP, and other supported languages using the same configuration.

Which Option Should You Choose?

Both the Geekflare Python SDK and the Geekflare Playground support the same Web Scraping API. The best choice depends on how you plan to use it.

Conclusion

Web scraping using Python helps you collect website data for automation, research, AI applications, and data analysis. The Geekflare Python SDK makes it easy to scrape website content, while AI Extraction lets you retrieve only the information you need from a webpage.

To learn more about the Web Scraping API, supported parameters, AI Extraction types, and SDKs, refer to the official Geekflare documentation.

Frequently Asked Questions