Scrape The Web For Free

The AI Surfer

Web scraping is a perfect example of the 90% — the repetitive, technical, time-consuming work that eats your day before you ever get to the part that actually matters. The AI Surfer is the community Dallas built around flipping that ratio. The whole model is simple: identify the work AI should be doing for you, automate it, and reclaim your time for the 10% only you can do.

ScrapeGraph AI is exactly the kind of tool the community runs on. Free. Open source. No code required if you use it right.

If you want a full system — not just individual tools — The AI Surfer is where that gets built.

What Is ScrapeGraph AI?

GitHub: github.com/ScrapeGraphAI/Scrapegraph-ai

ScrapeGraph AI is a free, MIT-licensed Python library that uses an LLM to scrape any website with a single plain-English prompt. No CSS selectors. No XPath. No brittle rules that break when a site updates.

You give it a URL + a prompt. It returns structured data. That's the whole thing.

Step 1 — Install It (2 minutes)

Open your terminal and run:

pip install scrapegraphai
playwright install

Step 2 — Pick Your LLM

You need an LLM to power the extraction. Two options:

• Grab your API key from platform.openai.com or console.anthropic.com
• Cost per scrape is pennies for most use cases

• Install Ollama at ollama.com
• Run: ollama pull llama3.2
• Keeps everything on your machine

Step 3 — Run Your First Scrape

from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {
        "api_key": "YOUR_OPENAI_API_KEY",
        "model": "openai/gpt-4o-mini",
    },
    "verbose": True,
    "headless": False,
}

smart_scraper_graph = SmartScraperGraph(
    prompt="Extract all company names, founder names, and LinkedIn URLs from this page",
    source="https://yourTargetURL.com",
    config=graph_config
)

result = smart_scraper_graph.run()
print(result)

graph_config = {
    "llm": {
        "model": "ollama/llama3.2",
        "model_tokens": 8192,
        "format": "json",
    },
    "verbose": True,
    "headless": False,
}

Swap the config. Everything else stays the same.

Step 4 — Let Claude Code or Codex Build It For You

If you don't want to write the script yourself, paste this into Claude Code or OpenAI Codex:

Using the ScrapeGraphAI Python library with OpenAI GPT-4o-mini,
write a script that scrapes [TARGET URL] and extracts [WHAT YOU WANT].
Output the results as a clean JSON file.

Claude Code or Codex will write the full working script. You paste it, run it, done.

This is the real unlock — you don't need to know Python. You just need to know what data you want.

What You Can Scrape

• LinkedIn profiles — names, titles, companies, bios
• Email directories — contact pages, business listings
• Amazon listings — prices, reviews, product specs
• Zillow — property data, agent contacts, prices
• Glassdoor — employer ratings, review sentiment
• Twitter/X — handles, bios, follower data
• Any multi-page dataset — paginated results, entire domains

Available Scraping Pipelines

• SmartScraperGraph — single page, one prompt, one source
• SmartScraperMultiGraph — multiple URLs, one prompt, parallel calls
• SearchGraph — scrapes top N results from a search query
• ScriptCreatorGraph — generates a reusable Python scraping script for you
• SpeechGraph — extracts data and outputs an audio file

For most use cases, SmartScraperGraph is all you need.

The Agency Play

Lead gen operators are already charging $2,000–$5,000 per scrape job using this exact tool:

1. Client wants a targeted list — industry, geo, job title, platform
2. You run ScrapeGraph AI against the relevant sources
3. Clean structured JSON delivered in hours
4. Invoice sent

One laptop. Zero platform fees. The margin is almost entirely yours.

Quick Notes

• The tool is free — but if you use OpenAI/Anthropic, inference costs apply (typically cents per run)
• Local models via Ollama = truly $0 — performance varies on complex pages
• No GUI — this is a Python library; use Claude Code or Codex if you want to skip writing code yourself
• Heavy-scale scraping — add basic proxy rotation to avoid rate limiting at volume

GitHub: github.com/ScrapeGraphAI/Scrapegraph-ai — clone it, read the README, run your first prompt.