Claude Code for ZenRows Scraping Workflow Tutorial

Web scraping has evolved significantly with the emergence of AI-powered tools. ZenRows provides a powerful API that handles anti-bot protection, CAPTCHA solving, and proxy rotation—making it an excellent choice for developers who need reliable scraping capabilities. When combined with Claude Code, you can create intelligent, maintainable scraping workflows that adapt to dynamic websites.

This tutorial walks you through building a complete ZenRows scraping workflow using Claude Code. You’ll learn how to set up the integration, handle common scraping challenges, and create reusable code that scales.

Prerequisites and Environment Setup

Before diving into the code, ensure you have Node.js installed and a ZenRows API key. If you haven’t already, sign up at ZenRows to get your API key.

Installing Required Dependencies

Create a new directory for your project and install the necessary packages:

mkdir zenrows-scraper && cd zenrows-scraper
npm init -y
npm install axios dotenv

Create a .env file to store your API key securely:

ZENROWS_API_KEY=your_zenrows_api_key_here

Start an interactive Claude Code session in your project directory:

claude

You can create a CLAUDE.md file where you define project-specific instructions for Claude Code, which it will read automatically at session start.

Building Your First ZenRows Scraper

Now let’s create a basic scraper that uses ZenRows to fetch webpage content. Create a file called scraper.js:

require('dotenv').config();
const axios = require('axios');

class ZenRowsScraper {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.baseUrl = 'https://www.zenrows.com/v1';
  }

  async scrape(url, options = {}) {
    try {
      const response = await axios.post(
        `${this.baseUrl}/`,
        {
          url: url,
          js_rendering: options.jsRendering || true,
          premium_proxy: options.premiumProxy || false,
          country: options.country || 'us',
        },
        {
          headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${this.apiKey}`,
          },
        }
      );

      return response.data;
    } catch (error) {
      console.error('Scraping error:', error.message);
      throw error;
    }
  }
}

module.exports = ZenRowsScraper;

This basic class provides the foundation for all your scraping needs. The key parameters include js_rendering for JavaScript-heavy sites, premium_proxy for difficult targets, and country for geo-specific requests.

Integrating Claude Code for Smart Scraping

The real power emerges when you combine ZenRows with Claude Code. Create a CLAUDE.md file to guide Claude Code on your scraping project:

# ZenRows Scraping Project

## Project Overview
This project uses ZenRows API for web scraping with Claude Code assistance.

## Key Files
- `scraper.js` - Main scraping class
- `parser.js` - Content extraction utilities
- `workflow.js` - Orchestration scripts

## Guidelines
- Always respect robots.txt when possible
- Add rate limiting between requests
- Store results in structured JSON format
- Log all scraping activities for debugging

Now you can ask Claude Code to help with specific scraping tasks:

“Help me extract product information from this e-commerce page, including name, price, and availability.”

Claude Code can analyze the HTML structure and create targeted extraction functions.

Handling Dynamic Content

Many modern websites use JavaScript to load content dynamically. ZenRows handles this with its JavaScript rendering feature. Here’s an enhanced scraper that handles various dynamic content scenarios:

class DynamicScraper extends ZenRowsScraper {
  async scrapeWithRetry(url, maxRetries = 3) {
    let lastError;
    
    for (let attempt = 1; attempt <= maxRetries; attempt++) {
      try {
        console.log(`Attempt ${attempt} for ${url}`);
        
        const result = await this.scrape(url, {
          jsRendering: true,
          waitFor: '#content-loaded',
          waitTimeout: 10000,
        });
        
        return result;
      } catch (error) {
        lastError = error;
        console.log(`Attempt ${attempt} failed: ${error.message}`);
        
        if (attempt < maxRetries) {
          await this.delay(2000 * attempt);
        }
      }
    }
    
    throw new Error(`Failed after ${maxRetries} attempts: ${lastError.message}`);
  }

  delay(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

This pattern handles transient failures gracefully, which is crucial for production scraping systems.

Creating Reusable Extraction Functions

One of Claude Code’s strengths is generating targeted extraction logic. Here’s how to structure your extraction utilities:

class ContentExtractor {
  static extractProducts(html) {
    const products = [];
    const productRegex = /<div class="product"(.*?)>/gi;
    let match;
    
    while ((match = productRegex.exec(html)) !== null) {
      const productBlock = match[1];
      
      products.push({
        name: this.extractField(productBlock, 'product-name'),
        price: this.extractField(productBlock, 'product-price'),
        url: this.extractField(productBlock, 'product-link', true),
      });
    }
    
    return products;
  }

  static extractField(html, className, isAttribute = false) {
    const regex = new RegExp(
      isAttribute 
        ? `${className}="([^"]+)"`
        : `<span class="${className}">([^<]+)</span>`,
      'i'
    );
    
    const match = html.match(regex);
    return match ? match[1].trim() : null;
  }
}

Ask Claude Code to adapt these patterns for specific websites by describing the HTML structure you’re working with.

Building Production Workflows

For larger scraping projects, create orchestration scripts that manage multiple pages:

class ScraperWorkflow {
  constructor(scraper) {
    this.scraper = scraper;
    this.results = [];
  }

  async scrapeMultiple(urls, concurrency = 3) {
    const chunks = this.chunkArray(urls, concurrency);
    
    for (const chunk of chunks) {
      const promises = chunk.map(url => 
        this.scraper.scrapeWithRetry(url)
          .then(result => ({ url, success: true, data: result }))
          .catch(error => ({ url, success: false, error: error.message }))
      );

      const chunkResults = await Promise.all(promises);
      this.results.push(...chunkResults);
      
      await this.scraper.delay(1000);
    }

    return this.results;
  }

  chunkArray(array, size) {
    const chunks = [];
    for (let i = 0; i < array.length; i += size) {
      chunks.push(array.slice(i, i + size));
    }
    return chunks;
  }
}

This workflow processes URLs in controlled batches, respecting rate limits while maximizing throughput.

Best Practices for Claude Code Scraping Projects

When working on ZenRows projects with Claude Code, keep these practices in mind:

Error Handling: Always implement comprehensive error handling. Network requests fail for numerous reasons, and your scraper should handle each case gracefully.

Rate Limiting: ZenRows provides different tier plans with varying rate limits. Implement exponential backoff to avoid hitting these limits.

Data Validation: After scraping, validate the extracted data. Claude Code can help you write validation logic that ensures data quality before processing.

Logging: Maintain detailed logs of your scraping activities. This helps debug issues and provides audit trails for compliance purposes.

Conclusion

Combining Claude Code with ZenRows creates a powerful scraping solution that balances simplicity with sophistication. The ZenRows API handles the complex anti-bot mechanisms, while Claude Code helps you write maintainable, adaptive extraction logic.

Start with the basic scraper class, then progressively add features like retry logic, extraction utilities, and workflow orchestration. As your needs grow, Claude Code can help you refactor and scale your scraping infrastructure.

Remember to always scrape responsibly and respect the websites you target. With the right approach, you’ll have a reliable, AI-assisted scraping system that serves your data needs while maintaining ethical standards.

Built by theluckystrike — More at zovo.one