The Practical Developer

Web Scraping with Node.js and Playwright: When the API Does Not Exist

July 7, 2026 by The Practical Developer

Building a production-grade web scraper with Playwright in Node.js: rate limiting, error recovery, stealth patterns, and data extraction that survives DOM changes.

nodejs

playwright

automation

data-engineering

Server room with blinking lights representing automated data gathering infrastructure

Your competitor launched a pricing page update. The documentation site you depend on got redesigned and all your CSS selectors broke. The SaaS platform you integrate with still has no public API and the CEO wants a demo by Friday.

Web scraping is the duct tape of data integration. It is ugly, it breaks constantly, and it is the difference between shipping and waiting. The question is not whether to scrape. The question is whether your scraper will survive the first DOM change or quietly return null for three weeks while your dashboard shows zeros.

This post walks through a production-grade scraping pipeline built on Playwright and Node.js. Every pattern here was learned the hard way: after a scraper broke in production, after a captcha blocked a crawl at 2 AM, after a rate limit triggered a permanent IP ban.

Why Playwright over bare HTTP requests

A surprising number of scraping projects start with axios and cheerio. That works for server-rendered HTML. It fails on anything that runs JavaScript in the browser.

Modern sites render their content through React, Vue, Angular, or Next.js. The DOM you get from a plain HTTP request is an empty shell with <div id="root"></div>. Playwright runs a real browser engine (Chromium, Firefox, or WebKit), executes JavaScript, and gives you the fully rendered DOM.

The tradeoff is resource usage. A browser context eats 100-200 MB of RAM and takes seconds to start. You carry that cost willingly because the alternative is empty data.

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

await page.goto('https://example.com/data', {
  waitUntil: 'networkidle',
  timeout: 30000
});

// Wait for the data table to actually render
await page.waitForSelector('table.data-table', { timeout: 10000 });

const rows = await page.$$eval('table.data-table tr', (rows) => {
  return rows.map((row) => {
    const cells = row.querySelectorAll('td, th');
    return Array.from(cells).map((cell) => cell.textContent.trim());
  });
});

console.log(rows);
await browser.close();

That is the hello world version. It works until the server sends a 429, the table takes 15 seconds to load because of a slow API call, or the site redirects you to a captcha page.

The three-layer architecture

A scraper that runs once on your laptop is a script. A scraper that runs unattended for months is a system with three layers.

Layer 1: Navigation and extraction. Playwright opens pages, waits for content, and extracts data. This is the thinnest layer. It should contain almost no logic beyond “go here, wait for this, pull that.”

Layer 2: Orchestration and resilience. Retries with backoff, rate limiting between requests, rotating user agents and viewports, detecting captchas and blocked pages, logging every failure with enough context to debug it.

Layer 3: Storage and diffing. Save raw HTML, extracted data, and metadata. Detect when extracted data changes structure (missing columns, new fields, different types) before it breaks downstream consumers.

Most scraper rewrites happen because people bake orchestration into extraction and end up with a 600-line function that is impossible to test or debug.

Rate limiting and polite crawling

The fastest way to get blocked is to fire 50 requests in parallel. Playwright makes it easy to open multiple pages in the same browser context, but that does not mean you should.

class PoliteCrawler {
  constructor(minDelay = 2000, maxDelay = 5000) {
    this.minDelay = minDelay;
    this.maxDelay = maxDelay;
    this.lastRequestTime = 0;
  }

  async waitForSlot() {
    const now = Date.now();
    const elapsed = now - this.lastRequestTime;
    const delay = Math.max(
      this.minDelay - elapsed,
      Math.random() * (this.maxDelay - this.minDelay)
    );

    if (delay > 0) {
      await new Promise((resolve) => setTimeout(resolve, delay));
    }

    this.lastRequestTime = Date.now();
  }
}

The random jitter between minDelay and maxDelay is not cosmetic. Sites detect perfectly regular request intervals and flag them as bot traffic. Adding 2-4 seconds of random wait drops your throughput by maybe 30% and multiplies your scraper’s lifespan by 10x.

Respect robots.txt if the target site has one. Parse it with the robots-parser npm package and skip disallowed paths. It costs nothing and keeps your scraper in the gray zone rather than the black one.

Stealth: looking like a real browser

Playwright’s headless mode is detectable. Sites check navigator.webdriver, the presence of window.chrome, missing plugins, and a dozen other signals. The playwright-extra package with the stealth plugin patches most of these.

import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';

chromium.use(stealth());

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

// Rotate user agents across requests
const USER_AGENTS = [
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ...',
  'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ...',
];

await page.setUserAgent(
  USER_AGENTS[Math.floor(Math.random() * USER_AGENTS.length)]
);

// Set realistic viewport
await page.setViewportSize({
  width: 1366 + Math.floor(Math.random() * 200),
  height: 768 + Math.floor(Math.random() * 200),
});

Do not set the same viewport every time. Do not use the default user agent. These are the scraping equivalent of wearing the same clothes every day and wondering why the security guard recognizes you.

Detecting and handling captchas and blocks

A scraper that does not know it is being blocked will extract the captcha page’s HTML as if it is real data. This is the most common silent failure in production scrapers.

async function detectBlock(page) {
  const url = page.url();
  const title = await page.title();
  const bodyText = await page.evaluate(() => document.body.innerText);

  const blockSignals = [
    'captcha',
    'are you a human',
    'access denied',
    'sorry, you have been blocked',
    'please verify you are a human',
    'too many requests',
  ];

  const blocked = blockSignals.some(
    (signal) =>
      title.toLowerCase().includes(signal) ||
      bodyText.toLowerCase().includes(signal)
  );

  const statusCode = await page.evaluate(() => {
    return (window as any).__statusCode || null;
  });

  return { blocked, reason: blocked ? title : null, statusCode };
}

Integrate this check after every page.goto() call. If the page is blocked, log the full HTML for later analysis, increment a backoff counter, and wait longer before retrying.

async function navigateWithResilience(page, url, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 });

    const { blocked, reason } = await detectBlock(page);

    if (!blocked) return;

    const backoff = Math.pow(4, attempt) * 1000 + Math.random() * 2000;
    console.warn(`Blocked on attempt ${attempt}: ${reason}. Waiting ${backoff}ms`);
    await new Promise((r) => setTimeout(r, backoff));
  }

  throw new Error(`Failed to navigate to ${url} after ${maxRetries} attempts`);
}

The exponential backoff uses base 4 (16s, 64s, 256s) instead of the common base 2. Captcha pages and rate limiters usually reset within 30-60 seconds. A 16-second wait catches most soft blocks. The 256-second wait on the third attempt is the last resort before the scraper gives up and logs the failure for manual review.

Data extraction that survives redesigns

The biggest maintenance cost in scraping is broken selectors. Every time the target site updates its CSS classes or DOM structure, your scraper returns null or wrong data.

Mitigate this with three strategies.

Strategy 1: Use data attributes when they exist.

// Fragile: depends on CSS class names
const price = await page.$eval('.product-card .price--large', (el) => el.textContent);

// More robust: data attributes are more stable
const price = await page.$eval('[data-testid="product-price"]', (el) => el.textContent);

Data attributes like data-testid, data-product-id, or data-price are often stable across redesigns because testing frameworks depend on them.

Strategy 2: Extract multiple selector candidates and vote.

async function extractPrice(page) {
  const candidates = [
    '[data-testid="product-price"]',
    '[class*="price"]',
    '.product-price',
    '.price-value',
    '[itemprop="price"]',
  ];

  for (const selector of candidates) {
    const el = await page.$(selector);
    if (el) {
      const text = await el.textContent();
      const cleaned = text.replace(/[^0-9.,]/g, '');
      if (cleaned.length > 0) return cleaned;
    }
  }

  return null;
}

This degrades gracefully. If the primary selector breaks, the fallback catches it. Log which selector succeeded so you know when the primary one stops matching.

Strategy 3: Snapshot the full DOM for every run.

const html = await page.content();
await fs.writeFile(`snapshots/${timestamp}.html`, html);

When the extraction breaks, the raw HTML snapshot is your forensic evidence. Without it, you are debugging blind.

Session management and authentication

Many scraping targets require login. Managing sessions in Playwright is straightforward but has traps.

// Save session state after login
const context = await browser.newContext();
const page = await context.newPage();

await page.goto('https://example.com/login');
await page.fill('[name="email"]', process.env.SCRAPER_EMAIL);
await page.fill('[name="password"]', process.env.SCRAPER_PASSWORD);
await page.click('[type="submit"]');
await page.waitForNavigation();

// Persist cookies and localStorage
await context.storageState({ path: 'session.json' });

// Next run: restore session without logging in again
const context2 = await browser.newContext({
  storageState: 'session.json',
});

The trap is session expiry. Sessions that last 24 hours will fail silently on day two. Always wrap authenticated navigation in a try-catch that detects a redirect to the login page and re-authenticates.

async function ensureAuthenticated(page, sessionPath) {
  const url = page.url();
  if (url.includes('/login') || url.includes('/auth')) {
    console.warn('Session expired, re-authenticating');
    // Re-login flow here
    await page.context().storageState({ path: sessionPath });
  }
}

Structuring the output for downstream consumers

Raw scraped data is useless without schema and metadata. Every extraction run should produce structured records that include when the data was collected and whether all fields extracted successfully.

function buildRecord(raw, metadata) {
  return {
    extractedAt: new Date().toISOString(),
    source: metadata.url,
    status: metadata.fields.every((f) => f.found) ? 'complete' : 'partial',

    // Only include fields in the specified order
    data: {
      title: raw.title || null,
      price: raw.price || null,
      availability: raw.availability || null,
      specs: raw.specs || [],
    },

    // Track which fields were actually found
    fields: metadata.fields,
    errors: metadata.errors,
  };
}

Write these records to a staging table in Postgres or to newline-delimited JSON files. Do not transform the data during extraction. Extraction and transformation are separate stages. Mixing them makes it impossible to re-transform historical data without re-scraping.

The complete pipeline skeleton

Here is how all the pieces fit together in a production scraper.

import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';
import fs from 'fs/promises';

chromium.use(stealth());

class ScraperPipeline {
  constructor(config) {
    this.config = config;
    this.crawler = new PoliteCrawler(config.minDelay, config.maxDelay);
  }

  async run() {
    const browser = await chromium.launch({ headless: true });

    try {
      for (const url of this.config.urls) {
        const context = await browser.newContext({
          userAgent: this.randomUserAgent(),
          viewport: this.randomViewport(),
        });

        const page = await context.newPage();

        try {
          await this.crawler.waitForSlot();
          await navigateWithResilience(page, url);

          const raw = await this.config.extractor(page);
          const record = buildRecord(raw, {
            url,
            fields: this.config.fields,
          });

          await this.config.storage.save(record);
          await fs.writeFile(
            `snapshots/${Date.now()}-${this.safeFilename(url)}.html`,
            await page.content()
          );
        } catch (err) {
          console.error(`Failed to scrape ${url}:`, err.message);
          await this.config.failureHandler(url, err);
        } finally {
          await context.close();
        }
      }
    } finally {
      await browser.close();
    }
  }

  randomUserAgent() { /* ... */ }
  randomViewport() { /* ... */ }
  safeFilename(url) { /* ... */ }
}

The pipeline is generic. The extractor function is the only site-specific code. To scrape a new site, you write a new extractor and plug it into the same pipeline.

Monitoring: know when it breaks

A scraper that fails silently is worse than no scraper. Set up three alerts.

Alert 1: Zero records extracted. If a run produces zero records when it should produce hundreds, something fundamental broke. This should fire within one run cycle.

Alert 2: Field coverage dropped. Track the percentage of successfully extracted fields per run. If coverage drops below 90%, log it and alert. This catches partial selector breakage before it becomes a full outage.

Alert 3: Extraction time variance. If a page that normally takes 5 seconds to scrape suddenly takes 60 seconds, something changed. Could be a slow API, a new animation, or an A/B test that loads different DOM.

// Simple health check emitted after each run
const healthReport = {
  runId: crypto.randomUUID(),
  timestamp: new Date().toISOString(),
  urlsAttempted: 50,
  urlsSucceeded: 48,
  successRate: 0.96,
  totalRecords: 1240,
  fieldCoverage: 0.94,
  avgExtractionTime: 3400,
  errors: [
    { url: 'https://example.com/product/42', error: 'Timeout: selector not found' },
  ],
};

Push this to your existing monitoring infrastructure (Prometheus, a webhook, or a Slack channel). If your scraper runs in a CI pipeline, fail the pipeline when success rate drops below a threshold.

When not to scrape

Scraping is a means of last resort. Before you write a single Playwright selector, check:

  • Does the site have an official API? Even a limited one is better than scraping.
  • Does the site have an RSS feed or sitemap? Sometimes the same data is available in a structured format.
  • Is there a third-party API that aggregates this data? Paying $50/month for an API is cheaper than maintaining a scraper.
  • Are you violating the site’s terms of service? Legal risk is real. Some companies have been sued over scraping.

If you must scrape, be polite, be respectful of the site’s resources, and extract only what you need. Do not scrape data you are not going to use.

The practical takeaway

Start every scraping project by writing the extractor function first. Run it against a single page in a REPL. Get the selectors right with a known snapshot. Then wrap it in the orchestration layer. Then add storage and monitoring.

The orchestrator should be generic enough to reuse across targets. The extractor should be small enough to rewrite in 30 minutes when the target site changes. That is the sweet spot. Any more coupling and every DOM change becomes a pipeline rewrite. Any less structure and you have a script that works once and fails forever.

A note from Yojji

The kind of work this post describes (building resilient data pipelines, handling edge cases in automated extraction, designing systems that survive upstream changes without breaking) is the difference between a prototype and a production service. It is also the kind of pragmatic backend engineering that Yojji has been shipping since 2016.

Yojji is an international custom software development company with offices in Europe, the US, and the UK. Their teams specialize in the JavaScript stack (React, Node.js, TypeScript), cloud platforms (AWS, Azure, Google Cloud), and microservices architectures. They run dedicated senior outstaffed teams alongside full-cycle product engagements covering discovery, design, development, QA, and DevOps.

If your team needs a reliable data pipeline built the right way (with proper error handling, monitoring, and maintainability baked in from day one), Yojji is worth a conversation.

Keep Reading