Crawler and sitemap generatorPHP Crawler

Crawl. Analyze. Generate.

A PHP crawler that walks a site concurrently and writes production-ready XML sitemaps, honoring canonical, noindex and nofollow.

Try the demo Source code

Year: 2026
Role: API design, implementation, benchmarking and demo page.
Stack: PHP 8.4
cURL multi
DOM
XMLWriter

The problem

Many sites don’t generate their own sitemap, and online tools cap the number of pages or don’t fit into an automated deployment process.

By the numbers

50kURLs per file before rotating
6events across the crawl lifecycle
4swappable parts: HTTP, logger, analyzer and writer
3indexing rules honored: canonical, noindex and nofollow

What was built

A curl_multi worker pool that fetches several pages at once, with a sequential fallback.
URL normalization: resolves relative paths, strips fragments and only processes text/html responses.
Honors robots meta tags (noindex, nofollow) and canonical URLs, and can stay on the same host and scheme.
A rotating writer that starts a new file every 50,000 URLs and writes sitemap-index.xml.
Events at every crawl step, with a swappable HTTP client, logger and page analyzer.

In practice

<?php

use Tonsoo\PhpCrawler\Extensions\SitemapExtension;
use Tonsoo\PhpCrawler\Sitemap\SitemapGenerator;
use Tonsoo\PhpCrawler\Sitemap\Writers\RotatingSitemapWriter;

crawler()
    ->multithreaded(workers: 8)
    ->preserveHost()
    ->maxPages(1000)
    ->extension(new SitemapExtension(
        generator: new SitemapGenerator(
            writer: new RotatingSitemapWriter(directory: __DIR__ . '/sitemap'),
        ),
    ))
    ->start('https://example.com');

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap-1.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap-2.xml</loc></sitemap>
</sitemapindex>

Screens

Docs with the usage example next to the explanation.
Demo: enter a URL, tune the rules and generate the sitemap.

Outcome

Started in 2024 as a script and was rewritten in 2026 as an extensible library, ready to run in CI or as a scheduled task.