Optimize your WordPress robots.txt with Cloudflare: Automated script for clean SEO reports

Why keeping your robots.txt up-to-date matters when using Cloudflare—and how to automate it across your WordPress installations.

If your WordPress sites use Cloudflare as a CDN or security layer, you may have noticed recurring crawl errors or anomalies in SEO reports. These issues often originate from the /cdn-cgi/ endpoint—a system path used internally by Cloudflare.

While these “errors” don’t affect SEO rankings or page performance, they can distort SEO reporting and hinder accurate website analysis.

Cloudflare’s Official Recommendation

As documented in Cloudflare’s official guidance, the recommended solution is simple: update your robots.txt file to disallow search engine access to the /cdn-cgi/ directory.

Disallow: /cdn-cgi/

This directive prevents crawlers from indexing this path, reducing clutter in reports and focusing indexing efforts on valuable site content.

A Scripted Solution for WordPress Users

To simplify the process—especially for developers or sysadmins managing multiple WordPress sites—an automated script is available to handle everything:

• Recursively scans directories for WordPress installations
• Checks for existing robots.txt files
• Adds the Cloudflare directive if missing
• Creates a properly formatted robots.txt if not found

Available on GitHub:
👉 WordPress-Cloudflare-robots.txt-Updater

Installation & Usage

1. Quick Install via curl

curl -O https://raw.githubusercontent.com/dcarrero/WordPress-Cloudflare-robots.txt-Updater/refs/heads/main/update_robots_cloudflare.sh
chmod +x update_robots_cloudflare.sh

2. Or Clone the Repo

git clone https://github.com/dcarrero/WordPress-Cloudflare-robots.txt-Updater.git
cd WordPress-Cloudflare-robots.txt-Updater
chmod +x update_robots_cloudflare.sh

3. Run the Script

Run in the current directory:

./update_robots_cloudflare.sh

Or specify a base path:

./update_robots_cloudflare.sh /var/www

Script Features

• Safe & Targeted: Only modifies folders with valid WordPress installations
• Smart Handling: Checks for duplicates before adding the directive
• Auto Formatting: Maintains clean line endings and comments
• Verbose Logging: Every step is printed to the terminal for transparency
• Optimized Scanning: Skips common WordPress system folders for performance

Why It Matters

Adding Disallow: /cdn-cgi/ helps prevent misleading errors in tools like Google Search Console, Ahrefs, or Screaming Frog. It ensures crawlers focus on the actual content of your site—especially useful when optimizing large-scale WordPress environments.

Final Thoughts

Whether you manage one site or dozens, this script helps automate a simple best practice that improves SEO clarity. It’s an example of how a small tweak, when automated, can save time and improve accuracy for web professionals.

✅ Get the script: GitHub – WordPress-Cloudflare robots.txt Updater
🛠 Created by David Carrero Fernández-Baillo – Stackscale