News November 19, 2026  ·  13 min read

Opting a WordPress Marketplace Out of AI Training (Without Breaking Search)

A Fisher Tech Solutions case study for developers and site owners. There is no single switch that keeps your images out of AI training. What works is a stack: a few opt-out signals that well-behaved crawlers read, a notice written into every image file, a closed API, enforcement at the edge, and policy that backs […]

Opting a WordPress Marketplace Out of AI Training (Without Breaking Search)

A Fisher Tech Solutions case study for developers and site owners.

There is no single switch that keeps your images out of AI training. What works is a stack: a few opt-out signals that well-behaved crawlers read, a notice written into every image file, a closed API, enforcement at the edge, and policy that backs it all up.

We built that stack for My Community Made, a WordPress and Dokan marketplace for independent makers. Vendors upload thousands of product photos, and they asked a fair question: is my work feeding someone’s AI model? This post walks through every layer we shipped, the gotchas that cost us time, and what none of it can do. (Read MCM’s announcement for vendors — link to the live news post.)

The short version:

  • robots.txt tells about 25 AI training crawlers to stay out, while search and AI-search crawlers keep full access.
  • Headers and meta tags (noai, noimageai, tdm-reservation: 1) mark every page and every uploaded file.
  • TDMRep publishes a machine-readable EU text-and-data-mining reservation.
  • The media REST API no longer lets anonymous callers list the whole upload library.
  • Each image file carries an IPTC “data mining prohibited” notice inside its own metadata, and loses its GPS location and serial numbers.
  • Cloudflare blocks AI bots that ignore the signals.
  • Terms, vendor agreement and privacy policy state the rule in plain language.

The problem: signals are not locks

Makers’ product photos are close to ideal training data. They are well lit, tightly framed, and come with a title, a description, a price and a category. A marketplace full of them is a labelled image dataset that someone else already paid to build.

Two kinds of protection exist, and it helps to keep them apart:

Opt-out signalsBlocking
Examplesrobots.txt, noai headers, TDMRep, in-file IPTC noticeBot blocking at the CDN, closed APIs
Who it stopsCrawlers that choose to honour itRequests that match the rule, honest or not
What else it doesCreates a clear, dated, machine-readable record that you objectedCuts load and casual scraping
RiskNone to search, if you scope it rightCan block search or your own apps if scoped wrong

Signals cost almost nothing and matter more than they look. Major AI companies publish crawler names and say they honour robots.txt. Under EU law, a machine-readable reservation is what takes your content out of the text-and-data-mining exception. Blocking catches the rest, as far as it can.

The hard constraint for a marketplace is that search must keep working. Product pages need Google, Bing, Google Shopping and, more and more, AI shopping answers. Every layer below is scoped to training only.

Layers of

Only two layers actually block anything; the rest are signals and cleanup. Each one is scoped so search, AI shopping answers and the mobile apps keep working.

Layer 1: robots.txt for training crawlers only

The plugin disallows 24 user agents whose job is collecting training data, and leaves everything else alone.

Disallowed (training)Still allowed (search and AI search)
GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, GoogleOther, Applebot-Extended, meta-externalagent, FacebookBot, Bytespider, Amazonbot, cohere-ai, cohere-training-data-crawler, Diffbot, ImagesiftBot, img2dataset, Timpibot, AI2Bot, Ai2Bot-Dolma, omgili, omgilibot, PanguBot, Kangaroo Bot, Webzio-ExtendedGooglebot, Googlebot-Image, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot

A few of these deserve a note. Google-Extended and Applebot-Extended are control tokens, not separate crawlers: blocking them opts you out of Gemini and Apple Intelligence training without touching Google Search or Applebot. CCBot matters most for images, because Common Crawl is the upstream source of LAION and most open image datasets. The list is behind a filter, so a site that also wants out of AI search answers can move those agents across.

We also add Cloudflare’s content-signal line for every agent:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

That says: index us, you may ground an answer in this page, do not train on it.

Gotcha: Rank Math rewrites robots.txt. Its robots.txt editor replaces the whole output on the robots_txt filter at the default priority. A filter that runs before it gets thrown away. The fix is to append at priority 99 and to check for our own marker so the block is never added twice. One more trap: if a physical robots.txt file sits in the web root, WordPress never runs the filter at all. Delete the file, or paste the block into it.

Layer 2: noai and noimageai on every page and file

robots.txt is read once per site. Headers travel with each response, so they still apply when someone fetches an image URL directly. We send two:

X-Robots-Tag: noai, noimageai
tdm-reservation: 1

noai and noimageai are not a formal standard, but img2dataset (the common tool for building image datasets) and several others honour them. Google and Bing ignore values they don’t know, so search is unaffected. The header is added alongside any existing X-Robots-Tag, so a page’s own noindex survives.

Pages also get the matching meta tags:

<meta name="robots" content="noai, noimageai">
<meta name="tdm-reservation" content="1">

The meta tags matter on a cached site. A page cache such as WP Rocket serves stored HTML without running PHP, so the PHP headers only reach uncached requests. The meta tags are baked into the cached page.

Gotcha: the .htaccess headers never arrived. Images are served by the web server, not WordPress, so the plugin writes a marked mod_headers block into the uploads .htaccess. It looked right and did nothing: behind the host’s stack and Cloudflare, the headers never reached the browser. We moved them to the edge with a Cloudflare Transform Rule:

  1. Rules → Transform Rules → Modify Response Header. It must be a response header rule. A request header rule edits what Cloudflare sends to your server, which crawlers never see.
  2. Match: URI path starts with /wp-content/uploads/.
  3. Add two separate headers: X-Robots-Tag = noai, noimageai, and tdm-reservation = 1. Don’t fold them into one value.

The edge rule now covers every file under uploads, whatever the origin server does.

Layer 3: TDMRep, the EU legal opt-out

Article 4 of the EU’s DSM Directive (2019/790) lets anyone mine lawfully accessible content, unless the rightsholder has reserved that right “in an appropriate manner, such as machine-readable means” (Art. 4(3)). The EU AI Act points general-purpose model providers back to that reservation. A robots.txt line is arguably enough; a dedicated standard removes the argument.

The W3C community’s TDMRep protocol is that standard. We publish it three ways: the tdm-reservation: 1 header and meta tag above, and a site-wide file at /.well-known/tdmrep.json:

[
  {
    "location": "/*",
    "tdm-reservation": 1
  }
]

WordPress serves the file from a parse_request hook, so there’s no file to deploy. TDMRep also allows a tdm-policy URL, but the spec requires that policy to be machine-readable ODRL, not an HTML terms page. We left it out rather than point at the wrong thing; the plugin has a filter for adding one later.

Layer 4: closing the media REST API

WordPress exposes /wp-json/wp/v2/media to anyone, logged in or not. Page through it and you get the URL of every image ever uploaded. One short script downloads the whole library, no crawling needed.

The fix is a rest_pre_dispatch filter that refuses GET and HEAD on that route, list and single item, for anyone who has no reason to browse the library. Anonymous callers now get a 401.

What we deliberately left open matters just as much:

  • Uploads from the mobile apps. MCM’s iOS and Android apps upload vendor photos with POST /wp/v2/media, authenticated with WordPress Application Passwords. The block only touches reads, and the route stays registered.
  • Dokan’s app helper, which calls the media controller directly for profile pictures and never reads from the route.
  • Vendors and admins: the block editor and vendor tools work as before.
  • Product images in product data. They are public on the storefront anyway. This layer removes the “list everything” shortcut, not the images.

If you do this on your own site, test a real upload from every app before you ship. A blanket “block the media route” rule breaks mobile uploads silently.

Layer 5: writing the notice into the files themselves

Every layer so far lives on the website. Once an image is downloaded, reposted or pulled from a dataset mirror, the headers and robots.txt are gone. So we put the notice inside the image.

IPTC Photo Metadata (2023.1 and later) defines a PLUS property for exactly this:

<rdf:Description rdf:about="" xmlns:plus="http://ns.useplus.org/ldf/xmp/1.0/"
  plus:DataMining="http://ns.useplus.org/ldf/vocab/DMI-PROHIBITED-EXCEPTSEARCHENGINEINDEXING"/>

“Prohibited except search engine indexing” is the same line we draw everywhere else. Tools such as exiftool show it as Data Mining.

Writing XMP without wrecking the image took byte-level work in each format. We did not use an image library, because re-encoding changes pixels and fights the optimizer.

FormatWhere XMP livesWhat we had to handle
JPEGAPP1 segment before the image dataMerge into an existing packet as an extra rdf:Description, or insert a new segment. Open up empty self-closing <rdf:RDF/> blocks that some phone apps write.
PNGiTXt chunk, keyword XML:com.adobe.xmpParse and rebuild the chunk, recompute its CRC-32, add a new chunk after IHDR when none exists.
WebPXMP chunk in the RIFF containerSimple files (a lone VP8 or VP8L chunk) can’t carry metadata. Convert them to the extended layout by adding a VP8X header with the canvas size read from the image chunk, set the XMP flag, fix the RIFF length.

In every case the image data is copied untouched. A file that already declares DataMining keeps its own value, and a file that’s already done is left byte-for-byte the same, so reprocessing is safe.

Keeping it there. MCM uses Imagify, which optimizes images after upload and can strip metadata. The plugin re-applies the notice after each Imagify run. It also covers what Imagify creates: the .webp copy of every size (what modern browsers are actually served) and Imagify’s backup of the original, which is public too. Our first live check found the .webp copies untagged; that’s why they’re in the list.

Backfill. New uploads are handled on the way in. The existing library is processed with WP-CLI:

wp mcm-ai backfill --dry-run
wp mcm-ai backfill

Then purge the CDN cache for images, or visitors keep getting the old files.

Layer 6: stripping private metadata

The live check turned up something worse than missing notices: full-size product photos carrying exact GPS coordinates. For a maker who shoots products at the kitchen table, that’s their home address, published next to their shop name.

We now remove privacy-sensitive metadata before adding the notice, and keep what’s useful or deliberate.

RemovedKept
The whole EXIF GPS block (coordinates, altitude, direction, timestamps)Camera make and model
Body and lens serial numbersCapture dates and exposure settings
Camera owner name, Artist, XPAuthorOrientation
MakerNote (vendor-specific data that can hold serials and names)Colour profiles
The same fields in XMP (exif:GPS*, aux:SerialNumber, aux:OwnerName and friends), as elements or attributesCredits added on purpose: copyright, dc:creator, dc:rights

The aim is to remove what identifies a person or a place, not to scrub authorship a vendor chose to add.

Technique: edit EXIF in place. EXIF is a small TIFF structure full of absolute offsets. Delete a field the obvious way and every offset after it is wrong, which corrupts the block. Re-encoding the image avoids that but costs quality and file size, and drops things you meant to keep. Instead we:

  1. Remove the entry from its directory (IFD) and shift the remaining 12-byte entries up to close the gap.
  2. Zero the bytes of the removed value where they lived.
  3. Leave the block the same length, so every other offset stays valid.

For GPS, the whole GPS directory and its data are zeroed and the pointer to it is dropped. The same cleaning runs on JPEG APP1, PNG eXIf and iTXt, and WebP EXIF and XMP chunks.

Verifying it live

None of this counts until it’s checked on the production site, through the CDN, as a stranger would see it. Our checklist:

CheckHowExpected
robots.txtOpen /robots.txtEnds with the # BEGIN MCM AI Protection block, after Rank Math’s rules
Image headerscurl -sI <image URL> | grep -iE "x-robots-tag|tdm-"Both headers present
TDMRepOpen /.well-known/tdmrep.jsonThe JSON rule
Media API/wp-json/wp/v2/media, logged out401
App uploadsUpload a product photo from the iOS and Android appsSucceeds
File contentsRead image bytes in the browser, or exiftool a downloadData Mining present; no GPS, serial or owner fields

For the last check we fetched images straight from the live site in a browser and scanned the bytes for the XMP property and for EXIF GPS tags, across originals, sizes, .webp copies and backups. That scan found every one of the gaps above: untagged WebP copies, empty XMP blocks, and the GPS data.

Two lessons from that pass:

  • A Cloudflare MISS doesn’t mean a fresh file. With Tiered Cache on, an edge MISS can still be filled from an upper-tier data centre holding the old copy. After the backfill we kept seeing untagged files with a MISS status. Purge the cache (images at least) after any change to file contents, then re-test.
  • WordPress forgets some originals. Since WordPress 5.3, large uploads are scaled down to a -scaled copy, and the attachment records the original under original_image. Some of our attachments pointed at the -scaled file with no original_image entry, so the full-size original, still public and still carrying GPS, was never in the list of files to clean. Four files. The plugin now derives the original from the -scaled or -rotated file name as well.

Enforcement, and the honest limits

Everything above is a signal or a cleanup. To stop crawlers that ignore signals, MCM turned on two Cloudflare features under Security → Bots:

  • Block AI Bots, which blocks known AI crawlers at the edge.
  • AI Crawl Control, which lets you choose per crawler: we block training crawlers and allow search.

If you also enable Cloudflare’s managed robots.txt, it merges with the plugin’s rules rather than replacing them.

What no stack can do:

  • Stop a determined bad actor. A scraper that ignores robots.txt and spoofs a browser user agent looks like a shopper. Bot scoring catches some; it won’t catch all.
  • Prevent screenshots. Anything a browser can show, someone can capture. Right-click blockers and image overlays only annoy real customers.
  • Reach images already collected. Opt-outs work going forward.
  • Keep the in-file notice everywhere. Some platforms strip metadata on upload. The notice survives a straight download and many reposts, not all.

What the stack does do is real: it removes the site from compliant training crawls, closes the easiest bulk-download path, puts a dated, machine-readable reservation on record in the form EU law asks for, and stops leaking vendors’ locations.

The policy layer

Technical signals need words behind them. Alongside the plugin we drafted, for legal review:

  • Terms of Service: a new section saying site content, including vendor images, may not be used to train AI models.
  • Privacy policy: that location and device metadata are removed from uploaded photos.

The pattern carries over to any site where users upload work they care about: a marketplace, a portfolio network, a photography community, a membership site.

Want this on your site?

Fisher Tech Solutions builds and maintains WordPress and WooCommerce sites, including marketplaces with mobile apps. We can audit what your site exposes today, put the full stack in place without breaking search, apps or image optimization, and verify it live. Get in touch.