AIA Matrix

Troubleshooting

Resolve common issues encountered when setting up AI-readable files.

403 Forbidden on /markdown/ or /semantic/

Symptom: Visiting yourdomain.com/markdown/ or yourdomain.com/semantic/ directly in a browser returns a 403 error.

Cause: Most web servers disable directory listing by default for security. This is normal behavior, not a broken setup — individual files inside the folder (e.g., /markdown/about.md) will still load correctly.

Fix: Add an index.html file inside the folder. When a browser requests the folder URL, the server will serve this file automatically instead of attempting (and failing) to list the directory contents. AIA Matrix generates this file automatically for /markdown/ on the Professional plan.

Google Won't Index .txt or .md Files

Symptom: Submitting llms.txt or files inside /markdown/ to Google Search Console's "Request Indexing" repeatedly returns "URL is not on Google," or the request fails outright.

Cause: Google's indexer is fundamentally built around HTML and JSON content. Plain .txt and .md files are often crawled but not added to Google's search index — robots.txt is essentially the only .txt file Google reliably indexes.

Fix: This isn't actually broken — AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) fetch these files directly regardless of Google's indexing status, so they still serve their purpose. If you specifically want this content to also appear in Google search results, pair it with an indexable .html version, such as your /markdown/index.html hub page.

"Oops! Something Went Wrong" When Requesting Indexing

Symptom: Google Search Console's "Request Indexing" button returns an error instead of submitting the request.

Cause: This is a known, intermittent issue with Google's own indexing request system, unrelated to your file setup.

Fix:

  • Wait 30–60 minutes and try again
  • Try the "Test Live URL" button first, then retry Request Indexing
  • Submit or resubmit your sitemap under Search Console → Sitemaps as a more reliable alternative
  • Ensure the page is linked from somewhere else on your site (e.g., your footer) — Google often discovers linked pages naturally within days, without needing a manual request

robots.txt Changes Aren't Showing Up

Symptom: You've uploaded an updated robots.txt with AI crawler rules, but visiting yourdomain.com/robots.txt still shows the old content.

Cause: On WordPress, SEO plugins like Yoast SEO, RankMath, or All in One SEO often generate a virtual robots.txt dynamically, which can override a static file you've uploaded via FTP.

Fix: Edit robots.txt through your SEO plugin's built-in editor instead of uploading via FTP:

  • Yoast SEO: Yoast SEO → Tools → File Editor
  • RankMath: RankMath → General Settings → Edit robots.txt
  • All in One SEO: All in One SEO → Tools → Robots.txt Editor

See WordPress Setup for full details.

A Page Suddenly Lost Most of Its Search Impressions

Symptom: Google Search Console shows a sharp drop (often 90%+) in impressions for a specific page, frequently a /tag/ or auto-generated archive page.

Cause: WordPress tag and category archive pages are often thin, auto-generated, and increasingly devalued by Google's quality systems, especially when a competitor has a dedicated, purpose-built page targeting the same keyword.

Fix:

  • Build a dedicated landing page for the keyword (not relying on an auto-generated tag page)
  • Add a noindex meta tag to the thin tag/archive page so it stops competing with your new dedicated page
  • Internally link to the new page from relevant content

Repeated 404s From a Deleted Page

Symptom: Crawl logs (Search Console → Crawl Stats) show the same dead URL being requested repeatedly, such as an old page builder template.

Cause: Something is still linking to the deleted URL internally, or the page was removed without a redirect.

Fix: Install the free Redirection plugin (WordPress) and add a 301 redirect from the dead URL to the most relevant current page. Avoid leaving repeated 404s in place — they reduce your Accessibility Index and waste crawl budget that could go toward your actual content.

Missing "item" Field in Breadcrumb Schema

Symptom: Search Console flags "Missing field 'item' (in 'itemListElement')" under Enhancements.

Cause: Your breadcrumb structured data is missing the required URL field for one or more list items.

Fix: Ensure every ListItem in your BreadcrumbList schema includes all four required fields:

{
  "@type": "ListItem",
  "position": 1,
  "name": "Home",
  "item": "https://yourdomain.com/"
}

This is typically generated by your SEO plugin (Yoast, RankMath) — check your breadcrumb settings there first before editing schema manually. After fixing, click Validate Fix in Search Console.

AdsBot or Other Crawlers Hitting /.well-known/apple-app-site-association

Symptom: Crawl logs show repeated 404s for /apple-app-site-association or /.well-known/apple-app-site-association.

Cause: This is a standard file location iOS checks for app-linking purposes. If you don't have an iOS app, this 404 is expected and harmless.

Fix: No action needed — this can be safely ignored.

Still Stuck?

If your issue isn't covered here, contact support@aiamatrix.com with your site URL and a description of what you're seeing.

Related