Home / AI visibility desk / Search, agents, and training

Configuration check

Search, agents, and training are three different doors.

Search, agents, and training are three different doors. Cloudflare documents separate access policies for Search, Agent, and Training traffic. Cloudflare's robots.txt setting documentation was updated November 2, 2026. A robots.txt allow line does not prove a crawler received the page. This page explains the check. It does not flip a live firewall, Bot Fight, or robots rule, and it does not report a crawl result for EXSPEEDITE or for Strong Tower Consulting, LLC. EXSPEEDITE TMS is freight transportation software from Exspeedite LLC. Exspeedite LLC has been in freight technology for more than 18 years. Founder Scott Elliott is the transportation systems architect. Strong Tower Consulting, LLC, founded in Minnesota in 2008, is a separately organized sister company under common ownership. Consulting does not require a software purchase.

Three doors, one check

Pick a traffic type. The panel shows the business objective and the check. The check has three possible records: a successful response, a blocked request, or a browser challenge. Those are categories an operator writes down after a real request. They are not a log from this page.

Choose a door. You will see the business objective and the three outcomes to record: a successful response, a blocked request, or a browser challenge.

Agent

Business objective

Let a prospect's assistant read the service and reach the assessment.

The check

Watch a real user-directed agent request that should read the service and reach the AI visibility assessment. Record one outcome. A robots.txt allow line does not prove the assistant received the page.

  • Successful response. The assistant received the page.
  • Blocked request. The request was refused, and the page was not delivered.
  • Browser challenge. The request stopped on a challenge instead of the page.

Training

Business objective

Training crawlers are a separate decision. A hard block on training can also hit crawlers used for both search and training. Cloudflare's Disallow AI Training control is meant to state a training preference while keeping search access for qualifying mixed-purpose crawlers.

The check

Watch a real training request on its own. Record one outcome. This page does not apply Disallow AI Training, Block, or any other live control.

  • Successful response. The crawler received the page.
  • Blocked request. The request was refused, and the page was not delivered.
  • Browser challenge. The request stopped on a challenge instead of the page.

An allow line is not delivery

A robots.txt allow line states a preference. Crawlers can ignore it, and a host, firewall, or challenge can still stop a request that the file allows. The proof of delivery is a successful response. A blocked request and a browser challenge are the other two records. None of the three is written by typing Allow into robots.txt.

The teaching line below is an illustration of that preference. It is not a line from the live robots file, and it is not a delivery record.

Allow: /

What OpenAI and Perplexity document

OpenAI documents that hosting or firewall rules must allow its search traffic. A robots.txt allow for the search crawler is not enough when the host, content delivery network, or firewall refuses the request. Perplexity documents the same requirement for its search traffic and points site owners to published IP ranges. This page does not paste those ranges. The ranges change, and the companies publish the current lists.

OpenAI's crawler overview is at developers.openai.com/api/docs/bots. Perplexity's crawler notes are at docs.perplexity.ai/docs/resources/perplexity-crawlers. Cloudflare's access policies are at AI Crawler and Agent access policies, and the robots.txt setting page, updated November 2, 2026, is at robots.txt setting.

Prepare, do not flip

Bing's newer grounding controls were announced October 30, 2026 and are not active yet. They begin February 1, 2027. Older NoArchive or NoCache preferences need a look before March 1, 2027, so a training change does not accidentally hide pages from answers. This page does not change meta tags on other pages.

Microsoft described the timeline on Your content, your choice. Reading that note is the preparation. Applying a new tag from this page is not.

Cloudflare and OpenAI announced a discovery pilot on October 15, 2026, and there is no public enrollment switch, so this page does not depend on that pilot.

Where to go next

EXSPEEDITE TMS stays the product of Exspeedite LLC. Strong Tower Consulting, LLC remains the sister consultancy under common ownership. Consulting does not require a software purchase. The desk and the assessment are the next public pages. Neither one changes a firewall.

Who publishes this page

Scott Elliott, founder of Exspeedite LLC, is the transportation systems architect. Exspeedite LLC has been in freight technology for more than 18 years. Strong Tower Consulting, LLC was founded in Minnesota in 2008. The two companies are separately organized sister companies under common ownership. Strong Tower Consulting, LLC is unaffiliated with strongtower.consulting and with strongtowerconsulting.com. Consulting does not require a software purchase.

Questions

Does a robots.txt allow line prove a crawler received the page?

No. A robots.txt allow line does not prove a crawler received the page. The check is the response on a real request: a successful response, a blocked request, or a browser challenge. This page does not report a crawl result for EXSPEEDITE or for Strong Tower Consulting, LLC.

Why are Search, agents, and training separate doors?

Cloudflare documents separate access policies for Search, Agent, and Training traffic. Cloudflare's robots.txt setting documentation was updated November 2, 2026. A hard block on training can also hit crawlers used for both search and training. Cloudflare's Disallow AI Training control is meant to state a training preference while keeping search access for qualifying mixed-purpose crawlers.

Does this page change a firewall, Bot Fight, or robots rule?

No. This page explains a configuration check. It does not flip a live firewall, Bot Fight, or robots rule, and it does not change meta tags on other pages. Consulting with Strong Tower Consulting, LLC does not require a software purchase.

Request a configuration conversation

This form asks Exspeedite LLC for a conversation. It does not change a firewall, Bot Fight, or robots rule. Consulting does not require a software purchase.

Call 763-244-1373.