America's Top-Rated Dentist Directory
TheToothDirectoryBot
If you found this page in your server logs, this is us. Here is exactly what we do and how to stop us.
What it is
TheToothDirectoryBot/1.0 reads dental practice websites so our directory can say accurate things about them — which treatments a practice offers, which insurers it accepts, whether it opens on a Saturday. We read what your site already says publicly and record it.
Its full user agent string is:
TheToothDirectoryBot/1.0 (+https://thetoothdirectory.com/bot/)How it behaves
- It obeys robots.txt. Disallow it and it will not fetch the page.
- One request per second per site, at most — slower if your robots.txt sets a crawl delay.
- At most 6 pages per practice, usually your homepage plus services, insurance and about pages.
- Only HTML, only on your own domain, nothing over 2MB, and it gives up after 15 seconds.
- It revisits at most every 90 days. This is not a continuous crawl.
- It does not submit forms, follow links off your site, or attempt to log in anywhere.
How to block it
Add this to your robots.txt:
User-agent: TheToothDirectoryBot
Disallow: /We check robots.txt before every page and cache it briefly, so a block takes effect on our next visit. Blocking the crawler does not remove your listing — it just means we will not read your site, so your listing will show less detail.
Removing or correcting your listing
Email hello@thetoothdirectory.com and we will fix or remove it. Corrections are free, they do not require an account, and they are not conditional on buying anything. If something we extracted from your site is wrong, tell us which page it came from and we will correct it.
More about how we source and rank things is on oureditorial policy page.