About our crawler

msppie's provider directory is partly built by fetching public pages from Australian IT and managed-service providers' own websites (an About page, a Services page, a Contact page) and reading what they say about themselves. This page is what our crawler's own request headers point to, and what it does with what it reads.

User agent

Requests identify as MSPPieBot/1.0, with this page's URL in the header. We fetch robots.txt before anything else on a domain and follow it: a site can name MSPPieBot specifically, or rely on its rules for *. A site with no reachable robots.txt is treated as fully disallowed, not fully allowed.

What we take

Publicly visible text: a business's own description of itself, the services it lists, the locations it names, named staff with a stated role. We do not create an account, submit a form, or fetch anything behind a login. We never generate a fact ourselves: an ABN, a phone number, or a founding year is only ever stored if it appears, verifiably, on the page we read, alongside a record of exactly where it came from.

What happens next

A page only publishes once we hold enough genuine, sourced facts about a business to make it useful: a thin result doesn't get a page. If your business appears and something is wrong, you can request a correction without creating an account, or claim the page to edit it directly once we've verified you control the domain.

Opting out

Disallow MSPPieBot in your robots.txt and we will not fetch your site. If your business is already listed and you'd rather it wasn't, email hello@msppie.com and we'll remove it.