What we can reach, and what we only know is out there

The reachability map

Every federal inspector general publishes their work. That is not the same as anyone being able to read it together. This page is a picture of the difference, drawn from the archive's own records rather than from an impression of them.

The population, and how it was asked

The set here is the 74 inspectors general who are members of the Council of the Inspectors General on Integrity and Efficiency, taken from the contact appendix of its own annual report. Asking them came to 83 hosts rather than 74, and the extra 9 are worth naming rather than absorbing: seven are a second host for an office already counted, and two are not an inspector general at all (oversight.gov, which is the aggregator, and the Smithsonian's main site, whose record in our own files carries a warning that it is the museum and not the OIG). Every count on this page that is about offices is taken over the 74. Every count that is about hosts says so. Among the 74 offices, 40 publish on their own domain and 32 on a path inside the parent agency's site. 6 of the 83 hosts are on oversight.gov itself, which is more than this project's own records say: the field that labels a host's shape tags only two of those six. The hostname is the fact. The label is something somebody filled in.

Each host was asked exactly one question, once: may we collect from you? The answer was read from the host's own robots.txt, literally, as written. Nothing else was fetched in that pass, and no answer was inferred from a neighbor.

From asking to holding

Permission is the first step and the cheapest one. Everything after it costs time, and the narrowing below is not a story about refusal. It is a story about how little of the permitted ground anyone has walked.

8 hosts sit inside the permitted set and have never been visited by any pass, and they are not the ones you would expect. They are exactly the hosts nobody could ask properly: the five that publish no robots.txt at all, and the three whose file grants collection while their edge turns a script away. Not one is a host that said yes and was then simply neglected. What happened to those eight is the next section, and it did not happen by accident.

What visiting turns up

Three passes have been run and they are different instruments. A link walk follows a site's own links and counts files by extension. A sitemap read asks a host for its own index of itself. A document index enumerates every published item and records where it came from. The first two tell you whether a host is worth the third.

The walk covered 17 hosts in 1,514 requests and found 6,761 files. That figure is a floor, not a total: 10 of those 17 hosts hit the stop bound with a queue still unread, and 6 returned nothing at all after a single page. Why those 6 came back empty is not recorded, so it is not claimed here. Not every absence has been explained, and an unexplained one is worth more written down than guessed at.

The sitemap pass is the wider instrument: 54 hosts enumerated, 211,075 URLs read out of 452 sitemap files. It is also the clearest example on this page of why a total needs its parts. That pass counted 44,865 URLs that look like documents, and 40,311 of them, 89.8 percent, came from a single host (www.archives.gov). Only 2 of the 54 hosts produced any documents at all, and 20 returned no URLs whatsoever. Quoted flat, that number would say something about federal oversight publishing that is simply not true.

That pass carries the same truncation caveat as the walk, and its own output does not say so: one host stopped at the sitemap bound with 200 files read, so its count is a floor in exactly the way the walk's is. Files that could not be parsed are counted nowhere, so a host can appear among those that returned no URLs when what happened is that nothing it returned could be read.

Who got left out, and on whose say so

29 hosts were skipped by that pass before a single sitemap was read, on the strength of the verdict recorded against them. Most of that is exactly right. 11 hosts declined, 8 of them member offices, and a pipeline that collects from a host that said no is not worth defending.

The gap the whole project is about

On 10 hosts, every published document was enumerated: title, URL, type, and the host that actually serves the file. That came to 20,428 documents.

Of those, none are held here. Not a small number. None. What sits in this project's document directory is 40 of its own bookkeeping files, which are the enumeration written down, plus 12 saved web pages under a single host, and not one of those pages is among that host's enumerated documents. No document has been downloaded. No digest has been taken of one.

Until an hour before this build, this page said 52 were held, because the count was measuring every file in that directory rather than every document in it. The tell was one office showing three where the rest showed four, which turned out to be a missing summary file. A number that moves when a bookkeeping file is written is not measuring the thing its label claims.

Something the enumeration turned up that an impression would not: the documents are not all served by the office whose page they appear on.

The refusals, re-asked

On 2026-09-08 this project re-asked every host it had recorded a refusal for, as itself, with the robots file read first and kept. 10 records across 5 hosts were in scope. 5 of those 10 records were a refusal. The rest were not.

What each answer is standing on

The last picture is the one this project cares about most, because it is the one that decides whether any of the others can be cited. A verdict is only worth what it can be checked against.