read-deleted-pages

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Read deleted pages

读取已删除页面

Archives work because removal is a decision made after publication. The beginner's mistake is treating the Wayback Machine as a website you browse: the calendar UI shows one page at a time, while the CDX API enumerates every URL the archive ever saw under a host, including paths nobody remembers.

归档之所以有效，是因为内容移除是在发布之后做出的决定。新手常犯的错误是将Wayback Machine当作普通浏览网站：日历UI一次只显示一个页面，而CDX API会枚举归档中记录的某个主机下的所有URL，包括那些无人记得的路径。

Which archive first

优先选择哪个归档工具

Situation	Go to	Why
A URL that 404s now	Wayback, then archive.today	Different corpora, different removal pressure
"What did this site say before X"	Wayback CDX with `collapse=digest`	Gives the dates content changed, not every capture
A JS-heavy page or a social post	archive.today	Captures the rendered DOM; Wayback often replays an empty shell
Wayback says the URL is excluded	archive.today, then Common Crawl	Neither answers to Wayback's exclusion process
Paths and subdomains you don't know about	CDX with `matchType=domain`	It is an enumeration tool, not a lookup tool
A URL Wayback never captured	Common Crawl	Independent crawler, different seeds
Regional or national-TLD content	A Memento aggregator, then the national archive	Some national archives offer full-text search
Evidence you cannot lose	Capture it yourself, then submit to two archives	Nobody else is preserving it for you

Coverage and removal policy per source: reference/archive-comparison.md.

场景	优先选择	原因
当前返回404的URL	Wayback，其次是archive.today	两者的数据集不同，面临的移除压力也不同
"该网站在X事件之前的内容是什么"	带 `collapse=digest` 参数的Wayback CDX	仅返回内容发生变化的日期，而非每次捕获记录
重度依赖JS的页面或社交帖子	archive.today	捕获渲染后的DOM；Wayback通常只会回放一个空框架
Wayback显示该URL被排除	archive.today，其次是Common Crawl	这两者不受Wayback的排除机制约束
未知的路径和子域名	带 `matchType=domain` 参数的CDX	它是枚举工具，而非查询工具
Wayback从未捕获过的URL	Common Crawl	独立爬虫，爬取种子不同
区域或国家顶级域名（TLD）内容	Memento聚合器，其次是国家归档库	部分国家归档库支持全文搜索
不能丢失的证据	自行捕获，然后提交至两个归档库	没有其他机构会为你保存这些内容

各来源的覆盖范围和移除政策： reference/archive-comparison.md。

The CDX API is the actual tool

CDX API才是核心工具

The CDX server indexes capture records — one row per capture — and lets you filter, collapse and enumerate them. Every URL a crawler ever saw under a host, deduplicated:

bash

curl -s 'http://web.archive.org/cdx/search/cdx?url=example.com/*&output=json&fl=original&collapse=urlkey' \
  | jq -r '.[1:][] | .[0]' | sort -u

Use

output=json

rather than the default text: it survives URLs containing spaces and quotes, which the text format mangles. The first JSON row is a header naming the columns, not a capture — hence the

.[1:]

Four parameters do nearly all the work:

matchType
—
```
exact
```
,
```
prefix
```
,
```
host
```
, or
```
domain
```
.
```
matchType=domain
```
returns the host and all subdomains, routinely surfacing staging, legacy and internal hosts long gone from DNS. Take them to
```
find-hidden-subdomains
```
.
filter=<field>:<regex>
, invertible with
```
!
```
and combinable. Filter to
```
statuscode:200
```
for captures that returned content, or
```
!mimetype:text/html
```
to isolate documents, archives and config files.
from
/
to
take a timestamp prefix at any precision —
```
from=2019
```
,
```
from=201903
```
— so you can bracket the window around a known event.
collapse
suppresses adjacent rows sharing a field.
```
collapse=urlkey
```
is the enumeration idiom;
```
collapse=digest
```
matters most.

digest

hashes the captured payload, so

collapse=digest

reduces a page's history to the captures where content actually changed. Four hundred captures become six rows, and those six dates are the story.

bash

curl -s 'http://web.archive.org/cdx/search/cdx?url=example.com/about/team&output=json&collapse=digest&fl=timestamp,digest,statuscode'

Full parameters, paging, and a ready enumeration pass: reference/cdx-cookbook.md.

CDX服务器为捕获记录建立索引——每条捕获对应一行记录——并支持过滤、合并和枚举。爬虫在某个主机下记录的所有URL都会去重：

bash

curl -s 'http://web.archive.org/cdx/search/cdx?url=example.com/*&output=json&fl=original&collapse=urlkey' \
  | jq -r '.[1:][] | .[0]' | sort -u

使用

output=json

而非默认的文本格式：它能兼容包含空格和引号的URL，而文本格式会损坏这类URL。JSON的第一行是列名表头，而非捕获记录——因此需要使用

.[1:]

来跳过表头。

以下四个参数几乎能完成所有操作：

matchType
—— 可选值为
```
exact
```
、
```
prefix
```
、
```
host
```
或
```
domain
```
。
```
matchType=domain
```
会返回主机及其所有子域名，通常会发现早已从DNS中消失的 staging、遗留和内部主机。可将这些主机用于
```
find-hidden-subdomains
```
操作。
filter=<field>:<regex>
，可通过
```
!
```
取反，也可组合使用。过滤
```
statuscode:200
```
可获取返回内容的捕获记录，过滤
```
!mimetype:text/html
```
可筛选出文档、归档文件和配置文件。
from
/
to
接受任意精度的时间戳前缀——例如
```
from=2019
```
、
```
from=201903
```
——因此你可以围绕已知事件限定时间范围。
collapse
会隐藏共享同一字段的相邻行。
```
collapse=urlkey
```
是枚举的惯用参数；
```
collapse=digest
```
则最为重要。

digest

会对捕获的负载进行哈希运算，因此

collapse=digest

会将页面历史精简为内容实际发生变化的捕获记录。原本四百条记录会变成六行，而这六个日期就是内容变化的脉络。

bash

curl -s 'http://web.archive.org/web/20190412093000id_/http://example.com/pricing' > pricing-2019.html

完整参数、分页机制及现成的枚举流程：reference/cdx-cookbook.md。

Replay is not the capture

回放内容不等于原始捕获

The page at

web.archive.org/web/<timestamp>/<url>

is reconstructed. The archive injects a toolbar, rewrites links and resource references back into the archive, and — the part that misleads people — pulls each resource from its own nearest capture, which may be months away from the HTML you asked for. The rendered page you screenshot may never have existed.

Append

id_

to the timestamp to get the raw archived response with no injection and no rewriting:

bash

curl -s 'https://web.archive.org/web/20190412093000id_/http://example.com/pricing' > pricing-2019.html

Use

id_

for anything you will parse, hash, diff, or cite. A page that replays as blank may have been captured fine, so fetch the raw response before concluding the archive has nothing — but archived JavaScript that called an API at view time has no data behind it, because the API response was never captured. Those pages are shells by nature, not by failure.

web.archive.org/web/<timestamp>/<url>

中的页面是重构而来的。归档会注入工具栏、重写链接和资源引用使其指向归档内容，而最容易误导人的是：每个资源都是从离它最近的捕获记录中提取的，这可能与你请求的HTML捕获时间相差数月。你截图的渲染页面可能从未真实存在过。

在时间戳后添加

id_

即可获取无注入、无重写的原始归档响应：

bash

curl -s 'https://web.archive.org/web/20190412093000id_/http://example.com/pricing' > pricing-2019.html

任何需要解析、哈希、对比或引用的内容，都应使用

id_

获取。某个页面回放时显示为空，可能实际已被正常捕获，因此在断定归档无相关内容前，请先获取原始响应——但归档的JavaScript如果在浏览时调用API，那么它背后不会有数据，因为API响应从未被捕获。这类页面本质上就是空框架，并非捕获失败。

What this actually recovers

实际可恢复的内容

Pull the raw captures at two dates from the

collapse=digest

list and compare them as text. The removals are the evidence: a person deleted from a team page between two dates is a departure with a date range attached, and a

find-anyone

lead with no remaining reason to be loyal. What surfaces, in practice:

Staff and team pages. The current site lists who is there now; the archive lists everyone who ever was, with titles, bios, and often direct email addresses from before the org moved to a contact form.
Prior pricing and terms — what was charged, what was promised, and what the refund policy said on the date a dispute began. Also removed posts and press releases: partnerships, funding or clients both sides stopped mentioning.
Old contact details. Phone numbers, street addresses and named individual contacts later replaced by a form. Straight into
```
whose-number-is-this
```
.
Pre-redaction copy. Diffing a page published in full against its edited successor shows exactly which words somebody paid a lawyer to remove — a very strong signal about what matters.
Forgotten files — document paths, backup filenames and admin surfaces indexed once and never cleaned up. See
```
secrets-in-git-history
```
.
The prior owner of a domain. A domain that changed hands shows a different business in its archive history — the fastest way to spot a reputation-laundering purchase.

从

collapse=digest

的结果中提取两个日期的原始捕获记录，并作为文本进行对比。被移除的内容就是证据：团队页面中某个人在两个日期之间消失，意味着此人已离职且有明确的时间范围，这也是

find-anyone

操作的线索，且此人不再有忠于原机构的理由。实际可挖掘的内容包括：

员工和团队页面：当前网站显示现有人员；归档则记录了所有曾任职的人员，包括职位、简介，以及机构改用联系表单之前的直接邮箱地址。
历史定价和条款——争议发生时的收费标准、承诺内容及退款政策。还包括已移除的帖子和新闻稿：双方都不再提及的合作、融资或客户信息。
旧联系方式：后来被表单取代的电话号码、街道地址和具体联系人。可直接用于
```
whose-number-is-this
```
操作。
编辑前的文本：将完整发布的页面与其编辑后的版本对比，能准确发现有人付费请律师删除的内容——这是关于核心关注点的强烈信号。
被遗忘的文件——文档路径、备份文件名和管理界面，被索引一次后从未清理。详见
```
secrets-in-git-history
```
。
域名的前所有者：易主的域名在归档历史中会显示不同的业务——这是发现声誉洗白式域名收购的最快方式。

Preserve it before you look further

在深入调查前先保存内容

Anything you find can be removed, and investigating sometimes causes the removal. Capture first, analyse second.

Submit the live URL to the Internet Archive's Save Page Now at

https://web.archive.org/save/

and to archive.today, so two independent organisations hold a timestamped copy you did not create. Then take your own record: a WARC preserves the full HTTP exchange including headers, which a screenshot does not.

bash

wget --warc-file=evidence-001 --page-requisites --no-parent https://example.com/pricing
sha256sum evidence-001.warc.gz | tee evidence-001.sha256

Record the capture time in UTC, the URL exactly as requested, and the hash. A screenshot alone is weak — trivially forged, no headers. A third-party archive URL plus your own hashed WARC is strong, because they corroborate each other and one is outside your control. That is what

write-the-intel-brief

expects.

你找到的任何内容都可能被移除，而调查有时会触发移除操作。先捕获，再分析。

将当前URL提交至Internet Archive的「立即保存页面」工具（地址为

https://web.archive.org/save/

）和archive.today，让两个独立机构保存你未创建的带时间戳副本。然后自行记录：WARC文件会保存包括请求头在内的完整HTTP交互，而截图做不到这一点。

bash

wget --warc-file=evidence-001 --page-requisites --no-parent https://example.com/pricing
sha256sum evidence-001.warc.gz | tee evidence-001.sha256

记录UTC时间的捕获时间、请求的完整URL以及哈希值。仅靠截图的证明力很弱——容易伪造，且无请求头。第三方归档URL加上你自己的带哈希WARC文件证明力很强，因为两者可相互佐证，且其中一个不受你控制。这也是

write-the-intel-brief

所要求的。

Verifying a capture is genuine

验证捕获内容的真实性

Anyone can put content into an archive: both Save Page Now and archive.today accept arbitrary URLs from anonymous submitters. A capture proves that URL served that content to that crawler at that time, and nothing about who controlled the URL. The realistic spoofing routes:

A forged screenshot of an archive page. Trivial. Never accept a screenshot of an archive; work from the archive URL itself and re-fetch it.
Content injected via the URL. A reflected query parameter, a user-generated content path or an open redirect lets someone archive attacker-chosen text on a legitimate-looking domain. Read the
```
original
```
field character by character: anything after
```
?
```
or
```
#
```
, any unfamiliar path segment, any lookalike host.
Capture after a domain changed hands. The domain was the buyer's, not the original business's. Check transfer dates against the capture timestamp with
```
who-owns-this-domain
```
.
Selective capture — a real page up for eleven minutes, archived by the person who put it up.

Corroboration, in order of strength: confirm the record exists independently in the CDX index with

statuscode

200 and a

digest

; look for adjacent captures made by the archive's own crawler rather than by on-demand submission, since crawler captures are much harder to stage; check a second independent archive near the same date; check consistency with the site's other history. A lone on-demand capture with no neighbours is an assertion by its submitter.

任何人都可以向归档提交内容：「立即保存页面」和archive.today都接受匿名提交的任意URL。一条捕获记录只能证明该URL在当时向该爬虫返回了该内容，无法证明谁控制了该URL。常见的伪造途径包括：

伪造的归档页面截图：非常容易。永远不要接受归档页面的截图；应直接从归档URL重新获取内容。
通过URL注入内容：反射型查询参数、用户生成内容路径或开放重定向，可让攻击者在看似合法的域名下归档其选择的文本。逐字符查看
```
original
```
字段：
```
?
```
或
```
#
```
之后的内容、任何不熟悉的路径段、任何相似域名。
域名易主后的捕获：该域名属于买家，而非原企业。使用
```
who-owns-this-domain
```
工具，将捕获时间与域名转移日期对比。
选择性捕获——真实页面仅上线11分钟，由发布者自行归档。

佐证力度从强到弱依次为：确认记录在CDX索引中独立存在，且

statuscode

为200并带有

digest

；查找由归档自身爬虫而非按需提交生成的相邻捕获记录，因为爬虫捕获记录更难伪造；检查同一日期附近的另一个独立归档记录；检查与网站其他历史内容的一致性。孤立的按需捕获记录无相邻记录，仅代表提交者的主张。

Where this goes wrong

常见误区

Absence is not evidence. No capture means unlinked, robots-blocked, behind a login, too obscure to crawl, or removed — not that the page never existed. Check two archives before writing "no record".
Exclusion looks like absence. An explicit exclusion notice means somebody acted; a plain 404 means nothing. Different facts — note which you got.
Retroactive removal. Wayback has historically applied a site's current robots.txt to its whole archive history, so a new
```
Disallow
```
could pull years of captures at once — making an expired-domain purchase plus a restrictive robots.txt a cheap way to launder a site's past. The practice has narrowed, but go to archive.today and Common Crawl when a history looks too clean.
Composite renders. Mixed-date resources mean the rendered page is a reconstruction. Cite the HTML capture timestamp, not "how it looked".
Capture time is not publication time. A page first captured in March may have been published in January. The capture bounds the date from above only.
Dynamic and personalised content. The crawler saw one variant — one geography, one A/B bucket, logged out. Prices and availability especially.
Sparse sampling hides everything. A page captured twice a year can change and revert with no trace;
```
collapse=digest
```
shows observed changes, not all changes. Timestamps are UTC, and forgetting that breaks a timeline by a day.

没有捕获记录不代表不存在：无捕获记录意味着内容未被链接、被robots协议阻止、需要登录、过于冷门未被爬取，或已被移除——而非页面从未存在过。在记录「无记录」前，请检查两个归档工具。
排除与缺失是不同的：明确的排除通知意味着有人采取了操作；单纯的404则无任何意义。这是两个不同的事实——请注意区分。
追溯性移除：Wayback曾有将网站当前的robots.txt应用于整个归档历史的做法，因此新的
```
Disallow
```
规则可能会一次性移除数年的捕获记录——这使得收购过期域名并设置严格的robots.txt成为洗白网站历史的廉价方式。这种做法现已减少，但如果某网站的历史记录过于「干净」，请使用archive.today和Common Crawl。
复合渲染：混合日期的资源意味着渲染页面是重构的。引用HTML的捕获时间戳，而非「页面显示效果」。
捕获时间不等于发布时间：首次捕获于3月的页面可能在1月就已发布。捕获时间仅能确定页面存在的最晚时间。
动态和个性化内容：爬虫只能看到一种变体——某个地区、某个A/B测试分组、未登录状态。尤其是价格和库存信息。
稀疏采样会隐藏所有变化：每年仅捕获两次的页面可能发生变化后又恢复原状，但无任何记录；
```
collapse=digest
```
仅显示观测到的变化，而非所有变化。时间戳为UTC时区，忽略这一点会导致时间线偏差一天。

Confidence grading

可信度分级

Confirmed — a crawler-initiated capture with
```
statuscode:200
```
, retrieved raw with
```
id_
```
, on a URL verified as the subject's, corroborated by a second archive or by adjacent captures in the same crawl series.
Probable — a single well-formed Wayback capture with no second source, or an archive.today capture consistent with the site's other history.
Unconfirmed — an on-demand capture with no neighbours; a URL carrying query parameters or user-generated content paths; a screenshot; content read from a composite replay rather than the raw response.
Rejected — a capture whose URL was not under the subject's control at that date, or one taken after a domain transfer and presented as the prior owner's.

Cite the full archive URL with timestamp, the original URL, the capture time in UTC, and how you retrieved it. An archive citation without a timestamp is not a citation.

已确认——由爬虫主动发起的捕获，
```
statuscode:200
```
，通过
```
id_
```
获取原始内容，URL已验证为目标主体所有，且有第二个归档或同一爬取系列中的相邻捕获记录佐证。
大概率真实——单个格式规范的Wayback捕获记录，无第二来源；或与网站其他历史内容一致的archive.today捕获记录。
未确认——无相邻记录的按需捕获记录；带有查询参数或用户生成内容路径的URL；截图；从复合回放而非原始响应中读取的内容。
已拒绝——捕获时URL不受目标主体控制的记录；或域名转移后捕获却被当作原所有者内容的记录。

引用时需包含完整的带时间戳归档URL、原始URL、UTC时区的捕获时间，以及获取方式。没有时间戳的归档引用不能算作有效引用。

Worked example

实战示例

A supplier's site claims an industry certification. The client wants it verified before signing.

collapse=digest

on the certifications page returns nine content changes over six years. The raw captures either side of the most recent one show the certification body's name changed — from a recognised accreditor to a similarly named entity — with no announcement.

The dead end: the accreditor's own register has no live entry either way, and its archived member lists replay as empty shells. Fetching them with

id_

shows why — the list was rendered client-side from an API call, so the crawler captured a page with no data in it. Nothing recoverable there.

CDX enumeration on

matchType=domain

then turns up a staging host holding an archived PDF of the original certificate. Its metadata carries an expiry date two years before the current site claims coverage, and

secrets-in-file-metadata

gets an author name from the same file.

Grade: the wording change confirmed — two crawler captures, raw, both read. The lapse probable — one document, no confirmation from the accreditor.

某供应商网站声称拥有行业认证。客户希望在签约前验证该认证的真实性。

对认证页面使用

collapse=digest

参数，返回六年内的九次内容变化。最新变化前后的原始捕获记录显示，认证机构名称已更改——从知名认证机构变为名称相似的实体——且无任何公告。

死胡同：认证机构自身的注册系统中也无有效记录，且其归档的会员列表回放为空框架。使用

id_

获取原始内容后发现原因——列表是通过API调用在客户端渲染的，因此爬虫捕获的页面中无数据。此处无任何可恢复内容。

随后使用

matchType=domain

参数进行CDX枚举，发现一个staging主机上保存着原始认证证书的归档PDF。其元数据显示证书已过期两年，而当前网站仍声称认证有效；通过

secrets-in-file-metadata

操作从该文件中获取了作者姓名。

分级：措辞变化已确认——两条爬虫捕获记录，均获取了原始内容。认证过期大概率真实——仅一份文档，无认证机构的确认。

Pivots

延伸操作

New selector	Goes to
Former staff names, titles, bios	`find-anyone` , `pattern-of-life-from-socials`
Email addresses from pre-form contact pages	`what-an-email-reveals`
Phone numbers and street addresses	`whose-number-is-this`
Subdomains and hosts from CDX enumeration	`find-hidden-subdomains` , `who-owns-this-domain`
Archived documents and PDFs	`secrets-in-file-metadata`
Archived images	`find-the-original-image` , `geolocate-from-pixels`
Forgotten paths and exposed file names	`google-like-a-spy` , `secrets-in-git-history`
Prior corporate identity of a domain	`x-ray-a-company` , `who-really-owns-it`
A timeline of edits and removals	`write-the-intel-brief` , `graph-the-network`

新筛选条件	对应操作
前员工姓名、职位、简介	`find-anyone` 、 `pattern-of-life-from-socials`
改用表单前联系页面中的邮箱地址	`what-an-email-reveals`
电话号码和街道地址	`whose-number-is-this`
CDX枚举得到的子域名和主机	`find-hidden-subdomains` 、 `who-owns-this-domain`
归档文档和PDF	`secrets-in-file-metadata`
归档图片	`find-the-original-image` 、 `geolocate-from-pixels`
被遗忘的路径和暴露的文件名	`google-like-a-spy` 、 `secrets-in-git-history`
域名的前企业身份	`x-ray-a-company` 、 `who-really-owns-it`
编辑和移除的时间线	`write-the-intel-brief` 、 `graph-the-network`

Legal and handling notes

法律和操作注意事项

Reading an archive is passive; it touches the archive's servers, not the target's. Fetching a recovered path against the live site is not passive and lands in the target's logs — decide deliberately which you are doing.

Archived personal data is still personal data. Content removed under a right-to-erasure request may persist in archives, and republishing it can create the liability the removal was meant to extinguish. Weigh the investigative need against the reason for removal, and minimise per ../../ETHICS.md. Reproducing substantial archived content in a published report is a separate copyright question from using it as evidence.

Bulk retrieval strains a nonprofit's infrastructure: rate-limit enumeration and raw fetches, and pull the index once rather than repeatedly.

读取归档内容是被动操作；仅会访问归档服务器，不会触及目标网站的服务器。向目标网站的当前服务器请求已恢复的路径则不属于被动操作，且会被目标网站记录日志——请明确决定你要执行的操作类型。

归档中的个人数据仍属于个人数据。根据被遗忘权请求移除的内容可能仍留存于归档中，重新发布这些内容可能会引发移除操作原本要规避的法律责任。请权衡调查需求与移除原因，并按照../../ETHICS.md要求最小化风险。在公开报告中复制大量归档内容，与将其用作证据涉及不同的版权问题。

批量检索会给非营利机构的基础设施带来压力：请限制枚举和原始内容获取的速率，且仅拉取一次索引而非重复拉取。