> For the complete documentation index, see [llms.txt](https://ascribed-dev.com/docs/llms.txt). This documentation describes Ascribe 0.4.4. Anything marked in main, not yet released isn't in that release yet.

# `ascribe.toml`: where pages are published

> Reference · How the content model sets the site's consumer, the builds, how pages are listed, what crawlers are told, and where retired permalinks went.

Part of the [`ascribe.toml` reference](https://ascribed-dev.com/docs/content-model.md), whose [contents](https://ascribed-dev.com/docs/content-model.md#contents) list every section. The rules for a loader are in the contract's [where pages are published](https://ascribed-dev.com/docs/contracts/content-model/output.md).

## 15\. `[consumer]`

The consumer profile ([SPEC §9.5](https://github.com/ascribed-dev/ascribe/blob/main/SPEC.md#95-consumer-profile)): how the site output fits a specific consumer. Spec 0.1's toolchain implements one profile, `astro` ([SPEC §9.6](https://github.com/ascribed-dev/ascribe/blob/main/SPEC.md#96-astro)). The profile sets defaults for every other key; a key given here overrides them, within the values the profile supports.

```toml
[consumer]
profile = "astro"
site = "https://docs.quill.dev"
base-path = "/"
trailing-slash = "always"
slugger = "github"
```

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `profile` | string | `"astro"` | The consumer profile. Supported: `"astro"`. |
| `site` | string (URL) | none | The published site's origin, such as `"https://docs.quill.dev"`: an `http` or `https` URL with no path, query, or [fragment](https://ascribed-dev.com/docs/glossary.md#fragment). The plain-markdown output needs it to write absolute links ([SPEC §9.4](https://github.com/ascribed-dev/ascribe/blob/main/SPEC.md#94-outputs)). Without it, plain-markdown links are root-relative (they start with `base-path`) and `ascribe build` warns. |
| `base-path` | string | `"/"` | **Routing.** The URL path every route starts with, such as `"/docs/"`, including any locale prefix (`"/en/"`). It must start with `/`. A trailing `/` is optional and doesn't change the meaning. |
| `trailing-slash` | string | `"always"` | **Routing.** Whether page URLs end in `/`: `"always"` (`/guides/setup/`) or `"never"` (`/guides/setup`). Match the consumer's own setting (Astro's `trailingSlash` and `build.format`). |
| `slugger` | string | `"github"` | **Slugging.** The algorithm for heading ids ([SPEC §5.5](https://github.com/ascribed-dev/ascribe/blob/main/SPEC.md#55-heading-ids)), which must be the one the consumer uses. Supported: `"github"`, a port of `github-slugger`, which Astro uses. |
| `html` | boolean | `true` | **HTML passthrough.** Whether the consumer renders raw HTML in markdown. The site output's custom elements need it, so the `astro` profile supports only `true`. |
| `agents` | boolean | `true` when `site` is set, `false` when it isn't | **Outputs for agents.** Whether each build also publishes what AI agents read, as the [Web Documentation Delivery Spec](https://agentdocsspec.com/spec/web/) lays it out: `llms.txt`, the index of every page, and each page's Markdown at its URL with `.md`. It needs `site`, since every link agents read is absolute: a site with an address publishes them unless this is `false`. See below. |
| `permalink-path` | string | `go/` under `base-path` | **[Permalinks](https://ascribed-dev.com/docs/glossary.md#permalink).** Where each [permalink](https://ascribed-dev.com/docs/writing/directives.md#permalinks)'s address is published, as a path from the site's root, such as `"/go/"`: `/go/cli-check/` forwards to the heading marked `cli-check`. It must start with `/`. Set it when the addresses a product prints shouldn't move when `base-path` does. With `@ascribed/astro`, it must be under Astro's `base`, since an Astro site serves nothing outside it, and the integration fails the build when it isn't. |
| `host` | string | none | **The host.** The host the site is deployed to, whose files the compiler writes into each build's site output, in `_ascribe/host/`: the redirects `ascribe.addresses` records, and the response headers a page's Markdown version needs, so that one that isn't listed isn't indexed either. Supported: `"netlify"`, the `_redirects` and `_headers` files Netlify reads. `@ascribed/astro` places them at the root of the built site ([the host's files](https://ascribed-dev.com/docs/publishing/site.md#the-hosts-files)); a site built another way copies them there. Without it, nothing is written for a host, and the redirects and headers are in `_ascribe/redirects.json` and `_ascribe/headers.json` for a deploy step to read. See below. |

**Heading ids, image attributes, and assets** ([SPEC §9.5](https://github.com/ascribed-dev/ascribe/blob/main/SPEC.md#95-consumer-profile)) are part of the profile, not keys. The `astro` profile has exactly one way to do each:

- **Heading ids, image attributes, and glossary links:** the site output writes each as a `<ascribe-attributes>` marker that the consumer's markdown plugin applies, so the consumer keeps its own heading, table-of-contents, and image processing, and a glossary link carries `data-ascribe-term`.
- **Assets:** copies mirror their source paths inside each output, and images are referenced relatively so Astro's image processing still applies. Other files a page links to are published under `_ascribe/files/`.

A later profile that offers a choice will add a key for it.

**How file paths become routes** (the `astro` profile): a page's route is `base-path`, then its path relative to the content root with the `.md` extension removed and each segment slugged the way Astro's content loader computes entry ids; a final `index` segment is dropped (`guides/index.md` → `/guides/`); then the trailing slash per `trailing-slash`. The root `index.md` (Astro's entry id `index`) is at `base-path`, which under `trailing-slash = "never"` loses its final `/` unless it's `/`. Two pages with one entry id (`My File.md` and `my-file.md`, or `index.md` and `index/index.md`) can't both be published, and `ascribe build --emit site` fails, naming them. The same router answers the reverse question, which page a route-like link names, for the `link-route` warning and its fix. Source files never contain routes ([SPEC §5.2](https://github.com/ascribed-dev/ascribe/blob/main/SPEC.md#52-links)).

**The outputs for agents** (`agents`, on by default when `site` is set) change what the plain-markdown output is, and add to the site output:

- **The plain-markdown output is laid out by URL,** so it can be copied as it is to `base-path` beside the site: each page at its route with `.md` (`Guides/My Setup.md` → `guides/my-setup.md`, the root `index.md` → `index.md`), each opening with a line pointing at `llms.txt`, and after its title saying what the page is for, its type and its description (`> Reference · Every option.`); the files they link to under `_ascribe/files/`, linked by absolute URL; and `llms.txt`. A link to a page the build publishes, a glossary link included, leads to that page's Markdown, so an agent that follows it stays in Markdown, and its `#fragment` to the heading's anchor there: the slug of the heading as the Markdown writes it. That's the heading's id on the page, unless it has an `@id`, or is in a variant arm, whose label it ends with in the Markdown.
- **`llms.txt`** lists every page the build publishes as `- [Title](../url): description`, linking to its Markdown. Its heading is the title of the page at `base-path`, and its summary that page's description. Its sections are the project's [navigation](https://ascribed-dev.com/docs/navigation.md)'s: the content root's pages first, under its label, **Start** unless `ascribe.nav` gives another, then each top section, in order, named by its label, with the pages of the sections nested in it in reading order. A page is listed by its title, not its `nav-label`. With `[releases]`, its summary, and each page's opening line, end with the releases the build describes: a versioned build's line, the version a build filters for, or else each target's live lines, adding that what's marked planned is in none of them yet ([output-layout contract](https://ascribed-dev.com/docs/contracts/output-layout.md)). An index over 50,000 characters is split into one `llms.txt` per top section, which the root file links to. A page's description is the field its type marks [`role = "description"`](https://ascribed-dev.com/docs/content-model.md#54-field-roles), as plain text, or the field's `default` when the page leaves the field out or empty; a type without one lists its pages without descriptions.
- **Each page of the site output** opens with a pointer to `llms.txt` and the page's Markdown, which the element library hides from sight ([element contract §8](https://github.com/ascribed-dev/ascribe/blob/main/packages/elements/CONTRACT.md#8-ascribe-for-agents)).

**The headers of a Markdown version.** Each build lists the response headers each Markdown version needs, `X-Robots-Tag` and a canonical `Link`, in `_ascribe/headers.json` in the site output ([output-layout contract](https://ascribed-dev.com/docs/contracts/output-layout.md#14-the-headers)), and with `host = "netlify"`, the build writes them in Netlify's `_headers` too, which `@ascribed/astro` places at the site's root, beside the `_redirects` that [`ascribe.addresses`](https://ascribed-dev.com/docs/publishing/redirects.md) gives. Why a host sends them is in [Publishing](https://ascribed-dev.com/docs/publishing.md#the-markdown-versions-headers).

`@ascribed/astro` publishes the plain output with the site, and serves it in `astro dev`. With another generator, copy `<output-dir>/<build>/plain/` to the site's `base-path`. `ascribe check` then reports two advice checks: a description too long for its line in `llms.txt` (`description-too-long`), on the page that writes it, or once on a field's `default` in `ascribe.toml`, and an `llms.txt` still over the limit when it's split (`llms-section-large`).

The `astro` profile's `site`, `base-path`, and `trailing-slash` repeat settings from `astro.config`. The Astro integration checks that they agree, and fails the build, naming each difference, when they don't. It compares `base-path` as a path with a leading and a trailing `/`, treats Astro's `trailingSlash: "ignore"` as agreeing with either value, and compares `site` by origin when both sides set it.

---

## 16\. `[builds.<name>]`

Named builds ([SPEC §9.3](https://github.com/ascribed-dev/ascribe/blob/main/SPEC.md#93-build-modes)). Each sets a variant mode and an [availability](https://ascribed-dev.com/docs/glossary.md#availability) mode. These are [SPEC §9.3](https://github.com/ascribed-dev/ascribe/blob/main/SPEC.md#93-build-modes)'s examples, in `ascribe.toml`:

```toml
[builds.site]
variants = "switch"
availability = "badge"

[builds.cloud-pdf]
variants = { deployment = "cloud" }
availability = { filter = "cloud" }

[builds."sm-3.3"]
variants = "switch"
availability = { filter = "self-managed 3.3" }
```

`<name>` is the build name (§1.2). It names the build on the command line (`ascribe build --build cloud`) and in output paths.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `variants` | string or table | `"switch"` | The variant mode. `"switch"` keeps every arm and page. A **selection** is a table from [dimension](https://ascribed-dev.com/docs/glossary.md#dimension) names to a value or an array of values: `{ deployment = "cloud" }`, `{ pm = ["npm", "pnpm"] }`. Every dimension MUST be declared and every value a value of that dimension. An empty table is an error; write `"switch"`. |
| `availability` | string or table | `"badge"` | The availability mode. `"badge"` keeps everything and annotates it. `{ filter = "<target> [<version>]" }` removes content not available for the target at the version. The target MUST be a declared dimension value, not a dimension name. A versioned target MUST have a version, and a versionless one MUST NOT. The version follows the version scheme (§7). |
| `versions` | string | none | The target the build is **versioned** by, such as `"self-managed"`: one output for each of its live lines (§7.1). The target MUST have a `[releases]` table with at least one live line, and the build's `availability` MUST be `"badge"`. |

**Rules.** Two build names can't differ only in case (`model-build-name-case`): they become folder names, and some file systems ignore case. A versioned build needs a target with a live line, and can't filter as well (`model-build-versions`). A build that filters for a target its own selection leaves out, such as one selecting `deployment = "cloud"` and filtering for `self-managed 3.3`, builds, but it's almost certainly a mistake, so you get a warning (`model-build-filter-excluded`). A build that filters for a version older than every line `[releases.<target>]` declares is in no line the project knows, so `ascribe versions` doesn't list it and a retirement counts it as a build its rewrite changes: that's advice (`build-filter-line-undeclared`), to declare the line in `archived` or to publish the lines with `versions` instead.

A versioned build's outputs, their names and addresses, and what each line keeps and shows are in [Versions](https://ascribed-dev.com/docs/versions.md#3-version-a-build). An older line's output says nothing to agents: it has no `llms.txt`, and its pages point to no Markdown version.

---

## 17\. `[listing]`

How pages are listed, set once for many: the whole site, the pages under a folder, or the pages of a content type. Each value is one of the values of a page's `listing` ([SPEC §2.1](https://github.com/ascribed-dev/ascribe/blob/main/SPEC.md#21-files)), from open to closed: `listed`, `unquoted`, `unlisted`.

```toml
[listing]
default = "listed"

[listing.folders]
"internal/" = "unlisted"

[listing.types]
changelog = "unquoted"
```

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `default` | string | `"listed"` | Every page's listing. |
| `folders` | table of strings | `{}` | A listing for every page under a folder, keyed by the folder's path from the content root, ending in `/`, with its name's case as it is on disk. A folder's pages are every page whose path starts with it, in folders under it too. |
| `types` | table of strings | `{}` | A listing for every page of a content type, keyed by the type's name. |

A page's listing is the strictest of its own and the rules it's under: the site's `default`, every folder it's under, and its type's. A folder and a type have no order between them: a page under both gets the stricter. A rule can only restrict. A page can set the same value as its rule, or a stricter one, and a page that sets a looser one is an error at the page (`listing-loosened`), since the project contradicts itself. A folder under another folder, and a folder or a type under the `default`, can't be looser than it (`model-listing-looser`).

**`older-lines`** (a string, `"listed"` by default) is the listing of every page of a live line of a versioned build other than its current line: `site@3.4`'s, and not `site`'s. It governs search engines and agents together. A page in an older line is under it as well as the rules above, between `default` and a folder's or a type's; it can't be looser than `default` (`model-listing-looser`); and a page that sets a value looser than it is an error in each older line that publishes it (`listing-loosened-in-line`), and fine in the current line. An older line's pages point their canonical links at the current line's, so a search engine sends readers to the current line however it's set: `older-lines = "unlisted"` keeps an older line out of the index altogether, its pages out of the sitemap and `llms.txt`, for a project whose old pages still outrank its new ones.

A `default` stricter than `listed` is for a site that isn't public yet: no page in it can be listed until the default changes, and a page that says otherwise is the same error.

Each page's record, in `_ascribe/pages.json`, says where its listing came from: the page, the rule's level and its line in `ascribe.toml`, or neither ([output-layout contract](https://ascribed-dev.com/docs/contracts/output-layout.md) §1.3).

What each value tells search engines and agents is in [Publishing](https://ascribed-dev.com/docs/publishing.md#how-a-page-is-listed).

In VS Code, the sidebar's [This page](https://ascribed-dev.com/docs/editor/sidebar.md#this-page) says whether a page is listed, and which rule says so; choosing the row opens the rule.

In the editor, `listing:` in a page's frontmatter completes its three values, and hovering it says which rule the page is under. A page looser than its rule has a quick fix that removes its `listing`.

**Rules.** Each value is one of the three (`model-invalid-value`). A folder is a path from the content root ending in `/`, with no `.` or `..` part or pattern characters, of a folder that's there, with its name's case as on disk (`model-listing-folder`). A type is one the project declares, or `page` when it declares none (`model-listing-type`). A rule isn't looser than a folder's it's under, or than `default` (`model-listing-looser`). A page isn't looser than a rule it's under (`listing-loosened`). No other key is allowed in `[listing]` (`model-unknown-key`).

In an older line of a versioned build, a page isn't looser than `older-lines` either (`listing-loosened-in-line`).

---

## 18\. `[crawlers]`

What the project says to the crawlers that read its site: search engines, AI assistants, and anything else that fetches pages without a person. Each build writes it into its page records, `_ascribe/pages.json` in the site output ([output-layout contract](https://ascribed-dev.com/docs/contracts/output-layout.md) §1.3), for a site to write its `robots.txt` from; `@ascribed/astro` writes it ([`robots.txt`](https://ascribed-dev.com/docs/publishing/site.md#robotstxt)).

```toml
[crawlers]
training = "disallow"
search = "allow"
on-request = "allow"
lines = [
  "Disallow: /drafts/",
  "User-agent: ExampleBot",
  "Disallow: /",
]
```

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `training` | `"allow"` or `"disallow"` | none | What crawlers gathering text to train a model may do. `disallow` writes one group naming every training crawler Ascribe knows, with `Disallow: /`. |
| `search` | `"allow"` or `"disallow"` | none | What crawlers building the index an AI search product answers from may do, written the same way. |
| `on-request` | `"allow"` or `"disallow"` | none | What fetchers that read a page because a person asked an assistant about it may do, written the same way. |
| `lines` | array of strings | `[]` | The project's own `robots.txt` lines, in order. Each is one line, not empty. A path in an `Allow` or `Disallow` line is from the project's own root, and the records hold it from the host's, with `base-path` before it: under `base-path = "/docs/"`, `Disallow: /drafts/` is `Disallow: /docs/drafts/`. Any other line is kept as written. The lines before any `User-agent` line are for every crawler, in the `User-agent: *` group. A `User-agent` line that names a crawler starts a group the project writes whole: a crawler obeys only the most specific group that names it, and not the `*` group as well. A crawler a disallowed purpose names can't be named in `lines` too, and no crawler can be named in two groups of `lines`. |

`allow` writes nothing, and neither does leaving a purpose out: a group that named those crawlers and said `Allow: /` would take them out of the `*` group, and with it out of every path `lines` disallows, drafts and all. So an allowed crawler stays under the lines for every crawler. `allow` is the project writing its choice down. A project that says nothing gets no rules about AI crawlers.

`robots.txt` has the purposes' groups first, in the order above, then the `User-agent: *` group with the lines for every crawler, then the groups `lines` writes whole, then the sitemap's address.

What each purpose's choice costs, and why a page is kept out of search by its listing and not by `robots.txt`, is in [Publishing](https://ascribed-dev.com/docs/publishing.md#what-crawlers-are-told).

`ascribe check` reads each search crawler the way it reads the file: the one group that applies to it. A page that isn't listed is reported when a search crawler may not fetch it, whether the lines for every crawler say so, a group of `lines` that names that crawler, or `search = "disallow"` (`listing-disallowed`).

### Crawlers by purpose

Each operator below runs a bot for each purpose, or a token that says what it may do with what its other bots fetch, and says on its own page which is which. A purpose's group names each of them. A bot whose operator states no single purpose isn't here, such as Common Crawl's `CCBot`, whose archive many train on, or Meta's `meta-externalagent`, which gathers for training and for indexing both; name such a bot in `lines`, in a group of its own. Each row says what its operator said on the day in its last column, linked to the page.

| Purpose | Token | Operator | What the operator says of `robots.txt` | Read |
| --- | --- | --- | --- | --- |
| `training` | `GPTBot` | OpenAI | Follows it. | [2026-10-12](https://developers.openai.com/api/docs/bots) |
| `training` | `ClaudeBot` | Anthropic | Follows it. | [2026-10-12](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| `training` | `Google-Extended` | Google | Follows it. A token, not a crawler: Googlebot fetches, and this says what Google may do with it. It covers training Gemini models and grounding answers in Gemini Apps, and doesn't affect Google Search. | [2026-10-12](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) |
| `training` | `Applebot-Extended` | Apple | Follows it. A token, not a crawler: Applebot fetches, and this says whether Apple may train its models on it. Pages it disallows can still appear in Apple's search. | [2026-10-12](https://support.apple.com/en-us/119829) |
| `training` | `MistralAI-Training` | Mistral AI | Follows it. | [2026-10-12](https://docs.mistral.ai/robots) |
| `search` | `OAI-SearchBot` | OpenAI | Follows it. | [2026-10-12](https://developers.openai.com/api/docs/bots) |
| `search` | `Claude-SearchBot` | Anthropic | Follows it. | [2026-10-12](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| `search` | `PerplexityBot` | Perplexity | Follows it. | [2026-10-12](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| `search` | `Amzn-SearchBot` | Amazon | Follows it. When `robots.txt` doesn't name it, it follows the rules given to other search bots. | [2026-10-12](https://developer.amazon.com/amazonbot) |
| `search` | `MistralAI-Index` | Mistral AI | Follows it. | [2026-10-12](https://docs.mistral.ai/robots) |
| `on-request` | `ChatGPT-User` | OpenAI | May not follow it. | [2026-10-12](https://developers.openai.com/api/docs/bots) |
| `on-request` | `Claude-User` | Anthropic | Follows it. | [2026-10-12](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| `on-request` | `Perplexity-User` | Perplexity | May not follow it. | [2026-10-12](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| `on-request` | `Amzn-User` | Amazon | May not follow it. | [2026-10-12](https://developer.amazon.com/amazonbot) |
| `on-request` | `MistralAI-User` | Mistral AI | Follows it. | [2026-10-12](https://docs.mistral.ai/robots) |
| `on-request` | `Meta-ExternalFetcher` | Meta | May not follow it. | [2026-10-12](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) |

The list changes, as operators add bots and rename them. Ascribe's release process rereads every source, and a newer Ascribe may name more crawlers for a purpose than an older one did.

**Rules.** Each line is one line (`model-crawlers-line`) and not empty (`model-empty-text`). A purpose is `allow` or `disallow` (`model-invalid-value`). A crawler is named in one group: not in a group of `lines` that a disallowed purpose names, and not in two groups of `lines` (`model-crawlers-named-twice`). Projects that build one site must agree on each purpose, since it has one `robots.txt`: one that allows a purpose and one that disallows it fail the site's build, naming both. No other key is allowed in `[crawlers]` (`model-unknown-key`).

---

## 19\. `[permalinks]`

`ascribe.toml` has no `[permalinks]` any more. What happened to a [permalink](https://ascribed-dev.com/docs/writing/directives.md#permalinks) that no heading or page declares now, retired with a reason, or moved to another permalink or to a page, is recorded in [`ascribe.addresses`](https://ascribed-dev.com/docs/publishing/redirects.md#ascribeaddresses), keyed `go:<name>`, with the rest of where the docs' addresses went (SPEC §7.10). The [guide](https://ascribed-dev.com/docs/permalinks.md#when-a-section-moves-or-goes-away) says when to use each.

[`ascribe addresses init`](https://ascribed-dev.com/docs/publishing/redirects.md#ascribe-addresses-init) moves a project's `[permalinks.retired]` and `[permalinks.aliases]` into `ascribe.addresses`, and removes the table. Until it does, the table is an error at the key that says so (`model-unknown-key`).