ascribe.toml: where pages are published

How the content model sets the site’s consumer, the builds, how pages are listed, what crawlers are told, and where retired permalinks went.

Part of the ascribe.toml reference, whose contents list every section. The rules for a loader are in the contract’s where pages are published.

15. [consumer]

The consumer profile (SPEC §9.5): how the site output fits a specific consumer. Spec 0.1’s toolchain implements one profile, astro (SPEC §9.6). The profile sets defaults for every other key; a key given here overrides them, within the values the profile supports.

[consumer]
profile = "astro"
site = "https://docs.quill.dev"
base-path = "/"
trailing-slash = "always"
slugger = "github"

Heading ids, image attributes, and assets (SPEC §9.5) are part of the profile, not keys. The astro profile has exactly one way to do each:

• Heading ids, image attributes, and glossary links: the site output writes each as a <ascribe-attributes> marker that the consumer’s markdown plugin applies, so the consumer keeps its own heading, table-of-contents, and image processing, and a glossary link carries data-ascribe-term.
• Assets: copies mirror their source paths inside each output, and images are referenced relatively so Astro’s image processing still applies. Other files a page links to are published under _ascribe/files/.

A later profile that offers a choice will add a key for it.

How file paths become routes (the astro profile): a page’s route is base-path, then its path relative to the content root with the .md extension removed and each segment slugged the way Astro’s content loader computes entry ids; a final index segment is dropped (guides/index.md → /guides/); then the trailing slash per trailing-slash. The root index.md (Astro’s entry id index) is at base-path, which under trailing-slash = "never" loses its final / unless it’s /. Two pages with one entry id (My File.md and my-file.md, or index.md and index/index.md) can’t both be published, and ascribe build --emit site fails, naming them. The same router answers the reverse question, which page a route-like link names, for the link-route warning and its fix. Source files never contain routes (SPEC §5.2).

The outputs for agents (agents, on by default when site is set) change what the plain-markdown output is, and add to the site output:

• The plain-markdown output is laid out by URL, so it can be copied as it is to base-path beside the site: each page at its route with .md (Guides/My Setup.md → guides/my-setup.md, the root index.md → index.md), each opening with a line pointing at llms.txt, and after its title saying what the page is for, its type and its description (> Reference · Every option.); the files they link to under _ascribe/files/, linked by absolute URL; and llms.txt. A link to a page the build publishes, a glossary link included, leads to that page’s Markdown, so an agent that follows it stays in Markdown, and its #fragment to the heading’s anchor there: the slug of the heading as the Markdown writes it. That’s the heading’s id on the page, unless it has an @id, or is in a variant arm, whose label it ends with in the Markdown.
• llms.txt lists every page the build publishes as - [Title](../url): description, linking to its Markdown. Its heading is the title of the page at base-path, and its summary that page’s description. Its sections are the project’s navigation’s: the content root’s pages first, under its label, Start unless ascribe.nav gives another, then each top section, in order, named by its label, with the pages of the sections nested in it in reading order. A page is listed by its title, not its nav-label. With [releases], its summary, and each page’s opening line, end with the releases the build describes: a versioned build’s line, the version a build filters for, or else each target’s live lines, adding that what’s marked planned is in none of them yet (output-layout contract). An index over 50,000 characters is split into one llms.txt per top section, which the root file links to. A page’s description is the field its type marks role = "description", as plain text, or the field’s default when the page leaves the field out or empty; a type without one lists its pages without descriptions.
• Each page of the site output opens with a pointer to llms.txt and the page’s Markdown, which the element library hides from sight (element contract §8).

The headers of a Markdown version. Each build lists the response headers each Markdown version needs, X-Robots-Tag and a canonical Link, in _ascribe/headers.json in the site output (output-layout contract), and with host = "netlify", the build writes them in Netlify’s _headers too, which @ascribed/astro places at the site’s root, beside the _redirects that ascribe.addresses gives. Why a host sends them is in Publishing.

@ascribed/astro publishes the plain output with the site, and serves it in astro dev. With another generator, copy <output-dir>/<build>/plain/ to the site’s base-path. ascribe check then reports two advice checks: a description too long for its line in llms.txt (description-too-long), on the page that writes it, or once on a field’s default in ascribe.toml, and an llms.txt still over the limit when it’s split (llms-section-large).

The astro profile’s site, base-path, and trailing-slash repeat settings from astro.config. The Astro integration checks that they agree, and fails the build, naming each difference, when they don’t. It compares base-path as a path with a leading and a trailing /, treats Astro’s trailingSlash: "ignore" as agreeing with either value, and compares site by origin when both sides set it.

16. [builds.<name>]

Named builds (SPEC §9.3). Each sets a variant mode and an availability mode. These are SPEC §9.3’s examples, in ascribe.toml:

[builds.site]
variants = "switch"
availability = "badge"

[builds.cloud-pdf]
variants = { deployment = "cloud" }
availability = { filter = "cloud" }

[builds."sm-3.3"]
variants = "switch"
availability = { filter = "self-managed 3.3" }

<name> is the build name (§1.2). It names the build on the command line (ascribe build --build cloud) and in output paths.

Rules. Two build names can’t differ only in case (model-build-name-case): they become folder names, and some file systems ignore case. A versioned build needs a target with a live line, and can’t filter as well (model-build-versions). A build that filters for a target its own selection leaves out, such as one selecting deployment = "cloud" and filtering for self-managed 3.3, builds, but it’s almost certainly a mistake, so you get a warning (model-build-filter-excluded). A build that filters for a version older than every line [releases.<target>] declares is in no line the project knows, so ascribe versions doesn’t list it and a retirement counts it as a build its rewrite changes: that’s advice (build-filter-line-undeclared), to declare the line in archived or to publish the lines with versions instead.

A versioned build’s outputs, their names and addresses, and what each line keeps and shows are in Versions. An older line’s output says nothing to agents: it has no llms.txt, and its pages point to no Markdown version.

17. [listing]

How pages are listed, set once for many: the whole site, the pages under a folder, or the pages of a content type. Each value is one of the values of a page’s listing (SPEC §2.1), from open to closed: listed, unquoted, unlisted.

[listing]
default = "listed"

[listing.folders]
"internal/" = "unlisted"

[listing.types]
changelog = "unquoted"

A page’s listing is the strictest of its own and the rules it’s under: the site’s default, every folder it’s under, and its type’s. A folder and a type have no order between them: a page under both gets the stricter. A rule can only restrict. A page can set the same value as its rule, or a stricter one, and a page that sets a looser one is an error at the page (listing-loosened), since the project contradicts itself. A folder under another folder, and a folder or a type under the default, can’t be looser than it (model-listing-looser).

older-lines (a string, "listed" by default) is the listing of every page of a live line of a versioned build other than its current line: site@3.4’s, and not site’s. It governs search engines and agents together. A page in an older line is under it as well as the rules above, between default and a folder’s or a type’s; it can’t be looser than default (model-listing-looser); and a page that sets a value looser than it is an error in each older line that publishes it (listing-loosened-in-line), and fine in the current line. An older line’s pages point their canonical links at the current line’s, so a search engine sends readers to the current line however it’s set: older-lines = "unlisted" keeps an older line out of the index altogether, its pages out of the sitemap and llms.txt, for a project whose old pages still outrank its new ones.

A default stricter than listed is for a site that isn’t public yet: no page in it can be listed until the default changes, and a page that says otherwise is the same error.

Each page’s record, in _ascribe/pages.json, says where its listing came from: the page, the rule’s level and its line in ascribe.toml, or neither (output-layout contract §1.3).

What each value tells search engines and agents is in Publishing.

In VS Code, the sidebar’s This page says whether a page is listed, and which rule says so; choosing the row opens the rule.

In the editor, listing: in a page’s frontmatter completes its three values, and hovering it says which rule the page is under. A page looser than its rule has a quick fix that removes its listing.

Rules. Each value is one of the three (model-invalid-value). A folder is a path from the content root ending in /, with no . or .. part or pattern characters, of a folder that’s there, with its name’s case as on disk (model-listing-folder). A type is one the project declares, or page when it declares none (model-listing-type). A rule isn’t looser than a folder’s it’s under, or than default (model-listing-looser). A page isn’t looser than a rule it’s under (listing-loosened). No other key is allowed in [listing] (model-unknown-key).

In an older line of a versioned build, a page isn’t looser than older-lines either (listing-loosened-in-line).

18. [crawlers]

What the project says to the crawlers that read its site: search engines, AI assistants, and anything else that fetches pages without a person. Each build writes it into its page records, _ascribe/pages.json in the site output (output-layout contract §1.3), for a site to write its robots.txt from; @ascribed/astro writes it (robots.txt).

[crawlers]
training = "disallow"
search = "allow"
on-request = "allow"
lines = [
  "Disallow: /drafts/",
  "User-agent: ExampleBot",
  "Disallow: /",
]

allow writes nothing, and neither does leaving a purpose out: a group that named those crawlers and said Allow: / would take them out of the * group, and with it out of every path lines disallows, drafts and all. So an allowed crawler stays under the lines for every crawler. allow is the project writing its choice down. A project that says nothing gets no rules about AI crawlers.

robots.txt has the purposes’ groups first, in the order above, then the User-agent: * group with the lines for every crawler, then the groups lines writes whole, then the sitemap’s address.

What each purpose’s choice costs, and why a page is kept out of search by its listing and not by robots.txt, is in Publishing.

ascribe check reads each search crawler the way it reads the file: the one group that applies to it. A page that isn’t listed is reported when a search crawler may not fetch it, whether the lines for every crawler say so, a group of lines that names that crawler, or search = "disallow" (listing-disallowed).

Crawlers by purpose

Each operator below runs a bot for each purpose, or a token that says what it may do with what its other bots fetch, and says on its own page which is which. A purpose’s group names each of them. A bot whose operator states no single purpose isn’t here, such as Common Crawl’s CCBot, whose archive many train on, or Meta’s meta-externalagent, which gathers for training and for indexing both; name such a bot in lines, in a group of its own. Each row says what its operator said on the day in its last column, linked to the page.

The list changes, as operators add bots and rename them. Ascribe’s release process rereads every source, and a newer Ascribe may name more crawlers for a purpose than an older one did.

Rules. Each line is one line (model-crawlers-line) and not empty (model-empty-text). A purpose is allow or disallow (model-invalid-value). A crawler is named in one group: not in a group of lines that a disallowed purpose names, and not in two groups of lines (model-crawlers-named-twice). Projects that build one site must agree on each purpose, since it has one robots.txt: one that allows a purpose and one that disallows it fail the site’s build, naming both. No other key is allowed in [crawlers] (model-unknown-key).

19. [permalinks]

ascribe.toml has no [permalinks] any more. What happened to a permalink that no heading or page declares now, retired with a reason, or moved to another permalink or to a page, is recorded in ascribe.addresses, keyed go:<name>, with the rest of where the docs’ addresses went (SPEC §7.10). The guide says when to use each.

ascribe addresses init moves a project’s [permalinks.retired] and [permalinks.aliases] into ascribe.addresses, and removes the table. Until it does, the table is an error at the key that says so (model-unknown-key).