Skip to content
Boeks

Appearance

Theme

Metadata

How Boeks reads a book's details from its EPUB, and writes edits back into it.

3. Metadata

3.1 Reading

Metadata is read from the OPF of EPUB 2 and EPUB 3 files. Only the title is required.

  • Title and subtitle come from EPUB 3's title-type (main, subtitle). Without one, the first title is the title, and the subtitle comes from boeks:subtitle. Other title types, such as collection and short, are kept.
  • Creators have MARC relator roles (aut, edt, trl, ill, nrt), and a file-as for sorting. The main title can have a file-as too.
  • Series come from EPUB 3's belongs-to-collection with collection-type series and a group-position, which may be a decimal. A book can be in several series. EPUB 2 has no series, so there they come from boeks:series and boeks:series_position.
  • Identifiers are free-form scheme:value, with the scheme names Calibre uses: isbn, goodreads, hardcover, openlibrary, google, url, uuid, and others. Amazon's IDs are asin, an ebook's and an audiobook's alike, as the ID is called; Calibre's amazon, amazon_<country>, and mobi-asin are read as asin. Boeks adds inventaire, a work's Inventaire URI: its Wikidata item (wd:Q190192), or Inventaire's own entity for one Wikidata hasn't (inv:…). A wikidata item stored by another tool (Q190192) is read as the same work.
  • Other apps' fields, such as Calibre's calibre:series and its custom columns, aren't read.
<metadata>
  <dc:title id="t1">Dune</dc:title>
  <meta refines="#t1" property="title-type">main</meta>
  <dc:creator id="c1">Frank Herbert</dc:creator>
  <meta refines="#c1" property="role" scheme="marc:relators">aut</meta>
  <meta refines="#c1" property="file-as">Herbert, Frank</meta>
  <meta property="belongs-to-collection" id="s1">Dune</meta>
  <meta refines="#s1" property="collection-type">series</meta>
  <meta refines="#s1" property="group-position">1</meta>
</metadata>

3.2 Writing

  • Only the OPF changes, and a new cover's image (§3.6). Unknown elements are kept, and an EPUB 2 file stays EPUB 2.

  • The zip is deterministic, so the same edits give byte-identical files:

    • mimetype comes first, then the other entries in the original file's order, then any files a transform or a new cover adds, by name.
    • Entries that didn't change are copied as they are, compressed bytes and headers included.
    • Entries written anew, mimetype and the OPF among them, are stored uncompressed, dated 1980-01-01 00:00 (the earliest a zip can hold), with Unix permissions 0644.

    Devices running the same Boeks write the same file. Another tool following this writes the same entries, but the zip's headers have details this doesn't fix (such as the version that made it, flags, and extra fields), and how the OPF is serialized isn't specified, so its file may differ in bytes: the recovery below handles that.

  • EPUB 3 gets the standard forms (title-type, belongs-to-collection). EPUB 2 has no place for some fields, so they're written as boeks: metas: boeks:subtitle, and boeks:series and boeks:series_position for one series.

  • The format's own fields use the boeks: prefix.

  • Other apps' fields are neither written nor removed.

  • Never written:

    • user fields, such as rating, status, and collections, which are in Book.md (§6.2)
    • DRM'd files
    • files that fail validation

    The edits to a file that can't be written stay in the logs, and the clients show them over the file's own metadata, which the index keeps as the file has it (§4, §12.2).

3.3 Edits are logged, then applied

Edits are written into the EPUB from the logs, like the files of §6.

  • Logging: an edit is a metadata.updated in the editing device's log, for one field of one edition.
  • Applying: a device that has the EPUB, downloaded and writable, writes the pending edits into its OPF. It then logs metadata.applied with the file's hash before and after, and the edits it wrote.
  • Not every device applies. A device may leave applying to others; it still logs edits.
  • Two devices applying at once is safe:
    1. The output is deterministic, so the same edits give identical bytes. A conflicted copy that the sync makes is an exact duplicate: delete it.
    2. The logs stay the truth until edits are applied. For copies that differ, keep the one whose hash matches the latest metadata.applied, and delete the other. Follow the before and after hashes back from the kept file to see which edits it holds. Edits that only reached the deleted copy are pending again, and are applied again. When neither copy matches, as when the applying device's log hasn't arrived, both stay, reported as a conflict, until one does.

3.4 Transforms

A transform changes an edition's file: it has a name and a version, and takes the file in and gives a file out. It uses no clock and no randomness, so the same input always gives the same bytes. It changes how the file is put together, not what it says: every content document's text stays as Locators count it (§7), so annotations hold. Applying edits (§3.3) is one; the upgrade to EPUB 3 (§3.5) is another.

  • Its version is an integer of its own. Raise it whenever the same input would give different bytes; a fix that doesn't change the output keeps it. A send transform's version is its release instead, such as kepubify's 5.0.0: it's never logged, only used to cache the output (below). It's neither an app's version nor the format's version (§1.2).

  • Names are lowercase letters, digits, and -, such as epub3.

  • Logged, then applied, like edits. Asking for a transform logs transform.requested {edition, name}. A device that applies edits runs it together with any pending edits, in one rewrite, and lists it with its version in metadata.applied. A device that doesn't apply can still ask.

    • The request names no version. A device runs the version it has. Two devices with different versions applying at once write different files, which the recovery of §3.3 sorts out.
    • Time: a transform that needs one, such as the upgrade's dcterms:modified, takes the request's timestamp, never the time of applying.
    • One that can't run on this file, such as epub3 on an EPUB 3, is still listed in metadata.applied, so it isn't tried again, and the device reports it. When nothing was written, hash is base.
    • One the device doesn't know stays requested, for a device that knows it. It holds up nothing else.
  • On import: the library's transforms setting (§1.2) lists, by hook, the transforms to request for each edition imported, in order:

    "transforms": {
      "import": [{ "name": "epub3" }]
    }
    
    • An entry is {name}. Keep the names you don't know in the setting; never drop them.
    • Each is requested right after edition.added, and applied like any other request by a device that has it.
  • At any other time, a transform can be requested for any edition.

  • send transforms change the copy a device is given, never the library's file. Their output isn't logged: it's derived, and cached by the file's hash and each transform's version.

3.5 Upgrading to EPUB 3

The transform epub3 upgrades an EPUB 2 edition to EPUB 3.

  • Only the package changes. Content documents stay byte for byte as they were, doctype included, and so do their paths, so every Locator and annotation still points where it did (§7). Content documents are never fixed, even for markup epubcheck reports under EPUB 3, such as <center> and <a name>.
  • What it does:
    • The OPF gets version="3.0" and a dcterms:modified.
    • Metadata moves to EPUB 3's forms: opf:role and opf:file-as become refines metas, boeks:series and boeks:series_position become belongs-to-collection, boeks:subtitle becomes title-type, and <meta name="cover"> also becomes the cover-image property.
    • An identifier's opf:scheme goes into its value (urn:isbn:…), or into an identifier-type meta where the value wouldn't read back the same, or it's the unique identifier. The unique identifier's value never changes, since it keys obfuscated fonts.
    • EPUB 3 has one date, so the publication date is kept (or the first date, without one) and the others, such as a conversion date, are removed.
    • A nav document is made from the NCX, with landmarks from the <guide>, outside the spine. The NCX and the <guide> stay.
    • Manifest properties (scripted, svg, mathml, and remote-resources) are found by reading the content documents, and fonts get EPUB 3's media types.
    • Obfuscated fonts are left as they are.
  • Not upgraded: EPUB 3 files, DRM'd files, files that fail validation, files without an NCX, and fixed-layout files. A file is never upgraded in part.

3.6 Covers

An edition's cover is the image its OPF marks: the manifest item with the cover-image property in EPUB 3, or the item <meta name="cover"> names.

  • A new cover is an edit: a metadata.updated whose changes hold cover, the image's SHA-256 (§5.3). A log line can't hold an image, so the device making the edit first saves it in the book's .boeks/covers/, named by that hash with its type's extension (<hex>.jpg). A cover can be replaced or added, not removed.
  • The image has the old cover's type. Whatever it came as (a pasted PNG, a HEIC photo, a provider's WebP), it's converted as it's saved, to the current cover's media type, or to JPEG when there's no cover. Applying writes it at the old cover's path, which only works for the same type. Image encoders differ, so applying never converts, and the same edit always gives the same bytes.
  • Applying:
    • Over a cover, the cover item's file gets the new image's bytes, at the same path. Content documents still point at it, so a cover page shows the new image, and no text changes.
    • Without one, the image is added as cover.<ext> next to the OPF, or cover-2.<ext> and so on when that's taken, with a manifest item named by <meta name="cover">, and also marked cover-image in EPUB 3. No cover page is added.
  • The image is deleted once a metadata.applied lists its edit and no pending edit names it. A file that can't be written keeps it for good, and the index shows it as the cover (§4).
  • Covers don't fan out. Editions often have different covers, so each keeps its own, and the book's is its primary edition's (§2.2).