Metadata
How Boeks reads a book's details from its EPUB, and writes edits back into it.
3. Metadata
3.1 Reading
Metadata is read from the OPF of EPUB 2 and EPUB 3 files. Only the title is required.
- Title and subtitle come from EPUB 3's
title-type(main,subtitle). Without one, the first title is the title, and the subtitle comes fromboeks:subtitle. Other title types, such ascollectionandshort, are kept. - Creators have MARC relator roles (
aut,edt,trl,ill,nrt), and afile-asfor sorting. The main title can have afile-astoo. - Series come from EPUB 3's
belongs-to-collectionwithcollection-typeseriesand agroup-position, which may be a decimal. A book can be in several series. EPUB 2 has no series, so there they come fromboeks:seriesandboeks:series_position. - Identifiers are free-form
scheme:value, with the scheme names Calibre uses:isbn,goodreads,hardcover,openlibrary,google,url,uuid, and others. Amazon's IDs areasin, an ebook's and an audiobook's alike, as the ID is called; Calibre'samazon,amazon_<country>, andmobi-asinare read asasin. Boeks addsinventaire, a work's Inventaire URI: its Wikidata item (wd:Q190192), or Inventaire's own entity for one Wikidata hasn't (inv:…). Awikidataitem stored by another tool (Q190192) is read as the same work. - Other apps' fields, such as Calibre's
calibre:seriesand its custom columns, aren't read.
<metadata>
<dc:title id="t1">Dune</dc:title>
<meta refines="#t1" property="title-type">main</meta>
<dc:creator id="c1">Frank Herbert</dc:creator>
<meta refines="#c1" property="role" scheme="marc:relators">aut</meta>
<meta refines="#c1" property="file-as">Herbert, Frank</meta>
<meta property="belongs-to-collection" id="s1">Dune</meta>
<meta refines="#s1" property="collection-type">series</meta>
<meta refines="#s1" property="group-position">1</meta>
</metadata>
3.2 Writing
-
Only the OPF changes, and a new cover's image (§3.6). Unknown elements are kept, and an EPUB 2 file stays EPUB 2.
-
The zip is deterministic, so the same edits give byte-identical files:
mimetypecomes first, then the other entries in the original file's order, then any files a transform or a new cover adds, by name.- Entries that didn't change are copied as they are, compressed bytes and headers included.
- Entries written anew,
mimetypeand the OPF among them, are stored uncompressed, dated 1980-01-01 00:00 (the earliest a zip can hold), with Unix permissions 0644.
Devices running the same Boeks write the same file. Another tool following this writes the same entries, but the zip's headers have details this doesn't fix (such as the version that made it, flags, and extra fields), and how the OPF is serialized isn't specified, so its file may differ in bytes: the recovery below handles that.
-
EPUB 3 gets the standard forms (
title-type,belongs-to-collection). EPUB 2 has no place for some fields, so they're written asboeks:metas:boeks:subtitle, andboeks:seriesandboeks:series_positionfor one series. -
The format's own fields use the
boeks:prefix. -
Other apps' fields are neither written nor removed.
-
Never written:
- user fields, such as rating, status, and collections, which are in
Book.md(§6.2) - DRM'd files
- files that fail validation
The edits to a file that can't be written stay in the logs, and the clients show them over the file's own metadata, which the index keeps as the file has it (§4, §12.2).
- user fields, such as rating, status, and collections, which are in
3.3 Edits are logged, then applied
Edits are written into the EPUB from the logs, like the files of §6.
- Logging: an edit is a
metadata.updatedin the editing device's log, for one field of one edition. - Applying: a device that has the EPUB, downloaded and writable, writes the pending edits into its OPF. It then logs
metadata.appliedwith the file's hash before and after, and the edits it wrote. - Not every device applies. A device may leave applying to others; it still logs edits.
- Two devices applying at once is safe:
- The output is deterministic, so the same edits give identical bytes. A conflicted copy that the sync makes is an exact duplicate: delete it.
- The logs stay the truth until edits are applied. For copies that differ, keep the one whose hash matches the latest
metadata.applied, and delete the other. Follow the before and after hashes back from the kept file to see which edits it holds. Edits that only reached the deleted copy are pending again, and are applied again. When neither copy matches, as when the applying device's log hasn't arrived, both stay, reported as a conflict, until one does.
3.4 Transforms
A transform changes an edition's file: it has a name and a version, and takes the file in and gives a file out. It uses no clock and no randomness, so the same input always gives the same bytes. It changes how the file is put together, not what it says: every content document's text stays as Locators count it (§7), so annotations hold. Applying edits (§3.3) is one; the upgrade to EPUB 3 (§3.5) is another.
-
Its version is an integer of its own. Raise it whenever the same input would give different bytes; a fix that doesn't change the output keeps it. A
sendtransform's version is its release instead, such as kepubify's5.0.0: it's never logged, only used to cache the output (below). It's neither an app's version nor the format'sversion(§1.2). -
Names are lowercase letters, digits, and
-, such asepub3. -
Logged, then applied, like edits. Asking for a transform logs
transform.requested{edition, name}. A device that applies edits runs it together with any pending edits, in one rewrite, and lists it with its version inmetadata.applied. A device that doesn't apply can still ask.- The request names no version. A device runs the version it has. Two devices with different versions applying at once write different files, which the recovery of §3.3 sorts out.
- Time: a transform that needs one, such as the upgrade's
dcterms:modified, takes the request's timestamp, never the time of applying. - One that can't run on this file, such as
epub3on an EPUB 3, is still listed inmetadata.applied, so it isn't tried again, and the device reports it. When nothing was written,hashisbase. - One the device doesn't know stays requested, for a device that knows it. It holds up nothing else.
-
On import: the library's
transformssetting (§1.2) lists, by hook, the transforms to request for each edition imported, in order:"transforms": { "import": [{ "name": "epub3" }] }- An entry is
{name}. Keep the names you don't know in the setting; never drop them. - Each is requested right after
edition.added, and applied like any other request by a device that has it.
- An entry is
-
At any other time, a transform can be requested for any edition.
-
sendtransforms change the copy a device is given, never the library's file. Their output isn't logged: it's derived, and cached by the file's hash and each transform's version.
3.5 Upgrading to EPUB 3
The transform epub3 upgrades an EPUB 2 edition to EPUB 3.
- Only the package changes. Content documents stay byte for byte as they were, doctype included, and so do their paths, so every Locator and annotation still points where it did (§7). Content documents are never fixed, even for markup epubcheck reports under EPUB 3, such as
<center>and<a name>. - What it does:
- The OPF gets
version="3.0"and adcterms:modified. - Metadata moves to EPUB 3's forms:
opf:roleandopf:file-asbecomerefinesmetas,boeks:seriesandboeks:series_positionbecomebelongs-to-collection,boeks:subtitlebecomestitle-type, and<meta name="cover">also becomes thecover-imageproperty. - An identifier's
opf:schemegoes into its value (urn:isbn:…), or into anidentifier-typemeta where the value wouldn't read back the same, or it's the unique identifier. The unique identifier's value never changes, since it keys obfuscated fonts. - EPUB 3 has one date, so the publication date is kept (or the first date, without one) and the others, such as a conversion date, are removed.
- A nav document is made from the NCX, with landmarks from the
<guide>, outside the spine. The NCX and the<guide>stay. - Manifest properties (
scripted,svg,mathml, andremote-resources) are found by reading the content documents, and fonts get EPUB 3's media types. - Obfuscated fonts are left as they are.
- The OPF gets
- Not upgraded: EPUB 3 files, DRM'd files, files that fail validation, files without an NCX, and fixed-layout files. A file is never upgraded in part.
3.6 Covers
An edition's cover is the image its OPF marks: the manifest item with the cover-image property in EPUB 3, or the item <meta name="cover"> names.
- A new cover is an edit: a
metadata.updatedwhose changes holdcover, the image's SHA-256 (§5.3). A log line can't hold an image, so the device making the edit first saves it in the book's.boeks/covers/, named by that hash with its type's extension (<hex>.jpg). A cover can be replaced or added, not removed. - The image has the old cover's type. Whatever it came as (a pasted PNG, a HEIC photo, a provider's WebP), it's converted as it's saved, to the current cover's media type, or to JPEG when there's no cover. Applying writes it at the old cover's path, which only works for the same type. Image encoders differ, so applying never converts, and the same edit always gives the same bytes.
- Applying:
- Over a cover, the cover item's file gets the new image's bytes, at the same path. Content documents still point at it, so a cover page shows the new image, and no text changes.
- Without one, the image is added as
cover.<ext>next to the OPF, orcover-2.<ext>and so on when that's taken, with a manifest item named by<meta name="cover">, and also markedcover-imagein EPUB 3. No cover page is added.
- The image is deleted once a
metadata.appliedlists its edit and no pending edit names it. A file that can't be written keeps it for good, and the index shows it as the cover (§4). - Covers don't fan out. Editions often have different covers, so each keeps its own, and the book's is its primary edition's (§2.2).