Skip to content
Boeks

Appearance

Theme

Logs

The append-only log each device keeps, and the events in it.

5. Logs

5.1 Files

ScopePath
Book<book>/.boeks/<device>.jsonl
Library.boeks/<device>.jsonl at the root
  • One JSON event per line, each ending in \n. A writer that finds its log doesn't end in \n, as after a write cut short, starts with one, so its line isn't joined to the torn one, which readers skip.
  • A device appends only to its own files. The one rewrite is compaction (§5.6), and that's of its own file too. Two exceptions: a device holding another's storage, as the Mac does a Kobo's over USB, writes that device's changes in that device's log, as it, under its sequence and stamped after what the library's logs have seen (§5.2, §17.3); and peer sync (§8.2) appends other devices' lines to their logs, as written, past the last sequence there.
  • To build current state, read every device's file and merge them (§5.4). Every .jsonl in a .boeks/ is read as a log, its lines naming their device, so a stray log under another name counts too, until its device folds it in (§8.1).

5.2 Envelope

{"timestamp":1760200000123,"offset":120,"device":"iphone-7k3","sequence":42,"type":"progress.updated","data":{…}}
  • timestamp is in milliseconds since the Unix epoch, chosen as max(clock now, my last timestamp + 1, highest timestamp read in other devices' logs + 1). So a device never writes a timestamp below anything it has already seen.
    • Example: a Kobo whose clock reset to 2024 has synced and read the iPhone's Monday event. Its next event is still stamped after Monday, so last-writer-wins correctly picks the Kobo.
    • A timestamp more than a day ahead of the device's own clock is taken as a mistake, such as a hand edit, and doesn't raise the floor, so one line can't push every device into the future.
  • offset is the UTC offset where the event's moment happened, in minutes east of UTC (120 in Amsterdam in summer, -240 in New York), as the writing device's time zone had it then: from −720 to 840. The moment is at when the event has one, a session's started for session.created, and otherwise timestamp. It's only for showing times as they were where reading happened (§5.3), and never orders or merges anything. Every line has one, as it has a timestamp: a device whose system can't tell where it is writes 0, UTC, as its clock then is. A line without a whole number in that range isn't an envelope.
  • device is the writing device's ID (§5.7), the same as the file's name.
  • sequence counts up per device, from 1. It detects duplicate lines, and it makes "have I read all of device X's log?" one number to compare. (device, sequence) identifies an event in the library. Two rewrites change what a sequence holds: compaction puts a session in the place of a line it drops, with that line's timestamp and sequence (§5.6), and a conflict copy's lines are folded back in under new sequences (§8.1). Sequences can have gaps, as where compaction dropped lines. Two lines with the same (device, sequence) are one event: the fold keeps the one ordered first, by timestamp, then by the line's bytes. Every device keeps its highest sequence outside the library too, as it may open the library before its own logs can be read, as an iPhone can while iCloud hasn't brought them down. One that knows none, as when what it kept was lost, finds it in its own logs across the library before it writes.
  • type is the event type (§5.3), and data its fields.
  • Order events by (timestamp, device, sequence).
    • A change ordered before what it changes was made, as a note on a highlight from a Kobo whose clock is days ahead, is kept until it's made, then applied in order: whoever changed it had it. This holds for annotation.updated, the read-through events after readthrough.started, and an edition's metadata.updated, transform.requested, and metadata.applied. A change to a deleted annotation stays dropped.
    • An annotation's deletion wins over its creation in whatever order they come, and only annotation.restored, ordered after it, undoes it: an annotation.created with a deleted ID, ordered after the deletion as from a clock behind, is passed over. An edition is the exception, since a removed edition put back is added again under its ID (§2.5): an edition.removed ordered before the edition's edition.added changes nothing.
  • The first line of a log is book.imported or log.started, which carries the book ID.
  • Schemas: each event type has a strict JSON Schema (draft 2020-12, additionalProperties: false), for what this version writes. Data of your own goes under keys prefixed with x-, which every object takes and readers keep or ignore.
  • Integers, timestamp and sequence among them, stay at most 2^53 − 1, the largest that JavaScript and LuaJIT (KOReader's) hold exactly.
  • Validation: validate every event before writing it, against this version's schemas.
  • Read leniently, so a newer version's lines don't stop an older reader:
    • Ignore fields you don't know, in the envelope and in data.
    • Read a value you don't know in an optional field as if the field were absent: a newer highlight color as the default one, in annotation.updated as no change.
    • Keep an event you can't read and pass over it: one of a type you don't know, or whose data lacks a required field, has a value of the wrong type, or has a value you don't know in a required field. The schemas are for writing, so don't validate what you read against them: a newer line with fields of its own fails them and is still read. Skip a line that isn't an envelope at all, and keep reading.
    • Never change a line you read: rewriting a log (§5.6) or passing a line on to another device copies it as it was.

5.3 Event catalog

Event names are object.verb, in the past tense. Changes to state take fixed verbs, and merge by them (§5.4):

  • updated for a single value
  • created, updated, and deleted for items with an ID
  • added and removed for set membership

Other events take a verb of their own: a moment in a book's history, such as readthrough.finished, or a change with a merge of its own, such as readthrough.reclassified, whose order matters, or book.merged.

A file is a path in the book's folder, with / between its parts: at most two parts (an edition's folder and its file, or an audiobook's folder, ending in /, §9), with none empty, hidden, . or .., and no \. An event naming another is passed over. Fields marked ? are optional, and left out rather than null when they have no value; only rating.updated takes null, to clear the rating. Hashes are SHA-256, as 64 lowercase hex digits. IDs are ULIDs, written in lowercase and read in either case. Times are milliseconds since the Unix epoch, like timestamp, and happened where the event's offset says (§5.2); at is for when something happened earlier than its event was written, as in an import. Stated dates, such as a corrected read date, are times on the calendar and the clock instead, with no zone. A name, a collection's or a device's, has 1 to 200 characters, not counting whitespace at its ends (which the schemas can't check), and a review or note at most 100,000, counting Unicode scalar values as §7 does.

Book (book logs):

EventDataMeaning
book.imported{id}The book was imported, with its ID. The first line of the importing device's log.
log.started{book?}The first line of any other log: the book's ID in a book log, no book in a library log.
book.updated{primaryEdition?}Sets the primary edition.

Edition (book logs, §2):

EventDataMeaning
edition.added{id, file, hash, imported?}An edition was added, by the path of its file in the book folder (Ebook/Dune - Frank Herbert.epub, §1.3) and the SHA-256 of the file, in hex. It's logged once for an edition, or again to put it back once it was removed (§2.5); another for an edition the book has changes nothing. imported is the IDs it has in other apps, by the app: {"calibre": "<uuid>"} for Calibre's ID for the book, which the copy Calibre put on a Kobo is known by (§17), or {"audiobookshelf": "<item id>"} for an Audiobookshelf item's.
edition.updated{id, file?, hash?, imported?}Changes the edition: file, its new path, when its file or folder was renamed; hash, when its file was replaced by another (§2.3); and imported, IDs in other apps added to those it has, as when its reading was brought in from one (§2.3).
edition.removed{id}The edition was removed from its book, and its folder moved to the trash (§2.5).
edition.moved{id, to}The edition moved to book to. Where to's logs are read with this one, as after a merge (§2.4), it's still an edition here.

Reading (book logs):

EventDataMeaning
progress.updated{edition, locator, at?}Where reading is: a Locator (§7) into that edition. at is when it was read there, when that's earlier than the event, as for a place brought in from another reader.
session.created{id, edition, started, ended, start?, end?, total?}A reading session already measured, from started to ended, with where reading was at each end: start and end are {totalProgression?, position?}: the Locator's totalProgression, through the book from 0 to 1, and its position, the page (§7). total (true) is reading over several sittings, known only as a total, as another reader counted it: it ends when the book was last read, and starts that much earlier, not when any sitting did. It hides no progress inside it, and the timeline shows it on the day it ended, without a time or device.
readthrough.started{id, edition?, at?, undated?}Opens a read-through. undated (true) is for a start nobody knows, as for a book imported as read or being read from another app: at is then only the earliest reading known of, to order it by, and the timeline shows no start.
readthrough.finished, readthrough.stopped, readthrough.paused{id, at?}Finishes, stops, or pauses it. A read-through can be paused more than once, and stays open.
readthrough.reclassified{id, as}Corrects how a read-through ended, keeping the date: its finish or stop becomes finished, stopped, or paused (as). A pause leaves it open again. It records no new moment.
readthrough.updated{id, started?, finished?, stopped?}Corrects a read-through's dates to a year, month, day, or minute: 2019, 2019-05, 2019-05-12, or 2019-05-12T21:40. A date left out stays as it is.
readthrough.created{id, started?, finished?, stopped?}A read-through someone states rather than one recorded, as for a book read before it was imported, added in Book.md (§6.2). It has stated dates, as readthrough.updated takes them, and has ended: finished or stopped, not both. Without started, nobody knows when it started. It records no moment, and never takes in the open read-through.
readthrough.deleted{id}Removes a read-through, as one taken out of Book.md. A tombstone.

Annotations (book logs, §7):

EventDataMeaning
annotation.created{id, edition, motivation, target, color?, body?, at?}A highlight or bookmark. motivation is highlighting or bookmarking, and target a Locator. A highlight has a color (yellow, the default, green, blue, pink, or purple) and an optional note, body, in Markdown. A bookmark marks a point, so its Locator's highlight is empty. at is when it was made, when that's earlier than the event, as for one brought in from another reader.
annotation.updated{id, color?, body?, target?}Changes the fields given. An empty body removes the note.
annotation.deleted{id}A tombstone.
annotation.restored{id}Undoes the annotation's deletion, as when its Edition.md is put back (§6.1): it's back as it was when deleted. The later of a deletion and a restore wins.

User fields (book logs):

EventDataMeaning
status.updated{status}unread (new books start here), reading, paused, read, or stopped (given up on).
rating.updated{rating}Whole stars, 1–5; null, or no rating, clears it.
review.updated{text}The review, in Markdown (CommonMark, plus strikethrough).
collection.added, collection.removed{name}This book was added to, or removed from, a collection (§5.4).

Metadata (book logs, §3):

EventDataMeaning
metadata.updated{edition, changes}Edits to an edition's metadata, per field. changes takes title, titleFileAs, subtitle, creators, series, identifiers, language, publisher, published, description, genres (the OPF's dc:subject), and cover, a new cover's image, by its hash (§3.6). A field left out is unchanged, and an empty string clears a text field. A list (creators, series, genres) and identifiers, a map of one value per scheme, replace the edition's whole. language and published are as the OPF has them: a BCP 47 tag, and a date such as 1965 or 1965-08-01.
transform.requested{edition, name}Asks for a transform of an edition's file (§3.4).
metadata.applied{edition, base, hash, edits, transforms?}Edits and transforms were written into the file: the SHA-256 before (base) and after (hash), the metadata.updated events written (edits, as {device, sequence}), and the transforms run, in order, each as {request, name, version} with the transform's own version.

Files (book logs, §6):

EventDataMeaning
file.applied{file, hash, values, edition?}Logged state was written into a file in the book folder, such as Book.md: its name, its SHA-256 as written, and the values written, to tell edits made to the file from changes in the logs. edition is set for an edition's Edition.md (§6.3), and keeps it tracked when its folder is renamed.
file.deleted{file, hash, edition?}A file of §6 was deleted, clearing what it held (§6.1): its name, the SHA-256 of the version deleted, and for an Edition.md, its edition. That version put back brings it back.

Library (library logs):

EventDataMeaning
device.registered{name, platform, model?, key?}Names a device (§5.7).
device.forgotten{id}A device the reader no longer has, such as a Kobo sold: its registration goes, with its key, so it isn't listed and can't sync directly, until it registers again. Its logs stay, as its reading happened.
book.deleted{id}A book was deleted from the library, and its folder moved to the trash (§2.5).
book.restored{id}A deleted book's files are back, as when it was put back from the trash: it's no longer deleted (§2.5).
book.merged{from, into}Book from was merged into book into (§2.1, §2.4).

Progress is written when reading pauses, or every 30 seconds while reading, not on every page turn. Opening a book isn't reading it: nothing is written until the reader moves from where it opened, by a page turn, a jump, or playing, so a book opened and closed again keeps its place and its last read.

Sessions aren't recorded as they start and end. They're found in each device's progress.updated, each at its at, else its timestamp: a run of one device's progress in one edition, none more than 5 minutes apart (exactly 5 minutes still is), lasting at least a minute from its first to its last. Progress of a device and edition within a session.created of its own that isn't a total is part of that session, not a session of its own. session.created holds the ones that aren't in the progress any more: compaction writes the sessions it finds as it drops their progress (§5.6), and an import that only knows how long reading took writes one too, as a total.

A session that starts behind where reading last was before it, and reads past that place, starts there: the device was catching up to where another had been, without having synced. Where reading last was is the latest progress before the session started, or the latest end of another session by then, on any device, in the session's edition, compared by totalProgression: another edition's places, an audiobook's or a translation's, run differently through the book.

Time spent reading a book is the time its sessions cover, a stretch counted once where sessions overlap: reading along to the audiobook, or the book open in two editions at once, is one stretch of reading. Sessions in different books both count.

Reading status:

  • A book has exactly one reading status.
  • Reading marks a book as being read. The first progress a device records in an unread book comes with status.updated reading and the readthrough.started it opens.
  • Status drives read-throughs:
    • Moving to reading opens a read-through.
    • Moving to paused pauses it, to read finishes it, and to stopped stops it.
    • Pausing, finishing, or stopping with no read-through open opens one first, at the same moment.
    • Moving straight between paused, read, and stopped with no read-through open corrects the latest one with readthrough.reclassified instead. Going through reading or unread starts a new one.
    • Setting the status a book already has writes no read-through events.
  • Rereads are new read-throughs.
  • One read-through is open at a time. Two devices that start reading before seeing each other's start open two: in the fold, one that starts while another is open takes the other in, keeping the earlier start and its pauses, and events for either apply to it.
  • A later ending replaces an earlier one: a read-through finished on one device and stopped on another ends as the later says.

Read dates can be corrected, as for a book read before it was imported. A corrected date is a calendar date of that precision, or a time on the clock, not a moment, and reads the same in any time zone: it floats, as iCalendar calls it. readthrough.updated is last-writer-wins per date, and only changes an end the read-through already has: it never finishes or stops one. A read-through can also be stated whole, with readthrough.created, and removed, with readthrough.deleted; a deletion wins over every other event for the read-through, in whatever order they come.

A corrected date only changes what's shown. Sessions stay where they were recorded, so the timeline can show one after a read-through's corrected end: the reader's statement next to the record, and nothing is moved to hide it.

Showing a time: a moment is shown as the time where it happened, its event's offset (§5.2) applied to it, so a trip shows as it was lived, wherever it's looked at. Devices whose time zone is set by hand, and so rarely follows a trip (nickel and koreader, §5.7), are the exception: their events take the offset of the latest event before them from an iPhone or iPad (ios), which sets its zone by itself and goes along; without one, their own. Every device has the same logs, so every device shows the same time, and writes the same Book.md. A stated date is shown as it was stated.

5.4 Merging

  • updated events: last-writer-wins per field. The one exception is edition.updated's imported, which adds to the IDs the edition has rather than replacing them.
  • Keyed items (annotations, read-throughs): merged by ID. A tombstone wins over earlier updates.
  • Set membership (collections): last-writer-wins per (collection, book) pair.
  • Collections are their books' names for them, as tags are: a collection exists while a book is in it, and only then. There's nothing else to one, so Book.md holds all of it (§6.2).
    • Names match case-insensitively, after NFC, with whitespace trimmed from their ends (§1.3 says how names compare): "Sci-fi" and "sci-fi" are one collection.
    • Its spelling is that of the newest add among the books in it. Adding a book again under another case changes how the collection is written, for every book in it.
    • Renaming a collection is taking each of its books out under the old name and adding it under the new one; renaming onto a name that's a collection already merges the two. Deleting one is taking each book out. A device that was offline and adds a book under the old name afterwards brings the old one back, with that book: as a tag would, it's there to see, and to rename again.

5.5 Continuing from another device

  • Follow the newest progress. When a book opens and the newest progress in that edition is from another device, open the book there.
  • Follow each progress once, so going back to an earlier page doesn't pull the reader forward again on the next open.
  • Don't follow progress that's where this device already is: in the same resource (its href, without a fragment), and less than 1% of it away (progression within 0.01).
  • Going back to where this device was, after following another's progress, saves that as the newest progress.

5.6 Compaction

  • A device may rewrite its own log, and only its own, dropping every event that a newer event of its own makes redundant:
    • older values of the same field
    • all but the latest file.applied per file
  • Keep history: read-throughs and editions; and deleted annotations' events, which annotation.restored brings back.
  • Write the sessions found in the progress it drops as session.created (§5.3), each in the place of the last line it drops from that session, with that line's timestamp and sequence. A device that already has the line by its sequence (§8.2) has the progress the session was found in, so its state is the same.
  • Event types: keep a newer format's.
  • The result folds to the same state and sessions, whatever other devices have written. What it drops is gone for what reads it later: a session caught up (§5.3) by a place it dropped, and the hashes of older versions of a §6 file, which a late copy of one is then read against as an edit.
  • Write it atomically: a temporary file, then a rename over the log. No other device writes the file, so this is safe over any sync layer. Don't compact where a rename over a file isn't atomic, as on FAT.
  • When: at about 1,000 lines or 256 KB.

5.7 Devices

  • ID format: <slug>-<suffix>, using only [a-z0-9-], such as iphone-7k3 or kobo-libra-x7m.
    • The slug comes from the device's model (MacBook Pro, iPhone, Kobo Libra Colour), not the name the user gave it.
    • The suffix is 3 random characters of RFC 4648's base32 alphabet (a–z, 2–7). A reset device, or the one that replaces it, never reuses the old ID.
  • Lifetime: generated on first run and stored on the device, outside the library. Never derived from hardware IDs. A backup or migration can copy it to another device, and two devices writing one log lose each other's lines, so a device also keeps, beside its ID, something that tells it from other devices, such as the system's ID for the app on this device. When that no longer matches, the ID was copied, and it makes a new one.
  • A reset or reinstall makes a new device ID. Old logs stay valid.
  • device.registered maps an ID to a display name, platform, and model, so a device can be renamed without touching files. key is the device's public key for syncing directly: Ed25519, its 32 bytes in unpadded base64url, as a JWK's x holds one (RFC 8037).
    • The name is the kind of device: MacBook, iMac, Mac (a mini, Studio, or Pro), iPhone, iPad, Kobo (in Nickel or KOReader alike), or Kindle. Not its model, which goes in model, nor the name the user gave it. Register again when either changes.
    • The platform is what wrote it: macos, ios, nickel (Boeks for Kobo), koreader, or cli; the first four are shown as macOS, iOS, Nickel, and KOReader. The model is the device's model as people know it, such as MacBook Pro or Kobo Clara BW, not a hardware identifier.
    • Register once the library can be read. A device that opens it before it can see into it, as before access to its folder or iCloud's files have come through, finding no book where it knew of some, registers at the first refresh that does: nothing is written into a library it can't read.
    • Show the name alone. Only when another device has that name too, show what tells them apart: the model (Kobo Clara BW), else the platform (Kobo (KOReader)), else the ID's suffix (iPhone (k3x)).
  • Forgetting a device (device.forgotten) is for one the reader no longer has. It leaves the lists, and its key stops letting it in (§8.2). A device in use registers again as it next writes, and is back.
  • A device was last active at the newest line in its logs, by its own timestamp, leaving out log.started and device.registered, which it writes without being used.