Logs
The append-only log each device keeps, and the events in it.
5. Logs
5.1 Files
| Scope | Path |
|---|---|
| Book | <book>/.boeks/<device>.jsonl |
| Library | .boeks/<device>.jsonl at the root |
- One JSON event per line, each ending in
\n. A writer that finds its log doesn't end in\n, as after a write cut short, starts with one, so its line isn't joined to the torn one, which readers skip. - A device appends only to its own files. The one rewrite is compaction (§5.6), and that's of its own file too. Two exceptions: a device holding another's storage, as the Mac does a Kobo's over USB, writes that device's changes in that device's log, as it, under its sequence and stamped after what the library's logs have seen (§5.2, §17.3); and peer sync (§8.2) appends other devices' lines to their logs, as written, past the last sequence there.
- To build current state, read every device's file and merge them (§5.4). Every
.jsonlin a.boeks/is read as a log, its lines naming their device, so a stray log under another name counts too, until its device folds it in (§8.1).
5.2 Envelope
{"timestamp":1760200000123,"offset":120,"device":"iphone-7k3","sequence":42,"type":"progress.updated","data":{…}}
timestampis in milliseconds since the Unix epoch, chosen asmax(clock now, my last timestamp + 1, highest timestamp read in other devices' logs + 1). So a device never writes a timestamp below anything it has already seen.- Example: a Kobo whose clock reset to 2024 has synced and read the iPhone's Monday event. Its next event is still stamped after Monday, so last-writer-wins correctly picks the Kobo.
- A timestamp more than a day ahead of the device's own clock is taken as a mistake, such as a hand edit, and doesn't raise the floor, so one line can't push every device into the future.
offsetis the UTC offset where the event's moment happened, in minutes east of UTC (120in Amsterdam in summer,-240in New York), as the writing device's time zone had it then: from −720 to 840. The moment isatwhen the event has one, a session'sstartedforsession.created, and otherwisetimestamp. It's only for showing times as they were where reading happened (§5.3), and never orders or merges anything. Every line has one, as it has atimestamp: a device whose system can't tell where it is writes0, UTC, as its clock then is. A line without a whole number in that range isn't an envelope.deviceis the writing device's ID (§5.7), the same as the file's name.sequencecounts up per device, from 1. It detects duplicate lines, and it makes "have I read all of device X's log?" one number to compare.(device, sequence)identifies an event in the library. Two rewrites change what a sequence holds: compaction puts a session in the place of a line it drops, with that line's timestamp and sequence (§5.6), and a conflict copy's lines are folded back in under new sequences (§8.1). Sequences can have gaps, as where compaction dropped lines. Two lines with the same(device, sequence)are one event: the fold keeps the one ordered first, by timestamp, then by the line's bytes. Every device keeps its highest sequence outside the library too, as it may open the library before its own logs can be read, as an iPhone can while iCloud hasn't brought them down. One that knows none, as when what it kept was lost, finds it in its own logs across the library before it writes.typeis the event type (§5.3), anddataits fields.- Order events by
(timestamp, device, sequence).- A change ordered before what it changes was made, as a note on a highlight from a Kobo whose clock is days ahead, is kept until it's made, then applied in order: whoever changed it had it. This holds for
annotation.updated, the read-through events afterreadthrough.started, and an edition'smetadata.updated,transform.requested, andmetadata.applied. A change to a deleted annotation stays dropped. - An annotation's deletion wins over its creation in whatever order they come, and only
annotation.restored, ordered after it, undoes it: anannotation.createdwith a deleted ID, ordered after the deletion as from a clock behind, is passed over. An edition is the exception, since a removed edition put back is added again under its ID (§2.5): anedition.removedordered before the edition'sedition.addedchanges nothing.
- A change ordered before what it changes was made, as a note on a highlight from a Kobo whose clock is days ahead, is kept until it's made, then applied in order: whoever changed it had it. This holds for
- The first line of a log is
book.importedorlog.started, which carries the book ID. - Schemas: each event type has a strict JSON Schema (draft 2020-12,
additionalProperties: false), for what this version writes. Data of your own goes under keys prefixed withx-, which every object takes and readers keep or ignore. - Integers,
timestampandsequenceamong them, stay at most 2^53 − 1, the largest that JavaScript and LuaJIT (KOReader's) hold exactly. - Validation: validate every event before writing it, against this version's schemas.
- Read leniently, so a newer version's lines don't stop an older reader:
- Ignore fields you don't know, in the envelope and in
data. - Read a value you don't know in an optional field as if the field were absent: a newer highlight color as the default one, in
annotation.updatedas no change. - Keep an event you can't read and pass over it: one of a type you don't know, or whose data lacks a required field, has a value of the wrong type, or has a value you don't know in a required field. The schemas are for writing, so don't validate what you read against them: a newer line with fields of its own fails them and is still read. Skip a line that isn't an envelope at all, and keep reading.
- Never change a line you read: rewriting a log (§5.6) or passing a line on to another device copies it as it was.
- Ignore fields you don't know, in the envelope and in
5.3 Event catalog
Event names are object.verb, in the past tense. Changes to state take fixed verbs, and merge by them (§5.4):
updatedfor a single valuecreated,updated, anddeletedfor items with an IDaddedandremovedfor set membership
Other events take a verb of their own: a moment in a book's history, such as readthrough.finished, or a change with a merge of its own, such as readthrough.reclassified, whose order matters, or book.merged.
A file is a path in the book's folder, with / between its parts: at most two parts (an edition's folder and its file, or an audiobook's folder, ending in /, §9), with none empty, hidden, . or .., and no \. An event naming another is passed over. Fields marked ? are optional, and left out rather than null when they have no value; only rating.updated takes null, to clear the rating. Hashes are SHA-256, as 64 lowercase hex digits. IDs are ULIDs, written in lowercase and read in either case. Times are milliseconds since the Unix epoch, like timestamp, and happened where the event's offset says (§5.2); at is for when something happened earlier than its event was written, as in an import. Stated dates, such as a corrected read date, are times on the calendar and the clock instead, with no zone. A name, a collection's or a device's, has 1 to 200 characters, not counting whitespace at its ends (which the schemas can't check), and a review or note at most 100,000, counting Unicode scalar values as §7 does.
Book (book logs):
| Event | Data | Meaning |
|---|---|---|
book.imported | {id} | The book was imported, with its ID. The first line of the importing device's log. |
log.started | {book?} | The first line of any other log: the book's ID in a book log, no book in a library log. |
book.updated | {primaryEdition?} | Sets the primary edition. |
Edition (book logs, §2):
| Event | Data | Meaning |
|---|---|---|
edition.added | {id, file, hash, imported?} | An edition was added, by the path of its file in the book folder (Ebook/Dune - Frank Herbert.epub, §1.3) and the SHA-256 of the file, in hex. It's logged once for an edition, or again to put it back once it was removed (§2.5); another for an edition the book has changes nothing. imported is the IDs it has in other apps, by the app: {"calibre": "<uuid>"} for Calibre's ID for the book, which the copy Calibre put on a Kobo is known by (§17), or {"audiobookshelf": "<item id>"} for an Audiobookshelf item's. |
edition.updated | {id, file?, hash?, imported?} | Changes the edition: file, its new path, when its file or folder was renamed; hash, when its file was replaced by another (§2.3); and imported, IDs in other apps added to those it has, as when its reading was brought in from one (§2.3). |
edition.removed | {id} | The edition was removed from its book, and its folder moved to the trash (§2.5). |
edition.moved | {id, to} | The edition moved to book to. Where to's logs are read with this one, as after a merge (§2.4), it's still an edition here. |
Reading (book logs):
| Event | Data | Meaning |
|---|---|---|
progress.updated | {edition, locator, at?} | Where reading is: a Locator (§7) into that edition. at is when it was read there, when that's earlier than the event, as for a place brought in from another reader. |
session.created | {id, edition, started, ended, start?, end?, total?} | A reading session already measured, from started to ended, with where reading was at each end: start and end are {totalProgression?, position?}: the Locator's totalProgression, through the book from 0 to 1, and its position, the page (§7). total (true) is reading over several sittings, known only as a total, as another reader counted it: it ends when the book was last read, and starts that much earlier, not when any sitting did. It hides no progress inside it, and the timeline shows it on the day it ended, without a time or device. |
readthrough.started | {id, edition?, at?, undated?} | Opens a read-through. undated (true) is for a start nobody knows, as for a book imported as read or being read from another app: at is then only the earliest reading known of, to order it by, and the timeline shows no start. |
readthrough.finished, readthrough.stopped, readthrough.paused | {id, at?} | Finishes, stops, or pauses it. A read-through can be paused more than once, and stays open. |
readthrough.reclassified | {id, as} | Corrects how a read-through ended, keeping the date: its finish or stop becomes finished, stopped, or paused (as). A pause leaves it open again. It records no new moment. |
readthrough.updated | {id, started?, finished?, stopped?} | Corrects a read-through's dates to a year, month, day, or minute: 2019, 2019-05, 2019-05-12, or 2019-05-12T21:40. A date left out stays as it is. |
readthrough.created | {id, started?, finished?, stopped?} | A read-through someone states rather than one recorded, as for a book read before it was imported, added in Book.md (§6.2). It has stated dates, as readthrough.updated takes them, and has ended: finished or stopped, not both. Without started, nobody knows when it started. It records no moment, and never takes in the open read-through. |
readthrough.deleted | {id} | Removes a read-through, as one taken out of Book.md. A tombstone. |
Annotations (book logs, §7):
| Event | Data | Meaning |
|---|---|---|
annotation.created | {id, edition, motivation, target, color?, body?, at?} | A highlight or bookmark. motivation is highlighting or bookmarking, and target a Locator. A highlight has a color (yellow, the default, green, blue, pink, or purple) and an optional note, body, in Markdown. A bookmark marks a point, so its Locator's highlight is empty. at is when it was made, when that's earlier than the event, as for one brought in from another reader. |
annotation.updated | {id, color?, body?, target?} | Changes the fields given. An empty body removes the note. |
annotation.deleted | {id} | A tombstone. |
annotation.restored | {id} | Undoes the annotation's deletion, as when its Edition.md is put back (§6.1): it's back as it was when deleted. The later of a deletion and a restore wins. |
User fields (book logs):
| Event | Data | Meaning |
|---|---|---|
status.updated | {status} | unread (new books start here), reading, paused, read, or stopped (given up on). |
rating.updated | {rating} | Whole stars, 1–5; null, or no rating, clears it. |
review.updated | {text} | The review, in Markdown (CommonMark, plus strikethrough). |
collection.added, collection.removed | {name} | This book was added to, or removed from, a collection (§5.4). |
Metadata (book logs, §3):
| Event | Data | Meaning |
|---|---|---|
metadata.updated | {edition, changes} | Edits to an edition's metadata, per field. changes takes title, titleFileAs, subtitle, creators, series, identifiers, language, publisher, published, description, genres (the OPF's dc:subject), and cover, a new cover's image, by its hash (§3.6). A field left out is unchanged, and an empty string clears a text field. A list (creators, series, genres) and identifiers, a map of one value per scheme, replace the edition's whole. language and published are as the OPF has them: a BCP 47 tag, and a date such as 1965 or 1965-08-01. |
transform.requested | {edition, name} | Asks for a transform of an edition's file (§3.4). |
metadata.applied | {edition, base, hash, edits, transforms?} | Edits and transforms were written into the file: the SHA-256 before (base) and after (hash), the metadata.updated events written (edits, as {device, sequence}), and the transforms run, in order, each as {request, name, version} with the transform's own version. |
Files (book logs, §6):
| Event | Data | Meaning |
|---|---|---|
file.applied | {file, hash, values, edition?} | Logged state was written into a file in the book folder, such as Book.md: its name, its SHA-256 as written, and the values written, to tell edits made to the file from changes in the logs. edition is set for an edition's Edition.md (§6.3), and keeps it tracked when its folder is renamed. |
file.deleted | {file, hash, edition?} | A file of §6 was deleted, clearing what it held (§6.1): its name, the SHA-256 of the version deleted, and for an Edition.md, its edition. That version put back brings it back. |
Library (library logs):
| Event | Data | Meaning |
|---|---|---|
device.registered | {name, platform, model?, key?} | Names a device (§5.7). |
device.forgotten | {id} | A device the reader no longer has, such as a Kobo sold: its registration goes, with its key, so it isn't listed and can't sync directly, until it registers again. Its logs stay, as its reading happened. |
book.deleted | {id} | A book was deleted from the library, and its folder moved to the trash (§2.5). |
book.restored | {id} | A deleted book's files are back, as when it was put back from the trash: it's no longer deleted (§2.5). |
book.merged | {from, into} | Book from was merged into book into (§2.1, §2.4). |
Progress is written when reading pauses, or every 30 seconds while reading, not on every page turn. Opening a book isn't reading it: nothing is written until the reader moves from where it opened, by a page turn, a jump, or playing, so a book opened and closed again keeps its place and its last read.
Sessions aren't recorded as they start and end. They're found in each device's progress.updated, each at its at, else its timestamp: a run of one device's progress in one edition, none more than 5 minutes apart (exactly 5 minutes still is), lasting at least a minute from its first to its last. Progress of a device and edition within a session.created of its own that isn't a total is part of that session, not a session of its own. session.created holds the ones that aren't in the progress any more: compaction writes the sessions it finds as it drops their progress (§5.6), and an import that only knows how long reading took writes one too, as a total.
A session that starts behind where reading last was before it, and reads past that place, starts there: the device was catching up to where another had been, without having synced. Where reading last was is the latest progress before the session started, or the latest end of another session by then, on any device, in the session's edition, compared by totalProgression: another edition's places, an audiobook's or a translation's, run differently through the book.
Time spent reading a book is the time its sessions cover, a stretch counted once where sessions overlap: reading along to the audiobook, or the book open in two editions at once, is one stretch of reading. Sessions in different books both count.
Reading status:
- A book has exactly one reading status.
- Reading marks a book as being read. The first progress a device records in an
unreadbook comes withstatus.updatedreadingand thereadthrough.startedit opens. - Status drives read-throughs:
- Moving to
readingopens a read-through. - Moving to
pausedpauses it, toreadfinishes it, and tostoppedstops it. - Pausing, finishing, or stopping with no read-through open opens one first, at the same moment.
- Moving straight between
paused,read, andstoppedwith no read-through open corrects the latest one withreadthrough.reclassifiedinstead. Going throughreadingorunreadstarts a new one. - Setting the status a book already has writes no read-through events.
- Moving to
- Rereads are new read-throughs.
- One read-through is open at a time. Two devices that start reading before seeing each other's start open two: in the fold, one that starts while another is open takes the other in, keeping the earlier start and its pauses, and events for either apply to it.
- A later ending replaces an earlier one: a read-through finished on one device and stopped on another ends as the later says.
Read dates can be corrected, as for a book read before it was imported. A corrected date is a calendar date of that precision, or a time on the clock, not a moment, and reads the same in any time zone: it floats, as iCalendar calls it. readthrough.updated is last-writer-wins per date, and only changes an end the read-through already has: it never finishes or stops one. A read-through can also be stated whole, with readthrough.created, and removed, with readthrough.deleted; a deletion wins over every other event for the read-through, in whatever order they come.
A corrected date only changes what's shown. Sessions stay where they were recorded, so the timeline can show one after a read-through's corrected end: the reader's statement next to the record, and nothing is moved to hide it.
Showing a time: a moment is shown as the time where it happened, its event's offset (§5.2) applied to it, so a trip shows as it was lived, wherever it's looked at. Devices whose time zone is set by hand, and so rarely follows a trip (nickel and koreader, §5.7), are the exception: their events take the offset of the latest event before them from an iPhone or iPad (ios), which sets its zone by itself and goes along; without one, their own. Every device has the same logs, so every device shows the same time, and writes the same Book.md. A stated date is shown as it was stated.
5.4 Merging
updatedevents: last-writer-wins per field. The one exception isedition.updated'simported, which adds to the IDs the edition has rather than replacing them.- Keyed items (annotations, read-throughs): merged by ID. A tombstone wins over earlier updates.
- Set membership (collections): last-writer-wins per (collection, book) pair.
- Collections are their books' names for them, as tags are: a collection exists while a book is in it, and only then. There's nothing else to one, so
Book.mdholds all of it (§6.2).- Names match case-insensitively, after NFC, with whitespace trimmed from their ends (§1.3 says how names compare): "Sci-fi" and "sci-fi" are one collection.
- Its spelling is that of the newest add among the books in it. Adding a book again under another case changes how the collection is written, for every book in it.
- Renaming a collection is taking each of its books out under the old name and adding it under the new one; renaming onto a name that's a collection already merges the two. Deleting one is taking each book out. A device that was offline and adds a book under the old name afterwards brings the old one back, with that book: as a tag would, it's there to see, and to rename again.
5.5 Continuing from another device
- Follow the newest progress. When a book opens and the newest progress in that edition is from another device, open the book there.
- Follow each progress once, so going back to an earlier page doesn't pull the reader forward again on the next open.
- Don't follow progress that's where this device already is: in the same resource (its
href, without a fragment), and less than 1% of it away (progressionwithin 0.01). - Going back to where this device was, after following another's progress, saves that as the newest progress.
5.6 Compaction
- A device may rewrite its own log, and only its own, dropping every event that a newer event of its own makes redundant:
- older values of the same field
- all but the latest
file.appliedper file
- Keep history: read-throughs and editions; and deleted annotations' events, which
annotation.restoredbrings back. - Write the sessions found in the progress it drops as
session.created(§5.3), each in the place of the last line it drops from that session, with that line's timestamp and sequence. A device that already has the line by its sequence (§8.2) has the progress the session was found in, so its state is the same. - Event types: keep a newer format's.
- The result folds to the same state and sessions, whatever other devices have written. What it drops is gone for what reads it later: a session caught up (§5.3) by a place it dropped, and the hashes of older versions of a §6 file, which a late copy of one is then read against as an edit.
- Write it atomically: a temporary file, then a rename over the log. No other device writes the file, so this is safe over any sync layer. Don't compact where a rename over a file isn't atomic, as on FAT.
- When: at about 1,000 lines or 256 KB.
5.7 Devices
- ID format:
<slug>-<suffix>, using only[a-z0-9-], such asiphone-7k3orkobo-libra-x7m.- The slug comes from the device's model (
MacBook Pro,iPhone,Kobo Libra Colour), not the name the user gave it. - The suffix is 3 random characters of RFC 4648's base32 alphabet (
a–z,2–7). A reset device, or the one that replaces it, never reuses the old ID.
- The slug comes from the device's model (
- Lifetime: generated on first run and stored on the device, outside the library. Never derived from hardware IDs. A backup or migration can copy it to another device, and two devices writing one log lose each other's lines, so a device also keeps, beside its ID, something that tells it from other devices, such as the system's ID for the app on this device. When that no longer matches, the ID was copied, and it makes a new one.
- A reset or reinstall makes a new device ID. Old logs stay valid.
device.registeredmaps an ID to a displayname,platform, andmodel, so a device can be renamed without touching files.keyis the device's public key for syncing directly: Ed25519, its 32 bytes in unpadded base64url, as a JWK'sxholds one (RFC 8037).- The name is the kind of device:
MacBook,iMac,Mac(a mini, Studio, or Pro),iPhone,iPad,Kobo(in Nickel or KOReader alike), orKindle. Not its model, which goes inmodel, nor the name the user gave it. Register again when either changes. - The platform is what wrote it:
macos,ios,nickel(Boeks for Kobo),koreader, orcli; the first four are shown as macOS, iOS, Nickel, and KOReader. The model is the device's model as people know it, such asMacBook ProorKobo Clara BW, not a hardware identifier. - Register once the library can be read. A device that opens it before it can see into it, as before access to its folder or iCloud's files have come through, finding no book where it knew of some, registers at the first refresh that does: nothing is written into a library it can't read.
- Show the name alone. Only when another device has that name too, show what tells them apart: the model (
Kobo Clara BW), else the platform (Kobo (KOReader)), else the ID's suffix (iPhone (k3x)).
- The name is the kind of device:
- Forgetting a device (
device.forgotten) is for one the reader no longer has. It leaves the lists, and its key stops letting it in (§8.2). A device in use registers again as it next writes, and is back. - A device was last active at the newest line in its logs, by its own
timestamp, leaving outlog.startedanddevice.registered, which it writes without being used.