# 5. Logs

> The append-only log each device keeps, and the events in it.

Canonical: https://boeks.app/docs/specs/logs

## 5.1 Files

| Scope | Path |
|---|---|
| Book | `<book>/.boeks/<device>.jsonl` |
| Library | `.boeks/<device>.jsonl` at the root |

- **One JSON event per line,** each ending in `\n`. A writer that finds its log doesn't end in `\n`, as after a write cut short, starts with one, so its line isn't joined to the torn one, which readers skip.
- **A device appends only to its own files.** The one rewrite is compaction (§5.6), and that's of its own file too. Two exceptions: a device holding another's storage, as the Mac does a Kobo's over USB, writes that device's changes in that device's log, as it, under its sequence and stamped after what the library's logs have seen (§5.2, §17.3); and peer sync (§8.2) appends other devices' lines to their logs, as written, past the last sequence there.
- **To build current state,** read every device's file and merge them (§5.4). Every `.jsonl` in a `.boeks/` is read as a log, its lines naming their device, so a stray log under another name counts too, until its device folds it in (§8.1).

## 5.2 Envelope

```json
{"timestamp":1760200000123,"offset":120,"device":"iphone-7k3","sequence":42,"type":"progress.updated","data":{…}}
```

- **`timestamp`** is in milliseconds since the Unix epoch, chosen as
  `max(clock now, my last timestamp + 1, highest timestamp read in other devices' logs + 1)`.
  So a device never writes a timestamp below anything it has already seen.
  - Example: a Kobo whose clock reset to 2024 has synced and read the iPhone's Monday event. Its next event is still stamped after Monday, so last-writer-wins correctly picks the Kobo.
  - A timestamp more than a day ahead of the device's own clock is taken as a mistake, such as a hand edit, and doesn't raise the floor, so one line can't push every device into the future.
- **`offset`** is the UTC offset where the event's moment happened, in minutes east of UTC (`120` in Amsterdam in summer, `-240` in New York), as the writing device's time zone had it then: from −720 to 840. The moment is `at` when the event has one, a session's `started` for `session.created`, and otherwise `timestamp`. It's only for showing times as they were where reading happened (§5.3), and never orders or merges anything. Every line has one, as it has a `timestamp`: a device whose system can't tell where it is writes `0`, UTC, as its clock then is. A line without a whole number in that range isn't an envelope.
- **`device`** is the writing device's ID (§5.7), the same as the file's name.
- **`sequence`** counts up per device, from 1. It detects duplicate lines, and it makes "have I read all of device X's log?" one number to compare. `(device, sequence)` identifies an event in the library. Two rewrites change what a sequence holds: compaction puts a session in the place of a line it drops, with that line's timestamp and sequence (§5.6), and a conflict copy's lines are folded back in under new sequences (§8.1). Sequences can have gaps, as where compaction dropped lines. Two lines with the same `(device, sequence)` are one event: the fold keeps the one ordered first, by timestamp, then by the line's bytes. Every device keeps its highest sequence outside the library too, as it may open the library before its own logs can be read, as an iPhone can while iCloud hasn't brought them down. One that knows none, as when what it kept was lost, finds it in its own logs across the library before it writes.
- **`type`** is the event type (§5.3), and **`data`** its fields.
- **Order** events by `(timestamp, device, sequence)`.
  - A change ordered before what it changes was made, as a note on a highlight from a Kobo whose clock is days ahead, is kept until it's made, then applied in order: whoever changed it had it. This holds for `annotation.updated`, the read-through events after `readthrough.started`, and an edition's `metadata.updated`, `transform.requested`, and `metadata.applied`. A change to a deleted annotation stays dropped.
  - **An annotation's deletion wins** over its creation in whatever order they come, and only `annotation.restored`, ordered after it, undoes it: an `annotation.created` with a deleted ID, ordered after the deletion as from a clock behind, is passed over. An edition is the exception, since a removed edition put back is added again under its ID (§2.5): an `edition.removed` ordered before the edition's `edition.added` changes nothing.
- **The first line** of a log is `book.imported` or `log.started`, which carries the book ID.
- **Schemas:** each event type has a strict JSON Schema (draft 2020-12, `additionalProperties: false`), for what this version writes. Data of your own goes under keys prefixed with `x-`, which every object takes and readers keep or ignore.
- **Integers,** `timestamp` and `sequence` among them, stay at most 2^53 − 1, the largest that JavaScript and LuaJIT (KOReader's) hold exactly.
- **Validation:** validate every event before writing it, against this version's schemas.
- **Read leniently,** so a newer version's lines don't stop an older reader:
  - Ignore fields you don't know, in the envelope and in `data`.
  - Read a value you don't know in an optional field as if the field were absent: a newer highlight color as the default one, in `annotation.updated` as no change.
  - Keep an event you can't read and pass over it: one of a type you don't know, or whose data lacks a required field, has a value of the wrong type, or has a value you don't know in a required field. The schemas are for writing, so don't validate what you read against them: a newer line with fields of its own fails them and is still read. Skip a line that isn't an envelope at all, and keep reading.
  - Never change a line you read: rewriting a log (§5.6) or passing a line on to another device copies it as it was.

## 5.3 Event catalog

Event names are `object.verb`, in the past tense. Changes to state take fixed verbs, and merge by them (§5.4):
- `updated` for a single value
- `created`, `updated`, and `deleted` for items with an ID
- `added` and `removed` for set membership

Other events take a verb of their own: a moment in a book's history, such as `readthrough.finished`, or a change with a merge of its own, such as `readthrough.reclassified`, whose order matters, or `book.merged`.

A `file` is a path in the book's folder, with `/` between its parts: at most two parts (an edition's folder and its file, or an audiobook's folder, ending in `/`, §9), with none empty, hidden, `.` or `..`, and no `\`. An event naming another is passed over. Fields marked `?` are optional, and left out rather than `null` when they have no value; only `rating.updated` takes `null`, to clear the rating. Hashes are SHA-256, as 64 lowercase hex digits. IDs are ULIDs, written in lowercase and read in either case. Times are milliseconds since the Unix epoch, like `timestamp`, and happened where the event's `offset` says (§5.2); `at` is for when something happened earlier than its event was written, as in an import. Stated dates, such as a corrected read date, are times on the calendar and the clock instead, with no zone. A name, a collection's or a device's, has 1 to 200 characters, not counting whitespace at its ends (which the schemas can't check), and a review or note at most 100,000, counting Unicode scalar values as §7 does.

**Book** (book logs):

| Event | Data | Meaning |
|---|---|---|
| `book.imported` | `{id}` | The book was imported, with its ID. The first line of the importing device's log. |
| `log.started` | `{book?}` | The first line of any other log: the book's ID in a book log, no `book` in a library log. |
| `book.updated` | `{primaryEdition?}` | Sets the primary edition. |

**Edition** (book logs, §2):

| Event | Data | Meaning |
|---|---|---|
| `edition.added` | `{id, file, hash, imported?}` | An edition was added, by the path of its file in the book folder (`Ebook/Dune - Frank Herbert.epub`, §1.3) and the SHA-256 of the file, in hex. It's logged once for an edition, or again to put it back once it was removed (§2.5); another for an edition the book has changes nothing. `imported` is the IDs it has in other apps, by the app: `{"calibre": "<uuid>"}` for Calibre's ID for the book, which the copy Calibre put on a Kobo is known by (§17), or `{"audiobookshelf": "<item id>"}` for an Audiobookshelf item's. |
| `edition.updated` | `{id, file?, hash?, imported?}` | Changes the edition: `file`, its new path, when its file or folder was renamed; `hash`, when its file was replaced by another (§2.3); and `imported`, IDs in other apps added to those it has, as when its reading was brought in from one (§2.3). |
| `edition.removed` | `{id}` | The edition was removed from its book, and its folder moved to the trash (§2.5). |
| `edition.moved` | `{id, to}` | The edition moved to book `to`. Where `to`'s logs are read with this one, as after a merge (§2.4), it's still an edition here. |

**Reading** (book logs):

| Event | Data | Meaning |
|---|---|---|
| `progress.updated` | `{edition, locator, at?}` | Where reading is: a Locator (§7) into that edition. `at` is when it was read there, when that's earlier than the event, as for a place brought in from another reader. |
| `session.created` | `{id, edition, started, ended, start?, end?, total?}` | A reading session already measured, from `started` to `ended`, with where reading was at each end: `start` and `end` are `{totalProgression?, position?}`: the Locator's `totalProgression`, through the book from 0 to 1, and its `position`, the page (§7). `total` (`true`) is reading over several sittings, known only as a total, as another reader counted it: it ends when the book was last read, and starts that much earlier, not when any sitting did. It hides no progress inside it, and the timeline shows it on the day it ended, without a time or device. |
| `readthrough.started` | `{id, edition?, at?, undated?}` | Opens a read-through. `undated` (`true`) is for a start nobody knows, as for a book imported as read or being read from another app: `at` is then only the earliest reading known of, to order it by, and the timeline shows no start. |
| `readthrough.finished`, `readthrough.stopped`, `readthrough.paused` | `{id, at?}` | Finishes, stops, or pauses it. A read-through can be paused more than once, and stays open. |
| `readthrough.reclassified` | `{id, as}` | Corrects how a read-through ended, keeping the date: its finish or stop becomes `finished`, `stopped`, or `paused` (`as`). A pause leaves it open again. It records no new moment. |
| `readthrough.updated` | `{id, started?, finished?, stopped?}` | Corrects a read-through's dates to a year, month, day, or minute: `2019`, `2019-05`, `2019-05-12`, or `2019-05-12T21:40`. A date left out stays as it is. |
| `readthrough.created` | `{id, started?, finished?, stopped?}` | A read-through someone states rather than one recorded, as for a book read before it was imported, added in `Book.md` (§6.2). It has stated dates, as `readthrough.updated` takes them, and has ended: `finished` or `stopped`, not both. Without `started`, nobody knows when it started. It records no moment, and never takes in the open read-through. |
| `readthrough.deleted` | `{id}` | Removes a read-through, as one taken out of `Book.md`. A tombstone. |

**Annotations** (book logs, §7):

| Event | Data | Meaning |
|---|---|---|
| `annotation.created` | `{id, edition, motivation, target, color?, body?, at?}` | A highlight or bookmark. `motivation` is `highlighting` or `bookmarking`, and `target` a Locator. A highlight has a `color` (`yellow`, the default, `green`, `blue`, `pink`, or `purple`) and an optional note, `body`, in Markdown. A bookmark marks a point, so its Locator's `highlight` is empty. `at` is when it was made, when that's earlier than the event, as for one brought in from another reader. |
| `annotation.updated` | `{id, color?, body?, target?}` | Changes the fields given. An empty `body` removes the note. |
| `annotation.deleted` | `{id}` | A tombstone. |
| `annotation.restored` | `{id}` | Undoes the annotation's deletion, as when its `Edition.md` is put back (§6.1): it's back as it was when deleted. The later of a deletion and a restore wins. |

**User fields** (book logs):

| Event | Data | Meaning |
|---|---|---|
| `status.updated` | `{status}` | `unread` (new books start here), `reading`, `paused`, `read`, or `stopped` (given up on). |
| `rating.updated` | `{rating}` | Whole stars, 1–5; `null`, or no `rating`, clears it. |
| `review.updated` | `{text}` | The review, in Markdown (CommonMark, plus strikethrough). |
| `collection.added`, `collection.removed` | `{name}` | This book was added to, or removed from, a collection (§5.4). |

**Metadata** (book logs, §3):

| Event | Data | Meaning |
|---|---|---|
| `metadata.updated` | `{edition, changes}` | Edits to an edition's metadata, per field. `changes` takes `title`, `titleFileAs`, `subtitle`, `creators`, `series`, `identifiers`, `language`, `publisher`, `published`, `description`, `genres` (the OPF's `dc:subject`), and `cover`, a new cover's image, by its hash (§3.6). A field left out is unchanged, and an empty string clears a text field. A list (`creators`, `series`, `genres`) and `identifiers`, a map of one value per scheme, replace the edition's whole. `language` and `published` are as the OPF has them: a BCP 47 tag, and a date such as `1965` or `1965-08-01`. |
| `transform.requested` | `{edition, name}` | Asks for a transform of an edition's file (§3.4). |
| `metadata.applied` | `{edition, base, hash, edits, transforms?}` | Edits and transforms were written into the file: the SHA-256 before (`base`) and after (`hash`), the `metadata.updated` events written (`edits`, as `{device, sequence}`), and the transforms run, in order, each as `{request, name, version}` with the transform's own version. |

**Files** (book logs, §6):

| Event | Data | Meaning |
|---|---|---|
| `file.applied` | `{file, hash, values, edition?}` | Logged state was written into a file in the book folder, such as `Book.md`: its name, its SHA-256 as written, and the values written, to tell edits made to the file from changes in the logs. `edition` is set for an edition's `Edition.md` (§6.3), and keeps it tracked when its folder is renamed. |
| `file.deleted` | `{file, hash, edition?}` | A file of §6 was deleted, clearing what it held (§6.1): its name, the SHA-256 of the version deleted, and for an `Edition.md`, its edition. That version put back brings it back. |

**Library** (library logs):

| Event | Data | Meaning |
|---|---|---|
| `device.registered` | `{name, platform, model?, key?}` | Names a device (§5.7). |
| `device.forgotten` | `{id}` | A device the reader no longer has, such as a Kobo sold: its registration goes, with its key, so it isn't listed and can't sync directly, until it registers again. Its logs stay, as its reading happened. |
| `book.deleted` | `{id}` | A book was deleted from the library, and its folder moved to the trash (§2.5). |
| `book.restored` | `{id}` | A deleted book's files are back, as when it was put back from the trash: it's no longer deleted (§2.5). |
| `book.merged` | `{from, into}` | Book `from` was merged into book `into` (§2.1, §2.4). |

**Progress is written** when reading pauses, or every 30 seconds while reading, not on every page turn. Opening a book isn't reading it: nothing is written until the reader moves from where it opened, by a page turn, a jump, or playing, so a book opened and closed again keeps its place and its last read.

**Sessions** aren't recorded as they start and end. They're found in each device's `progress.updated`, each at its `at`, else its timestamp: a run of one device's progress in one edition, none more than 5 minutes apart (exactly 5 minutes still is), lasting at least a minute from its first to its last. Progress of a device and edition within a `session.created` of its own that isn't a `total` is part of that session, not a session of its own. `session.created` holds the ones that aren't in the progress any more: compaction writes the sessions it finds as it drops their progress (§5.6), and an import that only knows how long reading took writes one too, as a `total`.

A session that starts behind where reading last was before it, and reads past that place, starts there: the device was catching up to where another had been, without having synced. Where reading last was is the latest progress before the session started, or the latest end of another session by then, on any device, in the session's edition, compared by `totalProgression`: another edition's places, an audiobook's or a translation's, run differently through the book.

**Time spent reading** a book is the time its sessions cover, a stretch counted once where sessions overlap: reading along to the audiobook, or the book open in two editions at once, is one stretch of reading. Sessions in different books both count.

**Reading status:**
- **A book has exactly one reading status.**
- **Reading marks a book as being read.** The first progress a device records in an `unread` book comes with `status.updated` `reading` and the `readthrough.started` it opens.
- **Status drives read-throughs:**
  - Moving to `reading` opens a read-through.
  - Moving to `paused` pauses it, to `read` finishes it, and to `stopped` stops it.
  - Pausing, finishing, or stopping with no read-through open opens one first, at the same moment.
  - Moving straight between `paused`, `read`, and `stopped` with no read-through open corrects the latest one with `readthrough.reclassified` instead. Going through `reading` or `unread` starts a new one.
  - Setting the status a book already has writes no read-through events.
- **Rereads** are new read-throughs.
- **One read-through is open at a time.** Two devices that start reading before seeing each other's start open two: in the fold, one that starts while another is open takes the other in, keeping the earlier start and its pauses, and events for either apply to it.
- **A later ending replaces an earlier one:** a read-through finished on one device and stopped on another ends as the later says.

**Read dates** can be corrected, as for a book read before it was imported. A corrected date is a calendar date of that precision, or a time on the clock, not a moment, and reads the same in any time zone: it floats, as iCalendar calls it. `readthrough.updated` is last-writer-wins per date, and only changes an end the read-through already has: it never finishes or stops one. A read-through can also be stated whole, with `readthrough.created`, and removed, with `readthrough.deleted`; a deletion wins over every other event for the read-through, in whatever order they come.

**A corrected date only changes what's shown.** Sessions stay where they were recorded, so the timeline can show one after a read-through's corrected end: the reader's statement next to the record, and nothing is moved to hide it.

**Showing a time:** a moment is shown as the time where it happened, its event's `offset` (§5.2) applied to it, so a trip shows as it was lived, wherever it's looked at. Devices whose time zone is set by hand, and so rarely follows a trip (`nickel` and `koreader`, §5.7), are the exception: their events take the offset of the latest event before them from an iPhone or iPad (`ios`), which sets its zone by itself and goes along; without one, their own. Every device has the same logs, so every device shows the same time, and writes the same `Book.md`. A stated date is shown as it was stated.

## 5.4 Merging

- **`updated` events:** last-writer-wins per field. The one exception is `edition.updated`'s `imported`, which adds to the IDs the edition has rather than replacing them.
- **Keyed items (annotations, read-throughs):** merged by ID. A tombstone wins over earlier updates.
- **Set membership (collections):** last-writer-wins per (collection, book) pair.
- **Collections are their books' names for them,** as tags are: a collection exists while a book is in it, and only then. There's nothing else to one, so `Book.md` holds all of it (§6.2).
  - **Names match** case-insensitively, after NFC, with whitespace trimmed from their ends (§1.3 says how names compare): "Sci-fi" and "sci-fi" are one collection.
  - **Its spelling** is that of the newest add among the books in it. Adding a book again under another case changes how the collection is written, for every book in it.
  - **Renaming** a collection is taking each of its books out under the old name and adding it under the new one; renaming onto a name that's a collection already merges the two. **Deleting** one is taking each book out. A device that was offline and adds a book under the old name afterwards brings the old one back, with that book: as a tag would, it's there to see, and to rename again.

## 5.5 Continuing from another device

- **Follow the newest progress.** When a book opens and the newest progress in that edition is from another device, open the book there.
- **Follow each progress once,** so going back to an earlier page doesn't pull the reader forward again on the next open.
- **Don't follow progress that's where this device already is:** in the same resource (its `href`, without a fragment), and less than 1% of it away (`progression` within 0.01).
- **Going back** to where this device was, after following another's progress, saves that as the newest progress.

## 5.6 Compaction

- **A device may rewrite its own log,** and only its own, dropping every event that a newer event of its own makes redundant:
  - older values of the same field
  - all but the latest `file.applied` per file
- **Keep** history: read-throughs and editions; and deleted annotations' events, which `annotation.restored` brings back.
- **Write the sessions** found in the progress it drops as `session.created` (§5.3), each in the place of the last line it drops from that session, with that line's timestamp and sequence. A device that already has the line by its sequence (§8.2) has the progress the session was found in, so its state is the same.
- **Event types:** keep a newer format's.
- **The result** folds to the same state and sessions, whatever other devices have written. What it drops is gone for what reads it later: a session caught up (§5.3) by a place it dropped, and the hashes of older versions of a §6 file, which a late copy of one is then read against as an edit.
- **Write it atomically:** a temporary file, then a rename over the log. No other device writes the file, so this is safe over any sync layer. Don't compact where a rename over a file isn't atomic, as on FAT.
- **When:** at about 1,000 lines or 256 KB.

## 5.7 Devices

- **ID format:** `<slug>-<suffix>`, using only `[a-z0-9-]`, such as `iphone-7k3` or `kobo-libra-x7m`.
  - The slug comes from the device's model (`MacBook Pro`, `iPhone`, `Kobo Libra Colour`), not the name the user gave it.
  - The suffix is 3 random characters of RFC 4648's base32 alphabet (`a`–`z`, `2`–`7`). A reset device, or the one that replaces it, never reuses the old ID.
- **Lifetime:** generated on first run and stored on the device, outside the library. Never derived from hardware IDs. A backup or migration can copy it to another device, and two devices writing one log lose each other's lines, so a device also keeps, beside its ID, something that tells it from other devices, such as the system's ID for the app on this device. When that no longer matches, the ID was copied, and it makes a new one.
- **A reset or reinstall** makes a new device ID. Old logs stay valid.
- **`device.registered`** maps an ID to a display `name`, `platform`, and `model`, so a device can be renamed without touching files. `key` is the device's public key for syncing directly: Ed25519, its 32 bytes in unpadded base64url, as a JWK's `x` holds one (RFC 8037).
  - **The name** is the kind of device: `MacBook`, `iMac`, `Mac` (a mini, Studio, or Pro), `iPhone`, `iPad`, `Kobo` (in Nickel or KOReader alike), or `Kindle`. Not its model, which goes in `model`, nor the name the user gave it. Register again when either changes.
  - **The platform** is what wrote it: `macos`, `ios`, `nickel` (Boeks for Kobo), `koreader`, or `cli`; the first four are shown as macOS, iOS, Nickel, and KOReader. **The model** is the device's model as people know it, such as `MacBook Pro` or `Kobo Clara BW`, not a hardware identifier.
  - **Register once the library can be read.** A device that opens it before it can see into it, as before access to its folder or iCloud's files have come through, finding no book where it knew of some, registers at the first refresh that does: nothing is written into a library it can't read.
  - **Show the name alone.** Only when another device has that name too, show what tells them apart: the model (`Kobo Clara BW`), else the platform (`Kobo (KOReader)`), else the ID's suffix (`iPhone (k3x)`).
- **Forgetting a device** (`device.forgotten`) is for one the reader no longer has. It leaves the lists, and its key stops letting it in (§8.2). A device in use registers again as it next writes, and is back.
- **A device was last active** at the newest line in its logs, by its own `timestamp`, leaving out `log.started` and `device.registered`, which it writes without being used.
