Compare commits

..

14 Commits
1.0.7 ... main

Author SHA1 Message Date
vorotamoroz
047429033f Merge pull request #1090 from vrtmrz/1_0_9
Releasing 1.0.9
2026-08-08 22:43:59 +09:00
vorotamoroz
9765569bb6 Add personal note for 1.0.9 2026-08-08 13:04:48 +00:00
github-actions[bot]
c5835a9da6 Releasing 1.0.9 2026-08-08 13:00:23 +00:00
vorotamoroz
8311060f77 Merge pull request #1089 from vrtmrz/reconcile/1.0.8-pre-release
Record the unpromoted 1.0.8 pre-release
2026-08-08 21:47:44 +09:00
vorotamoroz
13d9624407 Record unpromoted 1.0.8 pre-release 2026-08-08 12:33:42 +00:00
vorotamoroz
76560e3bf2 Merge pull request #1083 from calvinbui/fix/qr-aggregator-special-characters
Fix special characters in multi-part settings QR codes
2026-08-08 21:19:22 +09:00
vorotamoroz
1e190d042c Merge main into multi-part settings QR fix 2026-08-08 12:13:31 +00:00
vorotamoroz
40215032dd Merge pull request #1088 from vrtmrz/fix/fast-fetch-page-timeout
Complete bounded Fast Fetch pages on CouchDB 3.2
2026-08-08 20:39:19 +09:00
vorotamoroz
fd9a9175dd Use Commonlib 0.1.8 for reliable Fast Fetch pagination 2026-08-08 11:22:34 +00:00
vorotamoroz
1dfdb72fbd Merge pull request #1085 from vrtmrz/fix/fast-fetch-bounded-pages
Complete bounded Fast Fetch and stop repeated cancelled Fetch
2026-08-08 17:02:05 +09:00
vorotamoroz
23d9fa360d Use Commonlib 0.1.7 for bounded Fast Fetch 2026-08-08 07:44:33 +00:00
vorotamoroz
070b63c952 Document bounded Fast Fetch completion 2026-08-08 05:39:49 +00:00
vorotamoroz
d37af53858 Stop repeating cancelled scheduled Fetch 2026-08-08 04:29:57 +00:00
Calvin Bui
cf5181bb28 Fix special characters in aggregated QR settings 2026-08-07 21:48:14 +10:00
13 changed files with 187 additions and 61 deletions

View File

@@ -33,7 +33,8 @@
const id = params.get('id');
const total = parseInt(params.get('n') || '0');
const index = parseInt(params.get('i') || '-1');
const data = params.get('d');
// Keep the chunk percent-encoded so URI delimiters remain part of the settings payload.
const data = hash.match(/(?:^|&)d=([^&]*)/)?.[1];
const app = document.getElementById('app');

View File

@@ -19,8 +19,9 @@ initialisation workflow relies:
- every result from a batch write is checked;
- a checkpoint represents the last contiguous remote sequence which is durable
in the local database; and
- successful completion means that the captured remote target has been reached,
rather than that an estimated number of documents has been received.
- successful completion requires CouchDB to terminate every finite changes page,
every returned row to be durable, and a subsequent normal probe to report no
available rows.
The existing implementation combines line parsing, decryption, persistence, and
completion checks within one broad error handler. A decryption or persistence
@@ -30,35 +31,89 @@ cases the stream may continue, advance its checkpoint incorrectly, or wait
indefinitely for a completion condition which the failed row would have
satisfied.
Fast Fetch also estimates the number of documents from the changes feed's
`pending` value. That value is useful for progress reporting, but it is not an
authoritative completion boundary. The CouchDB sequence token is opaque and must
be handled using CouchDB's sequence semantics, without numeric-prefix comparison
or inferred row counts. On clustered CouchDB, database information and the
changes feed may encode the same position with different opaque tokens, so an
`update_seq` from database information must not be compared directly with a
changes-feed row.
Fast Fetch also reads the normal changes feed before opening each continuous
page. A normal response's `pending` value counts items which remain after the
response's `results`, so `pending` alone is not the available workload. With a
one-row probe, the page can contain `results.length + pending` rows.
The documented `limit=0` behaviour cannot be used as a portable zero-payload
probe. CouchDB's API documentation says that `limit=0` has the same effect as
`limit=1`, while CouchDB 3.5.0 with a two-shard database was observed to return
no result rows and leave the complete count in `pending`. Fast Fetch therefore
uses an explicit one-row normal probe and includes no document bodies.
Finite continuous-feed completion differs across supported CouchDB releases.
CouchDB 3.5.0 was observed to close a heartbeat-enabled feed with a
`{ "last_seq": ... }` line when its finite `limit` is met. CouchDB 3.2 instead
continues to wait for database updates after the limit has been consumed. With
a heartbeat configured, each wait emits another heartbeat and the page can
remain open indefinitely, even after every requested row has arrived.
On CouchDB 3.2, an explicit `timeout` without a heartbeat has different
semantics from a total request deadline. Shard-result waits may emit blank
keep-alive lines and continue processing. Once the currently available changes
have been exhausted and the feed is waiting for another database update, the
timeout stops that wait and returns the feed-level `last_seq`. The timeout can
therefore terminate a finite page without limiting the duration of an active
page transfer.
The CouchDB sequence token is opaque and must be handled using CouchDB's
sequence semantics, without parsing, ordering, or comparison. On clustered
CouchDB, a changes row and the feed-level `last_seq` may encode related
positions with different opaque tokens. Separate requests are also not one
locked snapshot: their rows may be partially ordered, and replica failover may
repeat changes. Fast Fetch must therefore be idempotent and must use each
terminal `last_seq` only by returning it to CouchDB as the next `since` value.
## Decision
### Remote snapshot and completion
### Remote page sizing and completion
Fast Fetch must obtain an authoritative target token from a normal changes-feed
snapshot before consuming the stream. A request from `since=now` with no result
rows provides a token in the same sequence domain as the streamed rows. If that
target cannot be obtained, the fetch fails instead of falling back to a database
information token, a document-count estimate, or another approximate sequence.
Fast Fetch obtains an approximate progress target from a normal changes-feed
request with `since=now`, `limit=1`, and `include_docs=false`. This token is for
progress reporting only. It is not compared with any other token and is not
used as a completion checkpoint.
The target token is treated as opaque. Fast Fetch completes only after the row
for the captured target has been processed and all work up to that row has been
persisted successfully. When a status request proves that no changes exist after
the current durable checkpoint, the captured target may be checkpointed without
opening the continuous stream. This includes an empty remote database. Changes
made remotely after the target was captured are outside this Fast Fetch snapshot
and are left for subsequent ordinary replication.
Before every bounded page, Fast Fetch requests a normal changes feed from the
current durable cursor with `limit=1` and `include_docs=false`. The probe and
the following continuous page use the same `since`, style, and filter
selection. Reading the probe does not consume rows from CouchDB; the continuous
request starts again from that same cursor.
The estimated document count remains available for progress reporting only. It
must not determine success.
The number currently available is `results.length + pending`. If it is zero,
Fast Fetch is caught up and completes without opening another stream. Otherwise,
the next continuous request uses the smaller of that count and 10,000 as its
finite `limit`.
Each finite page omits `heartbeat` and sets `timeout=1000`. This lets CouchDB
3.2 return the page's terminator one second after it exhausts the currently
available changes, rather than keeping the request open for future writes. The
client immediately reconnects from that terminator while another normal probe
reports available work. This bounded cycle also preserves the intent of the
earlier iOS and iPadOS heartbeat workaround: Fast Fetch no longer depends on a
silent continuous request eventually closing at CouchDB's default 60-second
timeout.
The probe and page are separate HTTP requests, not a transactional snapshot.
New writes, replica selection, or administrative changes may alter the rows
between them. A page which returns at least one row and a valid terminator may
therefore be shorter than the probe's estimate. Fast Fetch persists that page
and probes again. A page which terminates without making progress after a
positive probe is a retryable transport failure, avoiding an unbounded busy
loop.
Each continuous request ends with its own `{ "last_seq": ... }` line. Fast
Fetch treats that line separately from a changes row, flushes and validates all
preceding local writes, and only then persists the opaque `last_seq`. The exact
value is replayed as the next request's `since`; it is never parsed, ordered, or
compared with a row's `seq`, another request's `last_seq`, or the database's
`update_seq`.
The limit counts outer changes-result rows. A tombstone is one row and consumes
one page slot even when no document body is present. With `style=all_docs`,
multiple leaf revisions inside one row's `changes` array do not consume
additional slots. Changing `include_docs` between the lightweight probe and the
document-bearing continuous page changes the payload, not the row selection.
### Processing and persistence
@@ -68,8 +123,9 @@ stages:
1. parse and validate the changes-feed row;
2. decrypt and validate its document, when a document is present;
3. add the document to the pending local batch;
4. persist the batch; and
5. inspect every result returned by the batch write.
4. persist the batch;
5. inspect every result returned by the batch write; and
6. after the finite page ends, persist its `last_seq` terminator.
With `new_edits: false`, PouchDB follows CouchDB behaviour and may omit successful
results. Fast Fetch therefore inspects every returned result and treats any
@@ -79,8 +135,8 @@ the complete batch.
The checkpoint may advance only to the last contiguous sequence for which all
preceding documents are durable. A row which legitimately requires no local
write may advance the checkpoint only after any preceding buffered documents
have been flushed successfully. The target sequence is committed under the same
rule before the operation reports success.
have been flushed successfully. A page's `last_seq` is committed under the same
rule before that page reports success.
If a batch is partly written, its checkpoint is not advanced. Retrying the batch
with `new_edits: false` is expected to be idempotent, including for documents
@@ -90,8 +146,8 @@ Blank heartbeat lines are ignored. Malformed rows are failures; they are not
silently skipped. Logs may describe the stage and sequence involved, but must
not include the raw changes-feed line because it may be large or sensitive.
The continuous changes request and its decoded reader must be terminated on
every exit. Releasing a reader lock alone does not cancel the underlying
Each finite continuous changes request and its decoded reader must be terminated
on every exit. Releasing a reader lock alone does not cancel the underlying
request. Failure and completion paths therefore abort the request and attempt
to cancel the reader before the bounded remote-activity scope ends.
@@ -156,7 +212,7 @@ The responsibilities are divided at three injectable boundaries.
The Commonlib streaming implementation owns HTTP response validation, NDJSON
parsing, invocation of the decryption delegate, batch-write result validation,
contiguous checkpoint advancement, target-sequence completion, and classified
contiguous checkpoint advancement, finite-page completion, and classified
failures. It does not know about the Vault, setup dialogues, flag files, or
LiveSync settings.
@@ -187,7 +243,8 @@ This decision does not:
- add an automatic fallback from Fast Fetch to Standard Fetch;
- define the detailed failure dialogue or other setup user-interface changes;
or
- require Fast Fetch to include remote changes made after its captured target.
- provide a transaction or locked snapshot across the normal probe and the
following continuous page.
An explicit Standard Fetch choice remains available when a user needs the
ordinary replication path. Any automatic fallback or richer recovery dialogue
@@ -209,8 +266,20 @@ writer. Verify that:
- a partly failed batch leaves the checkpoint unchanged and reports a storage
failure;
- rows without a local write flush earlier buffered documents before advancing;
- an estimated document count cannot complete the fetch;
- the captured target cannot complete the fetch before its batch is durable;
- a returned probe row is counted in addition to `pending`, including when
`pending` is zero;
- every probe uses `limit=1`, excludes document bodies, and is repeated from the
previous page's opaque terminator;
- each bounded page omits `heartbeat`, uses `timeout=1000`, and can complete
under CouchDB 3.2 after its current rows have been delivered;
- deletion and document-less rows consume a page slot;
- a row count cannot complete a page without its `last_seq` terminator;
- a page terminator cannot advance the checkpoint before its batch is durable;
- a final row and `last_seq` with different opaque representations complete
normally without a token comparison;
- a shorter valid page is persisted and followed by another probe, while a
zero-row page after a positive probe fails without looping;
- workloads over 10,000 rows resume from each durable finite-page checkpoint;
- authentication and malformed-protocol responses are terminal;
- recognised transient transport failures are classified as retryable; and
- diagnostics do not log the raw changes-feed line.
@@ -236,9 +305,15 @@ existing setup sequence and cleanup.
### Integration and E2E tests
Commonlib's CouchDB integration test remains responsible for the real HTTP
changes feed, opaque sequence tokens, and local batch persistence. It should
include a data set large enough to cross a batch boundary and confirm that the
final checkpoint equals the captured target.
changes feed, opaque sequence tokens, deletion rows, and local batch
persistence. It should use the maintained CI CouchDB release, a two-shard
database, and a data set large enough to cross a local batch boundary, and
confirm that the final checkpoint can be passed back to CouchDB as `since` with
no result rows or pending changes. The test must not compare that token's
representation with a separately requested target or changes-row token. The
focused regression test covers CouchDB 3.2's page-tail behaviour; compatibility
with a real CouchDB 3.2 server can be confirmed manually without expanding the
permanent CI matrix.
LiveSync's real Obsidian Setup URI workflow remains responsible for the actual
Fast Fetch selection, E2EE passphrase, Vault reflection, ordinary file round
@@ -256,8 +331,11 @@ This follows [Real Obsidian E2E](2026_06_real_obsidian_e2e.md).
attempt without exposing the partial database to the Vault.
- Retry delays are no longer spent on authentication, corrupt content, protocol,
or local persistence failures which cannot repair themselves.
- Progress totals remain approximate and may change without affecting
correctness.
- Progress totals remain approximate and may grow when a later probe observes
new work, without affecting correctness.
- A completed page can spend up to one second waiting for its terminator before
Fast Fetch probes and reconnects. Active page transfer is not constrained to
one second.
- The implementation requires coordinated changes in Commonlib and LiveSync.
Commonlib remains the authoritative package for streaming and rebuilder
behaviour; LiveSync consumes an immutable Commonlib release and owns its setup
@@ -265,3 +343,9 @@ This follows [Real Obsidian E2E](2026_06_real_obsidian_e2e.md).
- Ordinary replication remains unchanged and continues to provide the reference
correctness contract for decrypting, persisting, and checkpointing replicated
documents.
## References
- [Apache CouchDB changes-feed API](https://docs.couchdb.org/en/stable/api/database/changes.html)
- [Apache CouchDB 2.0 upgrade notes for opaque update sequences](https://docs.couchdb.org/en/stable/whatsnew/2.0.html#upgrade-notes)
- [Apache CouchDB replication protocol](https://docs.couchdb.org/en/stable/replication/protocol.html)

View File

@@ -1,7 +1,7 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "1.0.7",
"version": "1.0.9",
"minAppVersion": "1.7.2",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",

18
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{
"name": "obsidian-livesync",
"version": "1.0.7",
"version": "1.0.9",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "obsidian-livesync",
"version": "1.0.7",
"version": "1.0.9",
"license": "MIT",
"workspaces": [
"src/apps/cli",
@@ -23,7 +23,7 @@
"@smithy/types": "^4.14.3",
"@smithy/util-retry": "^4.4.5",
"@vrtmrz/browser-ui-kit": "0.1.0",
"@vrtmrz/livesync-commonlib": "0.1.6",
"@vrtmrz/livesync-commonlib": "0.1.8",
"@vrtmrz/obsidian-plugin-kit": "0.1.3",
"@vrtmrz/ui-interactions": "0.1.2",
"diff-match-patch": "^1.0.5",
@@ -4775,9 +4775,9 @@
}
},
"node_modules/@vrtmrz/livesync-commonlib": {
"version": "0.1.6",
"resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.6.tgz",
"integrity": "sha512-rKpiTZYZRLaYcBQ2gcSyPGa9HONtataIB75dfP9+6BFlq/KzXimhKf1RqWxkAJj5GGzKmhtq+vXXExA+vAshMQ==",
"version": "0.1.8",
"resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.8.tgz",
"integrity": "sha512-Kn1AF41h2Dog37ThU7KgLcKxItCCerLEBWg1eSGAUoTk3TyPBYynvtmVFwWhy6LCePcuwB/+x7EQNTL3Sgzkyg==",
"license": "MIT",
"dependencies": {
"@aws-sdk/client-s3": "^3.808.0",
@@ -15924,7 +15924,7 @@
},
"src/apps/cli": {
"name": "self-hosted-livesync-cli",
"version": "1.0.7-cli",
"version": "1.0.9-cli",
"dependencies": {
"chokidar": "^4.0.0",
"minimatch": "^10.2.5",
@@ -15949,7 +15949,7 @@
},
"src/apps/webapp": {
"name": "livesync-webapp",
"version": "1.0.7-webapp",
"version": "1.0.9-webapp",
"dependencies": {
"octagonal-wheels": "^0.1.52"
},
@@ -15961,7 +15961,7 @@
}
},
"src/apps/webpeer": {
"version": "1.0.7-webpeer",
"version": "1.0.9-webpeer",
"dependencies": {
"octagonal-wheels": "^0.1.52"
},

View File

@@ -1,6 +1,6 @@
{
"name": "obsidian-livesync",
"version": "1.0.7",
"version": "1.0.9",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js",
"type": "module",
@@ -177,7 +177,7 @@
"@smithy/types": "^4.14.3",
"@smithy/util-retry": "^4.4.5",
"@vrtmrz/browser-ui-kit": "0.1.0",
"@vrtmrz/livesync-commonlib": "0.1.6",
"@vrtmrz/livesync-commonlib": "0.1.8",
"@vrtmrz/obsidian-plugin-kit": "0.1.3",
"@vrtmrz/ui-interactions": "0.1.2",
"diff-match-patch": "^1.0.5",

View File

@@ -1,7 +1,7 @@
{
"name": "self-hosted-livesync-cli",
"private": true,
"version": "1.0.7-cli",
"version": "1.0.9-cli",
"main": "dist/index.cjs",
"type": "module",
"scripts": {

View File

@@ -1,7 +1,7 @@
{
"name": "livesync-webapp",
"private": true,
"version": "1.0.7-webapp",
"version": "1.0.9-webapp",
"type": "module",
"description": "Browser-based Self-hosted LiveSync using FileSystem API",
"scripts": {

View File

@@ -1,7 +1,7 @@
{
"name": "webpeer",
"private": true,
"version": "1.0.7-webpeer",
"version": "1.0.9-webpeer",
"type": "module",
"scripts": {
"dev": "vite",

View File

@@ -150,7 +150,7 @@ export function createFetchAllFlagHandler(
// Select the remote database if there are multiple remotes configured.
const isRemoteActivated = await askAndActivateRemoteDatabase(host, log);
if (!isRemoteActivated) {
return false;
return await cancelScheduledInitialisation(host, cleanupFlag);
}
// Ask user for use Fast Setup

View File

@@ -581,6 +581,16 @@ describe("Red Flag Feature", () => {
expect(result).toBe(false);
expect(host.mocks.ui.confirm.confirmWithMessage).not.toHaveBeenCalled();
expect(host.mocks.storageAccess.files.has(FlagFilesOriginal.FETCH_ALL)).toBe(false);
await expect(handler.check()).resolves.toBe(false);
expect(host.mocks.setting.applyPartial).toHaveBeenCalledWith(
{
suspendFileWatching: true,
suspendParseReplicationResult: true,
},
true
);
expect(host.mocks.appLifecycle.performRestart).toHaveBeenCalledOnce();
});
it("should activate selected remote configuration", async () => {

View File

@@ -30,12 +30,15 @@ Deno.test({
const aggregator = await browser.newPage();
const assertNoAggregatorFailures = observePageFailures(aggregator);
const assertNoAggregatorNetworkFailures = observeNetworkFailures(aggregator);
await aggregator.goto(new URL("aggregator.html#id=pages-smoke&n=2&i=0&d=first-", server.baseUrl).href);
await aggregator.goto(new URL("aggregator.html#id=pages-smoke&n=2&i=0&d=before%2", server.baseUrl).href);
await aggregator.getByText("1 / 2 Loaded", { exact: true }).waitFor();
await aggregator.goto(new URL("aggregator.html#id=pages-smoke&n=2&i=1&d=second", server.baseUrl).href);
await aggregator.goto(
new URL("aggregator.html#id=pages-smoke&n=2&i=1&d=3after%26amp%2Bplus%25percent", server.baseUrl)
.href
);
assertEquals(
await aggregator.getByRole("link", { name: "Open Obsidian to complete setup" }).getAttribute("href"),
"obsidian://setuplivesync?settingsQR=first-second"
"obsidian://setuplivesync?settingsQR=before%23after%26amp%2Bplus%25percent"
);
assertNoAggregatorFailures();
assertNoAggregatorNetworkFailures();

View File

@@ -12,6 +12,32 @@ Earlier releases remain available in the 0.25 release history and the legacy rel
## Unreleased
## 1.0.9
8th August, 2026
For the first time in a while, I published a release that could not be promoted to a stable release. Sorry about that! I am glad that we caught it while it was still a pre-release.
### Setup and compatibility
#### Fixed
- Multi-part settings QR codes now preserve special characters in passwords, passphrases, and other settings (PR #1083). Thank you to @calvinbui for the improvement!
- Fast Setup now sizes each finite CouchDB changes page from a one-row status probe, counts the returned result together with `pending`, and resumes from the page's opaque `last_seq` without comparing token representations. Each page uses a one-second idle timeout instead of a heartbeat, allowing CouchDB 3.2 to return its terminator after the currently available rows have been persisted.
## 1.0.8
8th August, 2026
This version was published for pre-release validation only and was not promoted to a stable release.
### Setup and compatibility
#### Fixed
- Fast Setup now sizes each finite CouchDB changes page from a one-row status probe, counts the returned result together with `pending`, and resumes from the page's opaque `last_seq` without comparing token representations. Heartbeat-enabled feeds no longer wait for future writes after the currently available rows have been persisted (#1065).
- Cancelling remote selection during a scheduled Fetch now removes the Fetch flag before restarting with file and database reflection paused, preventing the same selection dialogue from reopening on every start-up.
## 1.0.7
8th August, 2026

View File

@@ -19,5 +19,7 @@
"1.0.4": "1.7.2",
"1.0.5": "1.7.2",
"1.0.6": "1.7.2",
"1.0.7": "1.7.2"
"1.0.7": "1.7.2",
"1.0.8": "1.7.2",
"1.0.9": "1.7.2"
}