Compare commits

...

9 Commits

Author SHA1 Message Date
vorotamoroz
a722244a92 fix(cli): replace the headless log handler 2026-08-01 06:20:47 +00:00
vorotamoroz
4f14d7f160 feat(cli): support adaptive WebDAV remotes 2026-08-01 06:09:32 +00:00
vorotamoroz
7955ab54d3 refactor: compose Journal remotes generically 2026-08-01 06:09:02 +00:00
vorotamoroz
133ef34d7b docs: model adaptive WebDAV trade-offs 2026-08-01 04:16:37 +00:00
vorotamoroz
0890a97222 test(obsidian): cover adaptive S3 journal upload 2026-08-01 03:52:57 +00:00
vorotamoroz
b7da78ce45 feat: expose adaptive journal options for S3 2026-07-31 19:24:40 +00:00
vorotamoroz
53d4bd527e docs: split adaptive journal decisions by provider 2026-07-31 19:24:40 +00:00
vorotamoroz
64870a32ea docs: specify adaptive journal sync design 2026-07-31 19:24:40 +00:00
vorotamoroz
5ede11d032 docs: plan adaptive journal synchronisation 2026-07-31 19:24:40 +00:00
37 changed files with 2880 additions and 64 deletions

View File

@@ -0,0 +1,90 @@
# Architectural Decision Record: Use Native Batch CAS for Adaptive PostgREST
## Status
Proposed as the final experimental provider, after the S3 and WebDAV delivery sequences have established the common
protocol, CLI, host, and end-to-end boundaries.
## Context
PostgREST can expose PostgreSQL uniqueness, row-level security, bounded set operations, and transactions. Treating it
as another opaque object store works, but leaves Metadata and Chunks combined and cannot use a multi-key Chunk query.
Using one HTTP request per logical Chunk would make latency dominate synchronisation.
The common protocol decision is recorded in
[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The tables, binary RPC framing,
transaction rules, and privacy model are specified in the
[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md).
## Decision
Adaptive PostgREST separates Metadata publication from immutable Chunk storage physically as well as logically.
- Vault- and repository-scoped Chunk rows use the Remote Chunk key as an insert-only unique address.
- Bounded binary RPCs implement `hasMany`, `getMany`, and `putMany` while preserving input order and per-entry status.
- Writer descriptors, Metadata batches, and commits remain small append-only records.
- A transactional commit verifies the sorted required-Chunk-key digest and the existence of every referenced Chunk
before making Metadata visible.
- Concurrent insertion of a different encrypted frame for the same logical Chunk reads, validates, and accepts the
winning plaintext only when it represents the same logical value.
- Row-level security scopes every operation to the configured Vault. Server-visible Remote Chunk keys reveal equality
within that repository but do not reveal plaintext Chunk IDs.
PostgREST does not use object packs, catalogue deltas, or Range retrieval for native Chunk rows. The shared binary Chunk
record remains independently verifiable, but PostgreSQL supplies the batch index and uniqueness boundary.
The SQL schema and RPC contract are versioned together with the Adaptive format. A missing or incompatible schema is
detected before publication and requires applying the reviewed schema or rebuilding the remote. The client does not
perform an implicit data-format migration.
## Staged acceptance
### Adapter and Commonlib integration
Unit tests own binary-envelope limits, ordering, status decoding, insert conflicts, rollback, error classification,
Vault isolation, and malformed responses. Disposable PostgreSQL and PostgREST integration tests own the real SQL
schema, row-level security, RPC transactions, and two-client Metadata and Chunk synchronisation.
### CLI end-to-end acceptance
The built CLI applies an Adaptive PostgREST Setup URI to a second independent database and synchronises text and binary
Chunk-backed files through disposable PostgreSQL and PostgREST services. The scenario proves bounded native batching at
the headless product boundary without repeating the SQL failure matrix.
### Host settings and UI
The PostgREST dialogue persists the endpoint, Vault identifier, authentication configuration, Adaptive format, and
expected repository ID. Focused tests own connection-string and Setup URI preservation, validation, and format-mismatch
guidance; they do not execute SQL.
### Real-host end-to-end acceptance
One real-Obsidian workflow applies the reviewed schema and performs a representative Chunk-backed transfer. The
disposable Commonlib integration remains authoritative for RLS, transaction rollback, and the complete RPC matrix.
## Alternatives rejected
### Reuse immutable object packs in PostgreSQL
This would preserve portability at the cost of native multi-key lookup and transaction guarantees. The repository
contract already permits a different physical representation.
### Store raw file content with Metadata
Metadata must remain small and must continue to refer to immutable logical Chunks. Combining content with Metadata
would break the maintained PouchDB model and duplicate unchanged content across revisions.
### Encode Chunk bodies as JSON values
Base64 and large JSON arrays add framing and memory overhead and make limits harder to enforce. Bounded binary `bytea`
RPC envelopes provide deterministic lengths and status ordering.
## Consequences
- PostgREST has the largest provider-specific implementation because it includes reviewed SQL, RLS, RPC framing, and
transaction behaviour.
- Placing it last lets the common protocol, CLI, and host boundaries stabilise before introducing that larger surface.
- Native batching can reduce request count substantially relative to object-per-Chunk storage, but actual speed still
depends on database, proxy, network, and workload measurements.
- PostgREST and object stores share logical records and correctness rules without pretending to share a physical
layout.

View File

@@ -0,0 +1,98 @@
# Architectural Decision Record: Introduce Adaptive Journal as an Explicit Protocol
## Status
Proposed for staged implementation. This decision does not change the current default Journal format or promise a
compatible in-place migration.
## Context
The existing Journal protocol publishes PouchDB Metadata and Chunk documents together in opaque immutable objects.
That representation is portable and remains the compatibility baseline, but it prevents a remote from answering a
bounded multi-key Chunk query or amortising object-store requests independently from Metadata publication.
S3-compatible Object Storage, WebDAV, and PostgREST have materially different physical capabilities. Making each one
implement a separate replication algorithm would duplicate ordering, encryption, recovery, and Chunk-delivery rules.
Making every one use an identical physical layout would discard useful native batching and transactions.
The detailed binary formats, state machines, privacy properties, and recovery rules are specified in the
[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md).
## Decision
Adaptive Journal is a second, explicit Journal protocol version owned by Commonlib. It keeps one logical repository
contract while allowing provider-specific physical storage.
- Metadata events and raw Chunk records remain separate throughout publication.
- Logical Chunks are immutable and content-addressable. A changed value has a new logical Chunk ID.
- A repository-scoped Remote Chunk key is derived locally from the exact logical Chunk ID after accepting the
repository manifest.
- Chunks become durable before a commit makes Metadata references visible.
- Ordinary publication creates immutable records. Catalogue snapshots, caches, and indexes are derived and
reconstructible.
- Each writer publishes a dense sequence identified by a stable host ID and a persisted random writer epoch. A reader
tracks one frontier per writer stream; it does not depend on a remote `startAfter` ordering contract.
- Remote operations return typed outcomes which distinguish absence, conflict, permanent rejection, retryable failure,
and an ambiguous mutation which must be verified before retrying.
The adaptive repository exposes batched Chunk availability, publication, and retrieval even when its implementation
uses immutable packs internally. The caller does not issue one remote request per Chunk.
### Repository identity and compatibility
A new repository has one immutable, conditionally created manifest. The first device generates its candidate locally,
and the winning manifest fixes the repository ID, Security Seed, protocol parameters, and required capabilities.
Every client pins the accepted repository ID in local repository state. A Setup URI exported from an accepted binding
should carry that non-secret expected ID.
`opaque-v1` and `adaptive-v1` are separate remote formats. Format selection is explicit in the remote configuration,
and a mismatch fails before publication. The implementation detects incompatible remote data, but it does not migrate
it. Changing format requires an explicit remote rebuild or a new remote namespace.
Deprecated data formats do not enlarge the Adaptive protocol. Compatibility remains at the existing decoding
boundaries, and rebuilding the remote is the recovery path for an unsupported Adaptive layout.
### Provider delivery sequence
Each provider is delivered and reviewed through four boundaries, in order:
1. **Adapter and Commonlib integration.** Implement semantic capabilities, typed failure handling, format detection,
unit tests, and disposable real-service integration tests.
2. **CLI end-to-end acceptance.** Use the built CLI and a real disposable service to apply a Setup URI, synchronise two
independent local databases, and restore text and binary Chunk content. This proves the headless product boundary
without Obsidian or Svelte.
3. **Host settings and UI.** Add only provider-specific controls, validation, profile persistence, Setup URI transport,
and focused host tests.
4. **Real-host end-to-end acceptance.** Exercise one representative setup and synchronisation path in a real Obsidian
instance. This test verifies host composition and does not repeat the adapter capability matrix or the complete CLI
conflict suite.
The initial order is S3-compatible Object Storage, WebDAV, then PostgREST. A provider completes these boundaries before
the next provider is presented for integration review. The sequence keeps each review independently attributable and
keeps the real-host tests small.
## Alternatives rejected
### One cross-provider implementation change
This hides which provider requires a shared-core change, makes failures difficult to attribute, and forces reviewers to
understand SQL RPCs, object packs, WebDAV behaviour, CLI composition, and Obsidian UI in one change.
### Add the Host UI before headless acceptance
This makes an application-level failure ambiguous between the protocol, adapter, CLI-independent host composition, and
presentation. The CLI provides the smaller executable boundary first.
### Use one physical representation for every provider
One-object-per-Chunk storage causes excessive object requests, while forcing packs into PostgreSQL discards bounded
multi-key RPCs and transactions. Common semantics do not require common physical storage.
## Consequences
- Commonlib owns protocol correctness and provider adapters; Self-hosted LiveSync owns CLI composition, Obsidian host
integration, and presentation.
- Provider reviews can stop at the first failed boundary without involving later UI or real-host tests.
- The final end-to-end suite remains an acceptance layer rather than a duplicate protocol test suite.
- Adding another provider requires the same semantic contract and staged evidence, not another replication algorithm.
- Remote rebuild remains an explicit operational requirement while Adaptive Journal is evolving.

View File

@@ -0,0 +1,96 @@
# Architectural Decision Record: Use S3 as the Reference Adaptive Object Store
## Status
Proposed as an improvement to the maintained S3-compatible Journal path. Adaptive Journal remains explicit and
opt-in; the existing opaque format remains the compatibility default.
## Context
S3-compatible Object Storage already supplies the maintained Journal object model and is the smallest provider on
which to prove Adaptive immutable packs. It has a standard conditional-create request, paginated listing, binary
objects, and optional byte-range retrieval, but compatible endpoints can still differ in consistency, proxy behaviour,
and Range handling.
The common protocol decision is recorded in
[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The complete object-pack and catalogue
formats are specified in the [Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md).
## Decision
S3-compatible storage is the reference implementation of the Adaptive object-store strategy.
- It stores the manifest, writer control records, Metadata batches, commits, Chunk packs, indexes, and catalogue records
as immutable objects under the configured bucket namespace.
- Manifest and every immutable publication use conditional create. A successful create response is sufficient; the
client does not add a confirmation request solely for caution.
- A lost or ambiguous mutation response returns `verify-first`. The caller reads the exact key before retrying.
- A conditional-request conflict which does not establish an existing immutable value remains retryable and is not
reported as a successful create.
- Listing follows every continuation token and proves complete prefix enumeration. Folder marker objects and stale
capability-probe objects are not repository data.
- Format inspection distinguishes empty, `opaque-v1`, `adaptive-v1`, and mixed repositories. Mixed or mismatched data
fails closed, and reset remains an explicit batched operation.
The portable retrieval policy is `whole-pack`. A user may select `range` when the endpoint capability probe has
confirmed exact byte-range behaviour. Range responses must use `206`, carry a matching `Content-Range`, and return the
requested number of bytes. Loss of optional Range support falls back to whole-pack retrieval after reporting the
capability change; it does not make valid packs unreadable.
Capability results which prove support or lack of support may be cached for the endpoint identity. A transient probe
failure is not cached as a permanent result. Probe objects use a reserved random prefix and are removed when possible;
incomplete cleanup is reported and ignored by format detection.
## Staged acceptance
### Adapter and Commonlib integration
Focused tests cover conditional creation, ambiguous responses, binary fidelity, read-after-write visibility, paginated
listing, deletion visibility, Range validation, format detection, reset batching, and probe cleanup. A disposable
MinIO integration proves two-client Adaptive Metadata and Chunk synchronisation and both retrieval paths.
### CLI end-to-end acceptance
One real-MinIO scenario uses two independent CLI databases. Device A uses whole-pack retrieval. Device B receives an
Adaptive S3 Setup URI selecting Range retrieval. The scenario verifies that the URI preserves the format and policy,
that both devices can publish and receive more than one synchronisation round, and that text and binary content are
reconstructed from Chunks.
This test owns the built CLI, settings persistence, Setup URI decoding, and headless composition. It does not repeat
every S3 error classification already owned by Commonlib.
### Host settings and UI
The Object Storage dialogue exposes `opaque-v1` and `adaptive-v1`, with whole-pack as the Adaptive default and Range as
an explicit preference. It preserves the expected repository ID and read policy through saved profiles and Setup URI
handling. Focused host tests own normalisation, validation, persistence, and warnings; they do not contact S3.
### Real-host end-to-end acceptance
One real-Obsidian workflow applies an Adaptive S3 configuration and proves a representative Chunk-backed file transfer
against disposable MinIO. It verifies the Obsidian composition boundary only. The CLI suite remains the broader
headless synchronisation acceptance test.
## Alternatives rejected
### Re-upload a mutable pack when one Chunk changes
Packs are immutable publication units. A changed logical Chunk is added to a new pack, and a new catalogue delta makes
that location discoverable. Existing packs are replaced only by a separately protected compaction process.
### Require Range for Adaptive S3
Whole-pack reads often provide better throughput and work on more endpoints. Range is a deployment preference, not a
correctness requirement.
### Confirm every successful mutation with another request
This doubles request count in the ordinary success path. Exact-key verification is reserved for ambiguous outcomes and
explicit diagnostics.
## Consequences
- S3 establishes the object-store contract before the more variable WebDAV implementation.
- The two retrieval policies are exercised without duplicating the complete CLI scenario.
- Existing opaque S3 repositories remain readable and never become Adaptive implicitly.
- Endpoint capability evidence, rather than the S3-compatible label alone, controls optional behaviour.

View File

@@ -0,0 +1,240 @@
# Architectural Decision Record: Gate Adaptive WebDAV with an Endpoint Safety Check
## Status
Proposed as an experimental provider after the S3 Adaptive path has completed adapter, CLI, host, and real-host
acceptance.
## Context
WebDAV exposes an object-shaped interface suitable for immutable Commit Bundles and Packs, but method support and
conditional semantics vary across servers, gateways, reverse proxies, and authentication layers. A server can
advertise WebDAV while failing binary fidelity, replacing an object despite `If-None-Match: *`, returning incomplete
listings, or ignoring Range.
The common protocol decision is recorded in
[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The Pack background, flat object
mapping, and detailed safety-check sequence are specified in the
[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md). Commonlib owns the implemented
`commit-bundle-v1` wire contract.
## Decision
Adaptive WebDAV uses the same immutable Commit Bundle and external Pack semantics proven by S3, with a WebDAV-specific
flat object mapping inside the configured collection. Its Catalogue is derived locally from authenticated routes in
received Bundles rather than stored as a separate mutable remote object. Logical writer ordering comes from the host
ID, writer epoch, and dense sequence embedded in authenticated records. Correctness does not depend on a
server-provided `startAfter` order.
Before a new Adaptive repository becomes writable, a non-destructive endpoint safety checker exercises random reserved
probe keys and reports observed semantic capabilities:
- binary write and exact read-back;
- read-after-write visibility;
- complete collection listing for the probe keys;
- `If-None-Match: *` preventing replacement;
- delete visibility; and
- exact byte-range behaviour, reported separately as optional.
The checker is an implementation acceptance target for compatible servers, not a claim that every WebDAV server must
support Adaptive Journal. It touches no repository objects, attempts to remove every probe, reports incomplete cleanup,
and never interprets authentication, permission, timeout, malformed response, or server failure as absence.
Conditional create, binary fidelity, complete listing, read-after-write visibility, and delete visibility are required.
Range remains optional. The user selects whole-pack or Range retrieval based on their endpoint and own latency and
throughput preference; the checker does not benchmark or recommend a policy.
Successful mutations do not receive unconditional confirmation requests. An ambiguous response is classified as
`verify-first`, and the exact immutable key is read before retrying. The existing opaque WebDAV layout and Adaptive
layout are detected separately; a mismatch requires a remote rebuild or another namespace.
## Working hypotheses and illustrative estimates
The following values are planning assumptions for the first WebDAV experiment. They are not benchmark results, a
provider-pricing statement, or a performance guarantee. The implementation must record the corresponding measurements
before this ADR presents either Journal format as generally faster or smaller.
### Cost and lifecycle assumptions
The initial deployment model is a self-hosted or quota-priced WebDAV service without a direct per-request tariff.
Request count still matters because every request consumes a round trip, server work, connection capacity, and a share
of any rate limit. Commercial services, gateways, and managed hosting may use a different charging model, so the
adapter must not infer cost policy from the WebDAV label.
Storage capacity is expected to be the more visible constraint. RFC 4331 defines optional
[`DAV:quota-used-bytes` and `DAV:quota-available-bytes`](https://www.rfc-editor.org/rfc/rfc4331.html) properties for
collections. When both are available, a connection diagnostic may report them as server-supplied evidence. Their
absence means 'unknown', not 'unlimited', and quota reporting is not an Adaptive safety requirement.
The first experiment has no ordinary remote Garbage Collection or repacking guarantee. Commit Bundles and external
Packs remain immutable, unreachable data may remain retained, and an interrupted publication may leave an unreferenced
Pack. A remote Rebuild is therefore the only guaranteed way to reclaim all experimental history. This makes retained
bytes and object count acceptance measurements rather than later operational details.
Adaptive and Opaque Journal consume the same local PouchDB Chunk split. A smaller average changed Chunk can reduce
retained history in either format. Adaptive adds authenticated routes and independently framed records, while Opaque
adds its own container, compression, and encryption overhead. No comparative storage saving is assumed before both
representations are measured with the same changes.
### Ordinary request model
The object-profile model is shared with S3. Let:
- `B` be the number of Metadata batches published as distinct Commit Bundles;
- `X` be the number of new external Pack objects needed by those batches;
- `J` be the number of Opaque Journal objects produced for the same changes;
- `W` be the number of visible Writer streams;
- `K` be the number of those Writer descriptors not already cached by the opened process;
- `P` be the number of additional complete Pack-container reads needed for missing Chunks; and
- `M` be the number of missing Chunk frames fetched with Range.
After the endpoint safety result and repository binding have been cached for the active configuration, ordinary
publication performs approximately `J` Opaque PUTs or `B + X` Adaptive PUTs. Every Adaptive batch creates one Bundle;
each external Pack adds one preceding PUT. A confirmed conditional create does not add a read-back. An ambiguous PUT
adds one exact-key GET before any retry and remains outside the successful-first-attempt estimate.
The generic object-store receive path performs approximately:
| Retrieval policy | Requests without WebDAV listing reuse |
| --- | ---: |
| `whole-pack` | `1 + W + K + B + P` |
| `range` | `1 + W + K + B + M` |
The `1 + W` term represents one Writer listing and one Commit listing per Writer. A flat WebDAV collection normally
cannot apply those logical prefixes on the server: each call may require another Depth-one `PROPFIND` over the same
collection. The WebDAV target is therefore one complete listing at the start of a receive phase, filtered locally for
every Writer and Commit prefix in that phase. This receive-phase listing must expire before the next receive phase; it
must never become an unbounded cross-synchronisation cache. A Commit which becomes visible after the listing response
is handled by the next receive phase, which is consistent with immutable eventual replication.
With that explicitly scoped reuse, the request estimates become `1 + K + B + P` and `1 + K + B + M`. For 100 new
inline-Pack Bundles which require no additional Pack read, the difference is illustrative rather than a benchmark:
| Visible Writers | Descriptor state | Repeated-prefix listing | One receive-phase listing |
| ---: | --- | ---: | ---: |
| 1 | first receive (`K = 1`) | 103 | 102 |
| 1 | warm process (`K = 0`) | 102 | 101 |
| 10 | first receive (`K = 10`) | 121 | 111 |
| 10 | warm process (`K = 0`) | 111 | 101 |
Opaque catch-up is approximately one listing plus one GET per new Journal object, or `1 + J`. Holding `B = J = 100`
would therefore give 101 operations. That comparison does not assume that the two batchers produce equal object counts
for a real editing history.
### Listing-volume hypothesis
Request count alone understates WebDAV discovery cost. Let `N` be the number of resources in the configured flat
collection and `L` the average XML response bytes per resource for a minimal Depth-one `PROPFIND`. Repeating the
listing for each logical prefix transfers roughly `(1 + W) * N * L`; one receive-phase listing transfers roughly
`N * L`.
Assuming `L = 1 KiB` only to make the scale visible gives:
| Collection resources (`N`) | One listing | Repeated listing, `W = 1` | Repeated listing, `W = 10` |
| ---: | ---: | ---: | ---: |
| 1,000 | about 1 MiB | about 2 MiB | about 11 MiB |
| 10,000 | about 10 MiB | about 20 MiB | about 107 MiB |
| 100,000 | about 98 MiB | about 195 MiB | about 1.05 GiB |
Actual XML size, selected properties, URL length, compression, server implementation, and collection scope can change
these values materially. The adapter should request only the properties it needs and measure both response bytes and
wall time. The table explains the reuse target; it is not an estimate of every WebDAV server.
Ignoring cleaned probe resources and abandoned Packs, the current Commit Bundle layout contains approximately
`1 + W + B + X` remote objects: one manifest, one descriptor per Writer, one object per Bundle, and one per external
Pack. Object count therefore grows with synchronisation batches even when the retained payload is small, and it feeds
back into subsequent `PROPFIND` cost.
### Retained-byte hypothesis
For a localised edit history, let `E` be the number of separately published changes and `D` the average bytes of newly
created encoded Chunk frames per change. Before Garbage Collection or Rebuild, retained Chunk bytes grow approximately
as `E * D`, plus Metadata, Commit, route, frame, server-metadata, and abandoned-publication overhead.
For a 10 MiB incompressible binary file whose content-defined split resumes after one changed Chunk:
| Published changes (`E`) | Average changed Chunk | Approximate new Chunk bytes retained | New Bundle objects when each Pack stays inline |
| ---: | ---: | ---: | ---: |
| 100 | 1 MiB | 100 MiB | 100 |
| 1,000 | 1 MiB | about 0.98 GiB | 1,000 |
| 100 | 256 KiB | 25 MiB | 100 |
| 1,000 | 256 KiB | 250 MiB | 1,000 |
These figures exclude Metadata and protocol overhead and do not predict text splitting. They show why a 10 MiB live
file can consume much more than 10 MiB remotely after many committed edits. They also show why reducing Chunk size is a
storage-versus-record-count decision shared with Opaque Journal, not an Adaptive-only recommendation.
### Pack-target hypothesis
Changing the external Pack target does not normally duplicate Chunk frames: a Pack is their concatenation without
capacity padding. Smaller targets chiefly increase object count, route data, PUTs, listing work, and later whole-Pack
GETs. Larger targets increase peak buffer size, retry cost, transfer duration, and exposure to mobile watchdog or proxy
timeouts.
For about 100 MiB of newly required encoded Chunk frames in one batch, the following serial-transfer model ignores RTT,
TLS setup, encryption, server processing, contention, and retries:
| Pack target | External Packs (`X`) | PUTs including the Bundle | One target-sized upload at 5 / 10 Mbit/s |
| ---: | ---: | ---: | ---: |
| 8 MiB | 13 | 14 | about 13.4 / 6.7 seconds |
| 16 MiB | 7 | 8 | about 26.8 / 13.4 seconds |
| 32 MiB | 4 | 5 | about 53.7 / 26.8 seconds |
| 64 MiB | 2 | 3 | about 107.4 / 53.7 seconds |
The total Chunk payload remains about 100 MiB in each row. The initial WebDAV experiment therefore retains the current
32 MiB preferred target as a compromise, not as a universal optimum or a user-facing recommendation. Measurements on
the disposable server must record peak memory, per-PUT duration, retry behaviour, total object count, and watchdog
survival. A later decision may lower the internal target for timeout-constrained endpoints without changing the wire
format.
## Staged acceptance
### Adapter and Commonlib integration
Unit tests own HTTP status classification, conditional-create verification, flat-name round trips, complete listing,
probe isolation, cleanup reporting, Range validation, and whole-pack fallback. A disposable WebDAV integration runs
the safety checker before exercising Adaptive Metadata and Chunk synchronisation. The integration also proves that one
receive phase reuses a single collection listing, then invalidates it so the next phase can observe a new Commit.
### CLI end-to-end acceptance
The built CLI applies an Adaptive WebDAV Setup URI to a second independent database and synchronises text and binary
Chunk-backed files through a real disposable server. One client uses whole-pack retrieval; Range is added to this layer
only when the selected test server proves it. The CLI test does not repeat the complete endpoint capability matrix.
### Host settings and UI
The WebDAV dialogue exposes Adaptive mode only with clear capability-check results. It persists the expected repository
ID and the selected retrieval policy, defaults to whole-pack, and reports that Range is optional. Focused tests own
profile and Setup URI preservation without contacting a server.
### Real-host end-to-end acceptance
One real-Obsidian workflow uses the same known disposable WebDAV implementation, runs the safety gate, and proves a
representative Chunk-backed transfer. Servers outside that fixture are diagnosed by the checker rather than added to a
large real-host matrix.
## Alternatives rejected
### Trust advertised WebDAV methods
Method advertisement does not prove the conditional, listing, visibility, and byte semantics required for immutable
publication.
### Treat Range as mandatory
Whole-pack retrieval is correct and often competitive for throughput. Requiring Range would exclude otherwise safe
servers for an optional optimisation.
### Add server-specific compatibility branches
The remote may be any implementation or proxy composition. Semantic checks produce a maintainable contract, while a
growing server-name table would remain incomplete and become stale.
## Consequences
- WebDAV remains experimental because suitability is endpoint-specific.
- The safety checker gives a concrete reason when a server cannot host Adaptive Journal.
- Common object-pack behaviour is inherited from the earlier S3 boundary, keeping WebDAV-specific tests focused on HTTP
semantics and name mapping.
- Optional Range support can improve request efficiency without becoming a data-availability requirement.

File diff suppressed because it is too large Load Diff

View File

@@ -301,6 +301,18 @@ Setting key: bucketCustomHeaders
Custom HTTP headers to include in every request sent to the Object Storage bucket. Specify them in the format `Header-Name: Value`, with each header on a new line.
#### Journal data format
Setting key: journalFormat
Existing Object Storage profiles use `opaque-v1` unless Adaptive Journal is selected explicitly. `adaptive-v1` uses an authenticated manifest and immutable Commit Bundles under a separate namespace, with larger Packs stored separately. Changing between formats requires an explicit remote Rebuild; LiveSync does not migrate or read both representations.
#### Pack retrieval
Setting key: packReadPolicy
Adaptive Journal can download a complete immutable Pack (`whole-pack`) or request only the required byte ranges (`range`). Complete Pack retrieval is the portable, throughput-oriented default. Selecting Range requires exact byte-range support from the configured S3-compatible endpoint. The connection test verifies that capability, and synchronisation refuses an unsupported selection before writing.
#### Test Connection
#### Apply Settings

View File

@@ -45,6 +45,7 @@
"test:e2e:cli": "npm run test:e2e:ci --workspace self-hosted-livesync-cli",
"test:e2e:cli:p2p": "npm run test:e2e:p2p --workspace self-hosted-livesync-cli",
"test:e2e:cli:adaptive-s3": "npm run test:e2e:adaptive-s3 --workspace self-hosted-livesync-cli",
"test:e2e:cli:adaptive-webdav": "npm run test:e2e:adaptive-webdav --workspace self-hosted-livesync-cli",
"test:e2e:cli:all": "npm run test:e2e:all --workspace self-hosted-livesync-cli",
"test:integration": "npx dotenv-cli -e .env -e .test.env -- vitest run --config vitest.config.integration.ts",
"test:unit:coverage": "vitest run --config vitest.config.unit.ts --coverage",
@@ -68,6 +69,7 @@
"test:e2e:obsidian:couchdb-manual-setup-workflow": "tsx test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts",
"test:e2e:obsidian:cli-to-obsidian-sync": "tsx test/e2e-obsidian/scripts/cli-to-obsidian-sync.ts",
"test:e2e:obsidian:minio-upload": "tsx test/e2e-obsidian/scripts/minio-upload.ts",
"test:e2e:obsidian:adaptive-s3": "tsx test/e2e-obsidian/scripts/minio-upload.ts --adaptive",
"test:e2e:obsidian:object-storage-setup-uri-workflow": "tsx test/e2e-obsidian/scripts/object-storage-setup-uri-workflow.ts",
"test:e2e:obsidian:p2p-setup-uri-workflow": "tsx test/e2e-obsidian/scripts/p2p-setup-uri-workflow.ts",
"pretest:e2e:obsidian:p2p-connection-check": "npm run build && npm run build --workspace webpeer",

View File

@@ -21,7 +21,7 @@ import { AbstractModule } from "./modules/AbstractModule";
import { ModulePeriodicProcess } from "./modules/core/ModulePeriodicProcess";
import { ModuleReplicator } from "./modules/core/ModuleReplicator";
import { ModuleReplicatorCouchDB } from "./modules/core/ModuleReplicatorCouchDB";
import { ModuleReplicatorMinIO } from "./modules/core/ModuleReplicatorMinIO";
import { ModuleReplicatorJournal } from "./modules/core/ModuleReplicatorJournal";
import { ModuleConflictChecker } from "./modules/coreFeatures/ModuleConflictChecker";
import { ModuleConflictResolver } from "./modules/coreFeatures/ModuleConflictResolver";
import { ModuleResolvingMismatchedTweaks } from "./modules/coreFeatures/ModuleResolveMismatchedTweaks";
@@ -139,7 +139,7 @@ export class LiveSyncBaseCore<
public registerModules(extraModules: AbstractModule[] = []) {
this._registerModule(new ModuleLiveSyncMain(this));
this._registerModule(new ModuleConflictChecker(this));
this._registerModule(new ModuleReplicatorMinIO(this));
this._registerModule(new ModuleReplicatorJournal(this));
this._registerModule(new ModuleReplicatorCouchDB(this));
this._registerModule(new ModuleReplicator(this));
this._registerModule(new ModuleConflictResolver(this));

View File

@@ -6,7 +6,7 @@ import {
type FilePathWithPrefix,
type ObsidianLiveSyncSettings,
REMOTE_COUCHDB,
REMOTE_MINIO,
isJournalRemoteType,
type EntryMilestoneInfo,
type EntryDoc,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
@@ -61,7 +61,7 @@ async function verifyRemoteState(
return false;
}
milestone = await dbRet.db.get(MILESTONE_DOCID);
} else if (settings.remoteType === REMOTE_MINIO) {
} else if (isJournalRemoteType(settings.remoteType)) {
const journalReplicator = replicator as LiveSyncJournalReplicator;
if (journalProtocolConfigurationForSettings(settings).journalFormat === "adaptive-v1") {
try {

View File

@@ -7,6 +7,7 @@ import {
REMOTE_COUCHDB,
REMOTE_MINIO,
REMOTE_P2P,
REMOTE_WEBDAV,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { describe, expect, it, vi, beforeEach, afterEach } from "vitest";
import { runCommand } from "./runCommand";
@@ -750,6 +751,32 @@ describe("runCommand abnormal cases", () => {
);
});
it("uses the Adaptive verification path for a WebDAV remote", async () => {
const core = createCoreMock();
const settings = core.services.setting.currentSettings();
settings.remoteType = REMOTE_WEBDAV;
settings.webDAVactiveConnectionURI = "sls+webdav://dav.example/dav";
settings.journalFormat = "adaptive-v1";
settings.packReadPolicy = "whole-pack";
const ensureCheckpointCachesAreFresh = vi.fn(async () => {});
core.services.replicator.getActiveReplicator.mockReturnValue({
nodeid: "test-node-id",
initializeDatabaseForReplication: vi.fn(async () => {}),
client: {
ensureCheckpointCachesAreFresh,
},
});
const result = await runCommand(makeOptions("mark-resolved", []), {
...context,
core,
});
expect(result).toBe(true);
expect(ensureCheckpointCachesAreFresh).toHaveBeenCalledTimes(1);
});
it("mark-resolved with remote-id temporarily activates it and runs markResolved", async () => {
const core = createCoreMock();
const settings = core.services.setting.currentSettings();

View File

@@ -37,6 +37,14 @@ defaultLoggerEnv.minLogLevel = LOG_LEVEL_DEBUG;
/** Injectable command boundary used by CLI integration probes. */
export type CliCommandRunner = (options: CLIOptions, context: CLICommandContext) => Promise<boolean>;
const SETTINGS_MANAGEMENT_COMMANDS: ReadonlySet<CLICommand> = new Set([
"setup",
"remote-add",
"remote-rm",
"remote-set",
"remote-activate",
]);
function printHelp(standardIo: StandardIo): void {
writeStdoutLine(
standardIo,
@@ -274,7 +282,10 @@ export async function main(
) {
const options = parseArgs(standardIo);
if (options.interval && options.command !== "daemon") {
writeStderrLine(standardIo, `Warning: --interval is only used in daemon mode, ignored for '${options.command}'`);
writeStderrLine(
standardIo,
`Warning: --interval is only used in daemon mode, ignored for '${options.command}'`
);
}
const avoidStdoutNoise =
options.command === "cat" ||
@@ -399,12 +410,15 @@ export async function main(
if (!options.verbose) return;
}
writeStderrLine(standardIo, prefix, message);
});
}, true);
// Prevent replication result from being processed automatically in non-daemon commands.
// In daemon mode the default handler must run so changes are applied to the filesystem.
if (options.command !== "daemon") {
serviceHubInstance.replication.processSynchroniseResult.addHandler(async () => {
writeStderrLine(standardIo, `[Info] Replication result received, but not processed automatically in CLI mode.`);
writeStderrLine(
standardIo,
`[Info] Replication result received, but not processed automatically in CLI mode.`
);
return await Promise.resolve(true);
}, -100);
}
@@ -506,7 +520,7 @@ export async function main(
// Save the settings file before any lifecycle events can mutate and persist them.
// suspendAllSync and other lifecycle hooks clobber sync settings in memory, and
// various code paths persist the clobbered state to disk. We restore on shutdown.
const settingsBackup = await fs.readFile(settingsPath, "utf-8").catch(() => null!);
let settingsBackup: string | null = await fs.readFile(settingsPath, "utf-8").catch(() => null);
// Restore settings file on any exit to undo lifecycle mutations.
// Write to a temp path first so a crash mid-write doesn't leave a truncated file.
@@ -576,6 +590,15 @@ export async function main(
settingsPath,
originalSyncSettings,
});
if (result && SETTINGS_MANAGEMENT_COMMANDS.has(options.command)) {
// Settings management is intentional, unlike the temporary changes made by suspendAllSync().
// setup replaces the complete configuration, while remote profile commands must retain the
// synchronisation mode which was active before the command lifecycle suspended it.
if (options.command !== "setup") {
await core.services.setting.applyPartial(originalSyncSettings, true);
}
settingsBackup = await fs.readFile(settingsPath, "utf-8");
}
if (!result) {
writeStderrLine(standardIo, `[Error] Command '${options.command}' failed`);
process.exitCode = 1;

View File

@@ -24,6 +24,8 @@
"test:e2e:p2p": "deno task --cwd testdeno test:p2p:compose",
"pretest:e2e:adaptive-s3": "npm run build",
"test:e2e:adaptive-s3": "deno task --cwd testdeno test:adaptive-journal-s3",
"pretest:e2e:adaptive-webdav": "npm run build",
"test:e2e:adaptive-webdav": "deno task --cwd testdeno test:adaptive-journal-webdav",
"test:e2e:mirror": "bash test/test-mirror-linux.sh",
"test:e2e:remote-commands": "bash test/test-remote-commands-linux.sh",
"pretest:e2e:all": "npm run build",

View File

@@ -35,7 +35,8 @@
"test:e2e-matrix:couchdb-enc1": "deno test --env-file=.test.env -A --no-check --filter='e2e matrix: COUCHDB-enc1' test-e2e-two-vaults-matrix.ts",
"test:e2e-matrix:minio-enc0": "deno test --env-file=.test.env -A --no-check --filter='e2e matrix: MINIO-enc0' test-e2e-two-vaults-matrix.ts",
"test:e2e-matrix:minio-enc1": "deno test --env-file=.test.env -A --no-check --filter='e2e matrix: MINIO-enc1' test-e2e-two-vaults-matrix.ts",
"test:adaptive-journal-s3": "deno test --env-file=.test.env -A --no-check test-adaptive-journal-s3.ts"
"test:adaptive-journal-s3": "deno test --env-file=.test.env -A --no-check test-adaptive-journal-s3.ts",
"test:adaptive-journal-webdav": "deno test --env-file=.test.env -A --no-check test-adaptive-journal-webdav.ts"
},
"imports": {
"@std/assert": "jsr:@std/assert@^1.0.13",

View File

@@ -23,6 +23,7 @@ export interface CliResult {
export const TEE_ENABLED = Deno.env.get("LIVESYNC_TEST_TEE") === "1";
const VERBOSE_ENABLED = Deno.env.get("LIVESYNC_CLI_VERBOSE") === "1";
const DEBUG_ENABLED = Deno.env.get("LIVESYNC_CLI_DEBUG") === "1";
const SETUP_URI_PREFIX = "obsidian://setuplivesync?settings=";
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
@@ -40,7 +41,13 @@ function concatChunks(chunks: Uint8Array[]): Uint8Array {
}
export function formatTeeCommand(args: string[]): string {
return ["node", CLI_DIST, ...args].map((part) => JSON.stringify(part)).join(" ");
const redactArgument = (argument: string): string => {
if (argument.startsWith(SETUP_URI_PREFIX)) {
return `${SETUP_URI_PREFIX}<redacted>`;
}
return argument.replace(/^(sls\+[^:]+:\/\/)[^/?#@]*@/u, "$1<redacted>@");
};
return ["node", CLI_DIST, ...args.map(redactArgument)].map((part) => JSON.stringify(part)).join(" ");
}
export function createLineTeeWriter(

View File

@@ -330,6 +330,39 @@ const MINIO_CONTAINER = "minio-test";
const MINIO_IMAGE = "minio/minio";
const MINIO_MC_IMAGE = "minio/mc";
const WEBDAV_CONTAINER = "webdav-test";
const WEBDAV_IMAGE = "httpd:2.4.68";
const WEBDAV_HTTPD_CONFIG = `ServerRoot "/usr/local/apache2"
Listen 80
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule unixd_module modules/mod_unixd.so
User www-data
Group www-data
ServerName localhost
DocumentRoot "/usr/local/apache2/htdocs"
PidFile "/tmp/httpd.pid"
ErrorLog "/proc/self/fd/2"
LogLevel warn
DavLockDB "/usr/local/apache2/var/DavLock"
DavLockDiscovery Off
<Directory "/usr/local/apache2/htdocs">
AllowOverride None
Require all granted
</Directory>
<Directory "/usr/local/apache2/htdocs/dav">
Dav On
</Directory>
`;
export async function stopCouchdb(): Promise<void> {
await stopAndRemoveContainer(COUCHDB_CONTAINER);
untrackContainer(COUCHDB_CONTAINER);
@@ -625,6 +658,113 @@ export async function startMinio(
await waitForMinioBucket(minioEndpoint, accessKey, secretKey, bucket);
}
// ---------------------------------------------------------------------------
// WebDAV
// ---------------------------------------------------------------------------
export async function stopWebDAV(): Promise<void> {
await stopAndRemoveContainer(WEBDAV_CONTAINER);
untrackContainer(WEBDAV_CONTAINER);
}
async function waitForWebDAV(endpoint: string): Promise<void> {
for (let attempt = 0; attempt < 30; attempt += 1) {
try {
const response = await fetch(endpoint, {
method: "PROPFIND",
headers: { Depth: "0" },
signal: AbortSignal.timeout(3000),
});
await response.body?.cancel().catch(() => {});
if (response.status === 207) return;
} catch {
// The container is still starting.
}
await sleep(500);
}
throw new Error(`WebDAV collection did not become ready: ${endpoint}`);
}
export async function startWebDAV(endpoint: string): Promise<void> {
const url = new URL(endpoint);
if (url.protocol !== "http:" || (url.hostname !== "127.0.0.1" && url.hostname !== "localhost")) {
throw new Error(`Managed WebDAV requires a local HTTP endpoint, received: ${endpoint}`);
}
if (url.pathname.replace(/\/+$/u, "") !== "/dav") {
throw new Error(`Managed WebDAV requires the /dav collection, received: ${endpoint}`);
}
const hostPort = url.port || "80";
const encodedConfig = btoa(WEBDAV_HTTPD_CONFIG);
const startCommand = `printf '%s' '${encodedConfig}' | base64 -d > /tmp/httpd.conf && exec httpd -DFOREGROUND -f /tmp/httpd.conf`;
console.log("[INFO] stopping leftover WebDAV container if present");
await stopWebDAV().catch(() => {});
console.log("[INFO] starting Apache WebDAV test container");
await dockerOrFail(
"run",
"-d",
"--name",
WEBDAV_CONTAINER,
"-p",
`${hostPort}:80`,
"--tmpfs",
"/usr/local/apache2/htdocs/dav:mode=0777",
"--tmpfs",
"/usr/local/apache2/var:mode=0777",
"--entrypoint",
"/bin/sh",
WEBDAV_IMAGE,
"-c",
startCommand
);
trackContainer(WEBDAV_CONTAINER);
await waitForWebDAV(endpoint);
}
function directWebDAVObjectUrl(collectionEndpoint: string, key: string): string {
return `${collectionEndpoint.replace(/\/+$/u, "")}/${encodeURIComponent(key)}`;
}
function webDAVRequestHeaders(): HeadersInit {
const username = Deno.env.get("WEBDAV_USERNAME") ?? "";
const password = Deno.env.get("WEBDAV_PASSWORD") ?? "";
if (!username && !password) return {};
return { Authorization: `Basic ${btoa(`${username}:${password}`)}` };
}
export async function listWebDAVObjectKeys(collectionEndpoint: string): Promise<string[]> {
const collectionUrl = new URL(`${collectionEndpoint.replace(/\/+$/u, "")}/`);
const response = await fetch(collectionUrl, {
method: "PROPFIND",
headers: { ...webDAVRequestHeaders(), Depth: "1" },
});
if (response.status !== 207) {
throw new Error(`Could not list WebDAV objects: HTTP ${response.status}`);
}
const xml = await response.text();
const hrefs = [
...xml.matchAll(/<(?:[A-Za-z_][\w.-]*:)?href\b[^>]*>([\s\S]*?)<\/(?:[A-Za-z_][\w.-]*:)?href>/giu),
].map((match) => match[1].trim());
const basePath = decodeURIComponent(collectionUrl.pathname);
const keys = new Set<string>();
for (const href of hrefs) {
const path = decodeURIComponent(new URL(href, collectionUrl).pathname);
if (!path.startsWith(basePath)) continue;
const key = path.slice(basePath.length).replace(/\/$/u, "");
if (key && !key.includes("/")) keys.add(key);
}
return [...keys].sort();
}
export async function readWebDAVObjectText(collectionEndpoint: string, key: string): Promise<string> {
const response = await fetch(directWebDAVObjectUrl(collectionEndpoint, key), { headers: webDAVRequestHeaders() });
if (!response.ok) {
throw new Error(`Could not read WebDAV object ${key}: HTTP ${response.status}`);
}
return await response.text();
}
// ---------------------------------------------------------------------------
// P2P relay (strfry)
// ---------------------------------------------------------------------------

View File

@@ -13,7 +13,12 @@ export async function initSettingsFile(settingsFile: string): Promise<void> {
* Generate a full setup URI from a settings file via the Commonlib package API.
* Mirrors the bash flow in test-setup-put-cat-linux.sh.
*/
export async function generateSetupUriFromSettings(settingsFile: string, setupPassphrase: string): Promise<string> {
export async function generateSetupUriFromSettings(
settingsFile: string,
setupPassphrase: string,
preserveSettings = false,
runtimeVaultPassphrase?: string
): Promise<string> {
const script = [
"import { fs } from '@vrtmrz/livesync-commonlib/node';",
"import { encodeSettingsToSetupURI } from '@vrtmrz/livesync-commonlib/compat/API/processSetting';",
@@ -21,13 +26,18 @@ export async function generateSetupUriFromSettings(settingsFile: string, setupPa
" const settingsPath = process.env.SETTINGS_FILE;",
" const passphrase = process.env.SETUP_PASSPHRASE;",
" const settings = JSON.parse(fs.readFileSync(settingsPath, 'utf-8'));",
" settings.couchDB_DBNAME = 'setup-put-cat-db';",
" settings.couchDB_URI = 'http://127.0.0.1:5999';",
" settings.couchDB_USER = 'dummy';",
" settings.couchDB_PASSWORD = 'dummy';",
" settings.liveSync = false;",
" settings.syncOnStart = false;",
" settings.syncOnSave = false;",
" if (process.env.RUNTIME_VAULT_PASSPHRASE !== undefined) {",
" settings.passphrase = process.env.RUNTIME_VAULT_PASSPHRASE;",
" }",
" if (process.env.PRESERVE_SETTINGS !== 'true') {",
" settings.couchDB_DBNAME = 'setup-put-cat-db';",
" settings.couchDB_URI = 'http://127.0.0.1:5999';",
" settings.couchDB_USER = 'dummy';",
" settings.couchDB_PASSWORD = 'dummy';",
" settings.liveSync = false;",
" settings.syncOnStart = false;",
" settings.syncOnSave = false;",
" }",
" const uri = await encodeSettingsToSetupURI(settings, passphrase);",
" process.stdout.write(uri.trim());",
"})();",
@@ -41,13 +51,18 @@ export async function generateSetupUriFromSettings(settingsFile: string, setupPa
await Deno.writeTextFile(scriptPath, script);
try {
const env: Record<string, string> = {
SETTINGS_FILE: settingsFile,
SETUP_PASSPHRASE: setupPassphrase,
PRESERVE_SETTINGS: preserveSettings ? "true" : "false",
};
if (runtimeVaultPassphrase !== undefined) {
env.RUNTIME_VAULT_PASSPHRASE = runtimeVaultPassphrase;
}
const cmd = new Deno.Command("npx", {
args: ["tsx", scriptPath],
cwd: CLI_DIR,
env: {
SETTINGS_FILE: settingsFile,
SETUP_PASSPHRASE: setupPassphrase,
},
env,
stdin: "null",
stdout: "piped",
stderr: "piped",
@@ -112,7 +127,7 @@ export async function applyCouchdbSettings(
export async function applyRemoteSyncSettings(
settingsFile: string,
options: {
remoteType: "COUCHDB" | "MINIO";
remoteType: "COUCHDB" | "MINIO" | "WEBDAV";
couchdbUri?: string;
couchdbUser?: string;
couchdbPassword?: string;
@@ -121,6 +136,7 @@ export async function applyRemoteSyncSettings(
minioEndpoint?: string;
minioAccessKey?: string;
minioSecretKey?: string;
webDAVConnectionURI?: string;
encrypt?: boolean;
passphrase?: string;
enableCompression?: boolean;
@@ -138,7 +154,7 @@ export async function applyRemoteSyncSettings(
data.couchDB_USER = options.couchdbUser;
data.couchDB_PASSWORD = options.couchdbPassword;
data.couchDB_DBNAME = options.couchdbDbname;
} else {
} else if (options.remoteType === "MINIO") {
data.remoteType = "MINIO";
data.bucket = options.minioBucket;
data.endpoint = options.minioEndpoint;
@@ -146,15 +162,18 @@ export async function applyRemoteSyncSettings(
data.secretKey = options.minioSecretKey;
data.region = "auto";
data.forcePathStyle = true;
if (options.journalFormat !== undefined) {
data.journalFormat = options.journalFormat;
}
if (options.expectedRepositoryId !== undefined) {
data.expectedRepositoryId = options.expectedRepositoryId;
}
if (options.packReadPolicy !== undefined) {
data.packReadPolicy = options.packReadPolicy;
}
} else {
data.remoteType = "WEBDAV";
data.webDAVactiveConnectionURI = options.webDAVConnectionURI;
}
if (options.journalFormat !== undefined) {
data.journalFormat = options.journalFormat;
}
if (options.expectedRepositoryId !== undefined) {
data.expectedRepositoryId = options.expectedRepositoryId;
}
if (options.packReadPolicy !== undefined) {
data.packReadPolicy = options.packReadPolicy;
}
data.liveSync = true;

View File

@@ -12,6 +12,7 @@ const TASKS = [
"test:e2e-matrix:minio-enc0",
"test:e2e-matrix:minio-enc1",
"test:adaptive-journal-s3",
"test:adaptive-journal-webdav",
] as const;
for (const [index, task] of TASKS.entries()) {

View File

@@ -0,0 +1,215 @@
import { assert, assertEquals } from "@std/assert";
import { TempDir } from "./helpers/temp.ts";
import { assertFilesEqual, runCli, runCliOrFail, runCliWithInputOrFail, sanitiseCatStdout } from "./helpers/cli.ts";
import { applyRemoteSyncSettings, generateSetupUriFromSettings, initSettingsFile } from "./helpers/settings.ts";
import { listWebDAVObjectKeys, readWebDAVObjectText, startWebDAV, stopWebDAV } from "./helpers/docker.ts";
const EXTERNAL_PACK_TEST_BYTES = 9 * 1024 * 1024;
function deterministicBytes(length: number, seed: number): Uint8Array {
const bytes = new Uint8Array(length);
let state = seed;
for (let index = 0; index < bytes.byteLength; index += 1) {
state ^= state << 13;
state ^= state >>> 17;
state ^= state << 5;
bytes[index] = state & 0xff;
}
return bytes;
}
function webDAVConnectionURI(endpoint: string, prefix: string): string {
const endpointUrl = new URL(endpoint);
const proxyUrl = new URL(`https://${endpointUrl.host}${endpointUrl.pathname}`);
const username = Deno.env.get("WEBDAV_USERNAME") ?? "";
const password = Deno.env.get("WEBDAV_PASSWORD") ?? "";
proxyUrl.username = username;
proxyUrl.password = password;
if (endpointUrl.protocol === "http:") proxyUrl.searchParams.set("insecure", "true");
proxyUrl.searchParams.set("prefix", prefix);
return `sls+webdav:${proxyUrl.toString().slice("https:".length)}`;
}
function setPackReadPolicy(connectionURI: string, policy: "range" | "whole-pack"): string {
const url = new URL(connectionURI);
url.searchParams.set("packReadPolicy", policy);
return url.toString();
}
function selectAdaptiveJournal(connectionURI: string): string {
const url = new URL(connectionURI);
url.searchParams.set("journalFormat", "adaptive-v1");
return url.toString();
}
function remoteIdFromListing(listing: string): string {
const line = listing
.split(/\r?\n/u)
.find((candidate) => candidate.includes("\tWebDAV Remote\t") || candidate.includes("\tWebDAV "));
const id = line?.split("\t", 1)[0];
if (!id) throw new Error(`WebDAV remote profile was not listed:\n${listing}`);
return id;
}
Deno.test("e2e: two CLI vaults synchronise through Adaptive Journal WebDAV", async () => {
const suffix = `${Date.now()}-${Math.floor(Math.random() * 100000)}`;
const endpoint = (Deno.env.get("WEBDAV_ENDPOINT") ?? "http://127.0.0.1:8088/dav").replace(/\/+$/u, "");
const prefix = `adaptive-cli-${suffix}/`;
const collectionEndpoint = `${endpoint}/${prefix}`;
const connectionURI = webDAVConnectionURI(endpoint, prefix);
const adaptiveConnectionURI = selectAdaptiveJournal(connectionURI);
const vaultPassphrase = "adaptive-journal-webdav-cli-e2ee";
const setupPassphrase = "adaptive-journal-webdav-cli-setup";
await using workDir = await TempDir.create("livesync-cli-adaptive-journal-webdav");
const vaultA = workDir.join("vault-a");
const vaultB = workDir.join("vault-b");
const settingsA = workDir.join("settings-a.json");
const settingsB = workDir.join("settings-b.json");
const binarySourceA = workDir.join("source-a.bin");
const binarySourceB = workDir.join("source-b.bin");
const binaryDestinationA = workDir.join("destination-a.bin");
const binaryDestinationB = workDir.join("destination-b.bin");
await Deno.mkdir(vaultA, { recursive: true });
await Deno.mkdir(vaultB, { recursive: true });
const shouldStartDocker = Deno.env.get("LIVESYNC_START_DOCKER") !== "0";
const keepDocker = Deno.env.get("LIVESYNC_DEBUG_KEEP_DOCKER") === "1";
if (shouldStartDocker) await startWebDAV(endpoint);
try {
await initSettingsFile(settingsA);
await applyRemoteSyncSettings(settingsA, {
remoteType: "WEBDAV",
webDAVConnectionURI: connectionURI,
encrypt: true,
passphrase: vaultPassphrase,
enableCompression: false,
journalFormat: "adaptive-v1",
packReadPolicy: "whole-pack",
});
const addedRemote = await runCliOrFail(
vaultA,
"--settings",
settingsA,
"remote-add",
"WebDAV E2E",
adaptiveConnectionURI
);
const remoteId = addedRemote.trim().split("\t", 1)[0];
assert(remoteId, `remote-add did not return a profile ID:\n${addedRemote}`);
const settingsAfterRemoteAdd = JSON.parse(await Deno.readTextFile(settingsA)) as {
remoteConfigurations?: Record<string, unknown>;
liveSync?: boolean;
};
assert(
settingsAfterRemoteAdd.remoteConfigurations?.[remoteId],
`remote-add did not persist profile ${remoteId}: ${Object.keys(settingsAfterRemoteAdd.remoteConfigurations ?? {}).join(", ")}`
);
assertEquals(settingsAfterRemoteAdd.liveSync, true, "remote-add changed the persisted synchronisation mode");
await runCliOrFail(vaultA, "--settings", settingsA, "remote-activate", remoteId);
const textPath = "adaptive/text.md";
const binaryPath = "adaptive/data.bin";
await runCliWithInputOrFail(`created-by-a-${suffix}\n`, vaultA, "--settings", settingsA, "put", textPath);
await Deno.writeFile(binarySourceA, deterministicBytes(EXTERNAL_PACK_TEST_BYTES, 0x1a2b3c4d));
await runCliOrFail(vaultA, "--settings", settingsA, "push", binarySourceA, binaryPath);
await runCliOrFail(vaultA, "--settings", settingsA, "sync");
const remoteListing = await runCliOrFail(vaultA, "--settings", settingsA, "remote-ls");
assertEquals(remoteIdFromListing(remoteListing), remoteId);
assert(
remoteListing
.split(/\r?\n/u)
.some((line) => line.startsWith(`${remoteId}\t`) && line.includes("\tactive\t")),
`Activated WebDAV profile was not listed as active:\n${remoteListing}`
);
const exportedConnection = (
await runCliOrFail(vaultA, "--settings", settingsA, "remote-export", remoteId)
).trim();
assert(exportedConnection.startsWith("sls+webdav://"));
assert(exportedConnection.includes("journalFormat=adaptive-v1"));
assert(!exportedConnection.includes("packReadPolicy="));
const setupURI = await generateSetupUriFromSettings(settingsA, setupPassphrase, true, vaultPassphrase);
await initSettingsFile(settingsB);
await runCliWithInputOrFail(`${setupPassphrase}\n`, vaultB, "--settings", settingsB, "setup", setupURI);
const settingsAfterSetup = JSON.parse(await Deno.readTextFile(settingsB)) as {
encryptedPassphrase?: string;
};
assert(
typeof settingsAfterSetup.encryptedPassphrase === "string" &&
settingsAfterSetup.encryptedPassphrase.length > 0,
"setup did not persist the encrypted Vault passphrase"
);
await runCliOrFail(
vaultB,
"--settings",
settingsB,
"remote-set",
remoteId,
setPackReadPolicy(exportedConnection, "range")
);
const rangeConnection = (await runCliOrFail(vaultB, "--settings", settingsB, "remote-export", remoteId)).trim();
assert(rangeConnection.includes("packReadPolicy=range"));
await runCliOrFail(vaultB, "--settings", settingsB, "sync");
assertEquals(
sanitiseCatStdout(await runCliOrFail(vaultB, "--settings", settingsB, "cat", textPath)).trimEnd(),
`created-by-a-${suffix}`
);
await runCliOrFail(vaultB, "--settings", settingsB, "pull", binaryPath, binaryDestinationB);
await assertFilesEqual(binarySourceA, binaryDestinationB, "Adaptive Journal Range transfer differs");
await runCliWithInputOrFail(`updated-by-b-${suffix}\n`, vaultB, "--settings", settingsB, "put", textPath);
await Deno.writeFile(binarySourceB, deterministicBytes(EXTERNAL_PACK_TEST_BYTES, 0x5e6f7788));
await runCliOrFail(vaultB, "--settings", settingsB, "push", binarySourceB, binaryPath);
await runCliOrFail(vaultB, "--settings", settingsB, "sync");
await runCliOrFail(vaultA, "--settings", settingsA, "sync");
assertEquals(
sanitiseCatStdout(await runCliOrFail(vaultA, "--settings", settingsA, "cat", textPath)).trimEnd(),
`updated-by-b-${suffix}`
);
await runCliOrFail(vaultA, "--settings", settingsA, "pull", binaryPath, binaryDestinationA);
await assertFilesEqual(binarySourceB, binaryDestinationA, "Adaptive Journal whole-Pack transfer differs");
await runCliOrFail(vaultA, "--settings", settingsA, "rm", binaryPath);
await runCliOrFail(vaultA, "--settings", settingsA, "sync");
await runCliOrFail(vaultB, "--settings", settingsB, "sync");
const deleted = await runCli(vaultB, "--settings", settingsB, "cat", binaryPath);
assert(deleted.code !== 0, `Deleted binary remained readable:\n${deleted.combined}`);
const objectKeys = await listWebDAVObjectKeys(collectionEndpoint);
assert(objectKeys.includes("a1~manifest.json"), `Adaptive manifest is missing:\n${objectKeys.join("\n")}`);
for (const objectPrefix of ["a1~writer~", "a1~pack~", "a1~commit~"]) {
assert(
objectKeys.some((key) => key.startsWith(objectPrefix)),
`Adaptive object with prefix ${objectPrefix} is missing:\n${objectKeys.join("\n")}`
);
}
const packKeys = objectKeys.filter((key) => key.startsWith("a1~pack~"));
assert(packKeys.length >= 2, `Expected external Packs from both CLI writers:\n${objectKeys.join("\n")}`);
for (const legacyPrefix of ["a1~index~", "a1~delta~", "a1~metadata~"]) {
assert(
!objectKeys.some((key) => key.startsWith(legacyPrefix)),
`Legacy Adaptive object with prefix ${legacyPrefix} was written:\n${objectKeys.join("\n")}`
);
}
const manifest = JSON.parse(await readWebDAVObjectText(collectionEndpoint, "a1~manifest.json")) as {
objectLayout?: unknown;
};
assertEquals(manifest.objectLayout, "commit-bundle-v1");
assert(
!objectKeys.some((key) => key.startsWith("a1~probe~")),
`Adaptive capability probe objects were not removed:\n${objectKeys.join("\n")}`
);
assert(
!objectKeys.includes("_00000000-milestone.json"),
`Legacy Journal milestone was written into the Adaptive repository:\n${objectKeys.join("\n")}`
);
} finally {
if (shouldStartDocker && !keepDocker) {
await stopWebDAV().catch(() => {});
}
}
});

View File

@@ -288,6 +288,13 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
zh: "生效中的远程配置",
"zh-tw": "目前啟用的遠端設定",
},
"Adaptive Journal (experimental)": {
def: "Adaptive Journal (experimental)",
},
"Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.":
{
def: "Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.",
},
"Add default patterns": {
def: "Add default patterns",
es: "Añadir patrones predeterminados",
@@ -784,6 +791,10 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
zh: "兼容性(问题修复)",
"zh-tw": "相容性(問題修復)",
},
"Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.":
{
def: "Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.",
},
"Compute revisions for chunks": {
def: "Compute revisions for chunks",
es: "Calcular revisiones para los chunks",
@@ -1585,6 +1596,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
ko: "문서 기록",
"zh-tw": "文件歷程",
},
"Download complete Packs": {
def: "Download complete Packs",
},
Duplicate: {
def: "Duplicate",
es: "Duplicar",
@@ -2605,6 +2619,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
ru: "Интервал (сек)",
zh: "间隔(秒)",
},
"Invalid Object Storage settings: ${reason}": {
def: "Invalid Object Storage settings: ${reason}",
},
INVERTED: {
def: "INVERTED",
es: "INVERTIDO",
@@ -2617,6 +2634,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
def: "It is strongly advised to create a backup before proceeding. Continuing without a backup may lead to data loss.",
es: "Se recomienda encarecidamente crear una copia de seguridad antes de continuar. Continuar sin copia de seguridad puede provocar pérdida de datos.",
},
"Journal data format": {
def: "Journal data format",
},
"Just for a minute, please!": {
def: "Just for a minute, please!",
es: "¡Solo un momento, por favor!",
@@ -6187,6 +6207,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
def: "On this device, switch to the camera app or use a QR code scanner to scan the displayed QR code.",
es: "En este dispositivo, cambia a la aplicación de cámara o usa un lector de QR para escanear el código mostrado.",
},
"Opaque Journal (current format)": {
def: "Opaque Journal (current format)",
},
Open: {
def: "Open",
es: "Abrir",
@@ -6447,6 +6470,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
ru: "P2P Sync с name начат.",
zh: "P2P Sync with ${name} have been started.",
},
"Pack retrieval": {
def: "Pack retrieval",
},
"paneMaintenance.markDeviceResolvedAfterBackup": {
def: "paneMaintenance.markDeviceResolvedAfterBackup",
es: "Marcar el dispositivo como resuelto después de hacer una copia de seguridad",
@@ -11069,6 +11095,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
def: "Use Random Number",
es: "Usar número aleatorio",
},
"Use S3 Range requests": {
def: "Use S3 Range requests",
},
"Use Segmented-splitter": {
def: "Use Segmented-splitter",
es: "Usar divisor segmentado",

View File

@@ -44,6 +44,8 @@
"Action": "Action",
"Activate": "Activate",
"Active Remote Configuration": "Active Remote Configuration",
"Adaptive Journal (experimental)": "Adaptive Journal (experimental)",
"Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.": "Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.",
"Add default patterns": "Add default patterns",
"Add new connection": "Add new connection",
"AddOn Module (ConfigSync) has not been loaded. This is very unexpected situation. Please report this issue.": "AddOn Module (ConfigSync) has not been loaded. This is very unexpected situation. Please report this issue.",
@@ -112,6 +114,7 @@
"Compatibility (Metadata)": "Compatibility (Metadata)",
"Compatibility (Remote Database)": "Compatibility (Remote Database)",
"Compatibility (Trouble addressed)": "Compatibility (Trouble addressed)",
"Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.": "Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.",
"Compute revisions for chunks": "Compute revisions for chunks",
"Configuration": "Configuration",
"Configuration Encryption": "Configuration Encryption",
@@ -212,6 +215,7 @@
"Doctor.Message.SomeSkipped": "We left some issues as is. Shall I ask you again on next startup?",
"Doctor.RULES.E2EE_V02500.REASON": "The End-to-End Encryption has got now more robust and faster. Also because, the previous E2EE was found to be compromised in a re-conducted code review. It should be applied as soon as possible. Really apologises for your inconvenience. And, this setting is not forward compatible. All synchronised devices must be updated to v0.25.0 or higher. Rebuilds are not required and will be converted from the new transfer to the new format, However, it is recommended to rebuild whenever possible.",
"Document History": "Document History",
"Download complete Packs": "Download complete Packs",
"Duplicate": "Duplicate",
"Duplicate remote": "Duplicate remote",
"E2EE Configuration": "E2EE Configuration",
@@ -355,9 +359,11 @@
"Initialise journal received history. On the next sync, every item except this device sent will be downloaded again.": "Initialise journal received history. On the next sync, every item except this device sent will be downloaded again.",
"Initialise journal sent history. On the next sync, every item except this device received will be sent again.": "Initialise journal sent history. On the next sync, every item except this device received will be sent again.",
"Interval (sec)": "Interval (sec)",
"Invalid Object Storage settings: ${reason}": "Invalid Object Storage settings: ${reason}",
"INVERTED": "INVERTED",
"Issue detection log:": "Issue detection log:",
"It is strongly advised to create a backup before proceeding. Continuing without a backup may lead to data loss.": "It is strongly advised to create a backup before proceeding. Continuing without a backup may lead to data loss.",
"Journal data format": "Journal data format",
"Just for a minute, please!": "Just for a minute, please!",
"JWT (JSON Web Token) authentication allows you to securely authenticate with the CouchDB server using tokens. Ensure that your CouchDB server is configured to accept JWTs and that the provided key and settings match the server's configuration. Incidentally, I have not verified it very thoroughly.": "JWT (JSON Web Token) authentication allows you to securely authenticate with the CouchDB server using tokens. Ensure that your CouchDB server is configured to accept JWTs and that the provided key and settings match the server's configuration. Incidentally, I have not verified it very thoroughly.",
"JWT Algorithm": "JWT Algorithm",
@@ -738,6 +744,7 @@
"On the source device, open Obsidian.": "On the source device, open Obsidian.",
"On this device, please keep this Vault open.": "On this device, please keep this Vault open.",
"On this device, switch to the camera app or use a QR code scanner to scan the displayed QR code.": "On this device, switch to the camera app or use a QR code scanner to scan the displayed QR code.",
"Opaque Journal (current format)": "Opaque Journal (current format)",
"Open": "Open",
"Open connection": "Open connection",
"Open P2P Setup...": "Open P2P Setup...",
@@ -768,6 +775,7 @@
"P2P.SyncAlreadyRunning": "P2P Sync is already running.",
"P2P.SyncCompleted": "P2P Sync completed.",
"P2P.SyncStartedWith": "P2P Sync with ${name} have been started.",
"Pack retrieval": "Pack retrieval",
"paneMaintenance.markDeviceResolvedAfterBackup": "paneMaintenance.markDeviceResolvedAfterBackup",
"paneMaintenance.remoteLockedAndDeviceNotAccepted": "paneMaintenance.remoteLockedAndDeviceNotAccepted",
"paneMaintenance.remoteLockedResolvedDevice": "paneMaintenance.remoteLockedResolvedDevice",
@@ -1418,6 +1426,7 @@
"Use JWT Authentication": "Use JWT Authentication",
"Use Path-Style Access": "Use Path-Style Access",
"Use Random Number": "Use Random Number",
"Use S3 Range requests": "Use S3 Range requests",
"Use Segmented-splitter": "Use Segmented-splitter",
"Use splitting-limit-capped chunk splitter": "Use splitting-limit-capped chunk splitter",
"Use the trash bin": "Use the trash bin",

View File

@@ -66,6 +66,21 @@ AddOn Module (HiddenFileSync) has not been loaded. This is very unexpected situa
situation. Please report this issue.
Advanced: Advanced
Advanced Settings: Advanced Settings
Adaptive Journal (experimental): Adaptive Journal (experimental)
Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.:
Adaptive Journal uses immutable objects and a separate remote format.
Existing Opaque Journal data is not migrated or read. Rebuild the remote when
changing formats.
Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.:
Complete Pack reads favour throughput. Range reads can reduce transferred
bytes. The connection test verifies exact Range support on this endpoint, and
synchronisation refuses an unsupported selection before writing.
Download complete Packs: Download complete Packs
"Invalid Object Storage settings: ${reason}": "Invalid Object Storage settings: ${reason}"
Journal data format: Journal data format
Opaque Journal (current format): Opaque Journal (current format)
Pack retrieval: Pack retrieval
Use S3 Range requests: Use S3 Range requests
After restarting, the data on this device will be uploaded to the server as the 'master copy'. Please be aware that any unintended data currently on the server will be completely overwritten.:
After restarting, the data on this device will be uploaded to the server as
the 'master copy'. Please be aware that any unintended data currently on the

View File

@@ -1,5 +1,5 @@
import { fireAndForget } from "octagonal-wheels/promises";
import { REMOTE_MINIO, REMOTE_P2P, type RemoteDBSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { isJournalRemoteType, REMOTE_P2P, type RemoteDBSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { LiveSyncCouchDBReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/couchdb/LiveSyncReplicator";
import type { LiveSyncAbstractReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/LiveSyncAbstractReplicator";
import { AbstractModule } from "@/modules/AbstractModule";
@@ -9,7 +9,7 @@ export class ModuleReplicatorCouchDB extends AbstractModule {
_anyNewReplicator(settingOverride: Partial<RemoteDBSettings> = {}): Promise<LiveSyncAbstractReplicator | false> {
const settings = { ...this.settings, ...settingOverride };
// If new remote types were added, add them here. Do not use `REMOTE_COUCHDB` directly for the safety valve.
if (settings.remoteType == REMOTE_MINIO || settings.remoteType == REMOTE_P2P) {
if (isJournalRemoteType(settings.remoteType) || settings.remoteType == REMOTE_P2P) {
return Promise.resolve(false);
}
return Promise.resolve(new LiveSyncCouchDBReplicator(this.core));
@@ -17,7 +17,7 @@ export class ModuleReplicatorCouchDB extends AbstractModule {
_everyAfterResumeProcess(): Promise<boolean> {
if (this.services.appLifecycle.isSuspended()) return Promise.resolve(true);
if (!this.services.appLifecycle.isReady()) return Promise.resolve(true);
if (this.settings.remoteType != REMOTE_MINIO && this.settings.remoteType != REMOTE_P2P) {
if (!isJournalRemoteType(this.settings.remoteType) && this.settings.remoteType != REMOTE_P2P) {
const LiveSyncEnabled = this.settings.liveSync;
const continuous = LiveSyncEnabled;
const eventualOnStart = !LiveSyncEnabled && this.settings.syncOnStart;

View File

@@ -1,7 +1,11 @@
import { describe, expect, it, vi } from "vitest";
import { REMOTE_WEBDAV } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { ModuleReplicatorCouchDB } from "./ModuleReplicatorCouchDB.ts";
function createModule(settings: { liveSync: boolean; syncOnStart: boolean }, isReplicationReady = true) {
function createModule(
settings: { liveSync: boolean; remoteType?: string; syncOnStart: boolean },
isReplicationReady = true
) {
const openReplication = vi.fn(async () => true);
const runFiniteReplicationActivity = vi.fn(async (task: () => unknown) => await task());
const services = {
@@ -86,4 +90,19 @@ describe("ModuleReplicatorCouchDB resume replication activity", () => {
expect(runFiniteReplicationActivity).not.toHaveBeenCalled();
expect(openReplication).not.toHaveBeenCalled();
});
it("does not claim or resume WebDAV as CouchDB", async () => {
const { module, openReplication, runFiniteReplicationActivity } = createModule({
liveSync: true,
remoteType: REMOTE_WEBDAV,
syncOnStart: true,
});
await expect(module._anyNewReplicator()).resolves.toBe(false);
await module._everyAfterResumeProcess();
await new Promise((resolve) => setTimeout(resolve, 0));
expect(runFiniteReplicationActivity).not.toHaveBeenCalled();
expect(openReplication).not.toHaveBeenCalled();
});
});

View File

@@ -1,13 +1,13 @@
import { REMOTE_MINIO, type RemoteDBSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { isJournalRemoteType, type RemoteDBSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { LiveSyncJournalReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalReplicator";
import type { LiveSyncAbstractReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/LiveSyncAbstractReplicator";
import type { LiveSyncCore } from "@/main";
import { AbstractModule } from "@/modules/AbstractModule";
export class ModuleReplicatorMinIO extends AbstractModule {
export class ModuleReplicatorJournal extends AbstractModule {
_anyNewReplicator(settingOverride: Partial<RemoteDBSettings> = {}): Promise<LiveSyncAbstractReplicator | false> {
const settings = { ...this.settings, ...settingOverride };
if (settings.remoteType == REMOTE_MINIO) {
if (isJournalRemoteType(settings.remoteType)) {
return Promise.resolve(new LiveSyncJournalReplicator(this.core));
}
return Promise.resolve(false);

View File

@@ -0,0 +1,47 @@
import { describe, expect, it, vi } from "vitest";
import { REMOTE_COUCHDB, REMOTE_MINIO, REMOTE_WEBDAV } from "@vrtmrz/livesync-commonlib/compat/common/types";
const journalReplicatorConstructor = vi.hoisted(() => vi.fn());
vi.mock("@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalReplicator", () => ({
LiveSyncJournalReplicator: class {
constructor(env: unknown) {
journalReplicatorConstructor(env);
}
},
}));
import { ModuleReplicatorJournal } from "./ModuleReplicatorJournal.ts";
function createModule(remoteType: string) {
const services = {
API: {
addCommand: vi.fn(),
addLog: vi.fn(),
addRibbonIcon: vi.fn(),
registerProtocolHandler: vi.fn(),
registerWindow: vi.fn(),
},
};
const core = {
_services: services,
services,
settings: { remoteType },
} as any;
return { core, module: new ModuleReplicatorJournal(core) };
}
describe("ModuleReplicatorJournal", () => {
it.each([REMOTE_MINIO, REMOTE_WEBDAV])("creates a Journal replicator for %s", async (remoteType) => {
const { core, module } = createModule(remoteType);
await expect(module._anyNewReplicator()).resolves.not.toBe(false);
expect(journalReplicatorConstructor).toHaveBeenLastCalledWith(core);
});
it("does not claim a CouchDB remote", async () => {
const { module } = createModule(REMOTE_COUCHDB);
await expect(module._anyNewReplicator()).resolves.toBe(false);
});
});

View File

@@ -83,7 +83,10 @@ function suggestRemoteConfigurationName(parsed: RemoteConfigurationResult): stri
if (parsed.type === "s3") {
return `S3 ${parsed.settings.bucket || parsed.settings.endpoint}`;
}
return `P2P ${parsed.settings.P2P_roomID || "Remote"}`;
if (parsed.type === "p2p") {
return `P2P ${parsed.settings.P2P_roomID || "Remote"}`;
}
return "WebDAV Remote";
}
export function paneRemoteConfig(

View File

@@ -28,6 +28,9 @@ describe("syncActivatedRemoteSettings", () => {
useCustomRequestHandler: false,
forcePathStyle: true,
bucketCustomHeaders: "",
expectedRepositoryId: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
journalFormat: "adaptive-v1" as const,
packReadPolicy: "range" as const,
};
syncActivatedRemoteSettings(target, source);
@@ -40,6 +43,9 @@ describe("syncActivatedRemoteSettings", () => {
expect(target.bucket).toBe("vault");
expect(target.region).toBe("sz-hq");
expect(target.bucketPrefix).toBe("folder/");
expect(target.expectedRepositoryId).toBe("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA");
expect(target.journalFormat).toBe("adaptive-v1");
expect(target.packReadPolicy).toBe("range");
expect(target.encrypt).toBe(true);
});

View File

@@ -407,6 +407,9 @@ describe("SetupManager", () => {
useCustomRequestHandler: false,
bucketCustomHeaders: "",
forcePathStyle: true,
expectedRepositoryId: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
journalFormat: "adaptive-v1",
packReadPolicy: "range",
})
.mockResolvedValueOnce(true);
@@ -419,6 +422,9 @@ describe("SetupManager", () => {
const activeProfile = current.remoteConfigurations[current.activeConfigurationId];
expect(activeProfile?.name).toBe("S3 notes");
expect(activeProfile?.uri).toContain("sls+s3://key:secret@storage.example");
expect(activeProfile?.uri).toContain("journalFormat=adaptive-v1");
expect(activeProfile?.uri).toContain("packReadPolicy=range");
expect(activeProfile?.uri).toContain("expectedRepositoryId=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA");
});
it("creates and selects a P2P profile during fresh manual onboarding", async () => {

View File

@@ -20,10 +20,16 @@
import { copyTo, pickBucketSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/utils";
import { TYPE_CANCELLED, type SetupRemoteBucketResultType } from "./setupDialogTypes";
import { $msg as translateMessage } from "@/common/translation";
import { normaliseS3JournalSettings } from "./s3JournalSettings";
const default_setting = pickBucketSyncSettings(DEFAULT_SETTINGS);
let syncSetting = $state<BucketSyncSetting>({ ...default_setting });
let syncSetting = $state<BucketSyncSetting>({
...default_setting,
expectedRepositoryId: default_setting.expectedRepositoryId ?? "",
journalFormat: default_setting.journalFormat ?? "opaque-v1",
packReadPolicy: default_setting.packReadPolicy ?? "whole-pack",
});
type Props = GuestDialogProps<SetupRemoteBucketResultType, BucketSyncSetting>;
@@ -58,11 +64,10 @@
isEndpointSupplied
);
});
const isAdaptive = $derived(syncSetting.journalFormat === "adaptive-v1");
function generateSetting() {
const connSetting: BucketSyncSetting = {
...syncSetting,
};
const connSetting = normaliseS3JournalSettings(syncSetting);
const trialSettings: BucketSyncSetting = {
...connSetting,
};
@@ -115,8 +120,13 @@
}
}
function commit() {
const setting = pickBucketSyncSettings(generateSetting());
setResult(setting);
error = "";
try {
const setting = pickBucketSyncSettings(generateSetting());
setResult(setting);
} catch (e) {
error = translateMessage("Invalid Object Storage settings: ${reason}", { reason: `${e}` });
}
}
function cancel() {
setResult(TYPE_CANCELLED);
@@ -220,6 +230,30 @@
</InfoNote>
<ExtraItems title={translateMessage("Advanced Settings")}>
<InputRow label={translateMessage("Journal data format")}>
<select name="s3-journal-format" bind:value={syncSetting.journalFormat}>
<option value="opaque-v1">{translateMessage("Opaque Journal (current format)")}</option>
<option value="adaptive-v1">{translateMessage("Adaptive Journal (experimental)")}</option>
</select>
</InputRow>
<InfoNote warning visible={isAdaptive}>
{translateMessage(
"Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats."
)}
</InfoNote>
{#if isAdaptive}
<InputRow label={translateMessage("Pack retrieval")}>
<select name="s3-pack-read-policy" bind:value={syncSetting.packReadPolicy}>
<option value="whole-pack">{translateMessage("Download complete Packs")}</option>
<option value="range">{translateMessage("Use S3 Range requests")}</option>
</select>
</InputRow>
<InfoNote>
{translateMessage(
"Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing."
)}
</InfoNote>
{/if}
<InputRow label={translateMessage("Custom Headers")}>
<textarea
name="bucket-custom-headers"

View File

@@ -0,0 +1,25 @@
import { DEFAULT_SETTINGS, REMOTE_MINIO, type BucketSyncSetting } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { journalProtocolConfigurationForSettings } from "@vrtmrz/livesync-commonlib/journal-storage";
export function normaliseS3JournalSettings(settings: BucketSyncSetting): BucketSyncSetting {
const journalFormat = settings.journalFormat ?? "opaque-v1";
const candidate: BucketSyncSetting = {
...settings,
bucket: settings.bucket.trim(),
bucketPrefix: settings.bucketPrefix.trim(),
endpoint: settings.endpoint.trim(),
expectedRepositoryId: journalFormat === "adaptive-v1" ? (settings.expectedRepositoryId ?? "").trim() : "",
journalFormat,
packReadPolicy: journalFormat === "adaptive-v1" ? (settings.packReadPolicy ?? "whole-pack") : "whole-pack",
region: settings.region.trim(),
};
const protocol = journalProtocolConfigurationForSettings({
...DEFAULT_SETTINGS,
...candidate,
remoteType: REMOTE_MINIO,
});
return {
...candidate,
...protocol,
};
}

View File

@@ -0,0 +1,79 @@
import { describe, expect, it } from "vitest";
import { DEFAULT_SETTINGS, type BucketSyncSetting } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { pickBucketSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/utils";
import { normaliseS3JournalSettings } from "./s3JournalSettings";
function settings(overrides: Partial<BucketSyncSetting> = {}): BucketSyncSetting {
return {
...pickBucketSyncSettings(DEFAULT_SETTINGS),
...overrides,
};
}
describe("normaliseS3JournalSettings", () => {
it("retains and validates Adaptive repository options", () => {
const repositoryId = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
const result = normaliseS3JournalSettings(
settings({
bucket: " vault ",
bucketPrefix: " journals/ ",
endpoint: " https://storage.example ",
expectedRepositoryId: ` ${repositoryId} `,
journalFormat: "adaptive-v1",
packReadPolicy: "range",
region: " auto ",
})
);
expect(result).toMatchObject({
bucket: "vault",
bucketPrefix: "journals/",
endpoint: "https://storage.example",
expectedRepositoryId: repositoryId,
journalFormat: "adaptive-v1",
packReadPolicy: "range",
region: "auto",
});
});
it("uses conservative Opaque defaults for older profiles", () => {
const legacy = settings();
delete legacy.expectedRepositoryId;
delete legacy.journalFormat;
delete legacy.packReadPolicy;
expect(normaliseS3JournalSettings(legacy)).toMatchObject({
expectedRepositoryId: "",
journalFormat: "opaque-v1",
packReadPolicy: "whole-pack",
});
});
it("clears Adaptive-only options when Opaque Journal is selected", () => {
expect(
normaliseS3JournalSettings(
settings({
expectedRepositoryId: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
journalFormat: "opaque-v1",
packReadPolicy: "range",
})
)
).toMatchObject({
expectedRepositoryId: "",
journalFormat: "opaque-v1",
packReadPolicy: "whole-pack",
});
});
it("rejects a non-canonical expected repository ID", () => {
expect(() =>
normaliseS3JournalSettings(
settings({
expectedRepositoryId: "not-a-repository-id",
journalFormat: "adaptive-v1",
})
)
).toThrow("expectedRepositoryId must be a canonical base64url-encoded 32-byte value");
});
});

View File

@@ -114,7 +114,7 @@ The mobile pass uses Obsidian's `app.emulateMobile(true)`, a 390 by 844 CSS-pixe
`test:e2e:obsidian:p2p-pane` starts one configured CouchDB-only session with no P2P profile and separate configured P2P sessions for desktop and mobile. It proves that the command remains registered while the retired command, automatic pane, and ribbon entry without a P2P configuration are absent. For the configured P2P profiles, it verifies that the desktop ribbon is available, the current status command reaches the pane without it opening at start-up, checks its connection control and horizontal layout, and captures unobstructed desktop and mobile screenshots. The mobile session uses a fresh Vault, profile, and Obsidian process, enters `app.emulateMobile(true)` through `lifecycle.beforePluginStart`, and requires the P2P view to belong to the right drawer rather than inheriting desktop workspace state. It deliberately uses no relay or peer: replacement of the active replicator is covered by focused unit tests, the Deno and Compose CLI P2P lifecycle suite covers the headless transport, and `p2p-setup-uri-workflow` owns the visible transfer path between two real Obsidian sessions.
`test:e2e:obsidian:local-suite` builds the plug-in and, unless `LIVESYNC_CLI_COMMAND` selects an external CLI, the local LiveSync CLI. It then runs discovery, smoke, the onboarding invitation, Svelte dialogue mounting, revision repair, settings UI, the Review Harness, the P2P status pane, Vault reflection, CouchDB upload and manual setup, CLI-to-Obsidian synchronisation, Object Storage upload and Setup URI round-trip, P2P Setup URI round-trip, startup scan, provisioned CouchDB Setup URI, two-vault synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export in sequence. Start the local CouchDB, MinIO, and P2P relay fixtures before running it, or use `test:e2e:obsidian:local-suite:services` to let the wrapper stop leftover fixtures, start fresh fixtures, and stop them again after the run.
`test:e2e:obsidian:local-suite` builds the plug-in and, unless `LIVESYNC_CLI_COMMAND` selects an external CLI, the local LiveSync CLI. It then runs discovery, smoke, the onboarding invitation, Svelte dialogue mounting, revision repair, settings UI, the Review Harness, the P2P status pane, Vault reflection, CouchDB upload and manual setup, CLI-to-Obsidian synchronisation, Opaque and Adaptive Object Storage uploads, the Object Storage Setup URI round-trip, the P2P Setup URI round-trip, startup scan, the provisioned CouchDB Setup URI, two-vault synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export in sequence. Start the local CouchDB, MinIO, and P2P relay fixtures before running it, or use `test:e2e:obsidian:local-suite:services` to let the wrapper stop leftover fixtures, start fresh fixtures, and stop them again after the run.
`test:e2e:obsidian:couchdb-upload` reuses the CouchDB variables from `.test.env` or the process environment. It expects a reachable CouchDB service, creates a unique database, starts from configured plug-in data without the device-local compatibility marker, and verifies the copied-or-restored Vault explanation in the actual compatibility dialogue. It captures the summary and details, resumes explicitly, confirms that the marker was recorded, creates a note in real Obsidian, commits the note into the local database, runs one-shot synchronisation, and verifies that the remote database contains both the metadata document and its chunk documents.
@@ -147,6 +147,8 @@ LIVESYNC_CLI_COMMAND="docker run --rm --network host --user $(id -u):$(id -g) --
`test:e2e:obsidian:minio-upload` reuses the Object Storage variables from `.test.env` or the process environment. It expects a reachable S3-compatible service and starts with isolated Object Storage settings and the device-local compatibility acknowledgement already in place, keeping the scenario focused on upload rather than unconfigured start-up or setup. It confirms those settings through `obsidian-cli eval`, creates a note in real Obsidian, runs one-shot Journal Sync, and verifies through the AWS SDK that objects were written under a unique bucket prefix. Adapter tests separately observe an in-progress SDK command, while this real-runtime workflow verifies the resulting request counters advance and rebalance.
`test:e2e:obsidian:adaptive-s3` reuses that one-device upload workflow with the experimental Adaptive Journal format and Range retrieval selected explicitly. It requires real Obsidian to retain those settings, publish a Chunk-backed note through one-shot synchronisation, and produce an authenticated manifest, a writer record, and an immutable Commit Bundle under a disposable MinIO prefix without writing the Opaque Journal milestone. Commonlib tests own the storage protocol and failure classification, while the CLI E2E owns bidirectional synchronisation, external Packs, and both Pack retrieval policies; this scenario verifies only the Obsidian composition boundary.
`test:e2e:obsidian:object-storage-setup-uri-workflow` uses the public Commonlib-backed tool to generate the initial Setup URI for a unique MinIO prefix, completes visible initialisation on the first device, and then asks that working real Obsidian device to create a new Setup URI through the registered command. A second real Obsidian device imports only the device-generated URI. The workflow verifies A-to-B and B-to-A notes, captures the documented onboarding choices, and removes the Object Storage prefix only after both sessions have stopped.
`test:e2e:obsidian:p2p-setup-uri-workflow` runs two concurrent isolated real Obsidian sessions against the local Compose Nostr relay fixture. The first device imports a generated initial Setup URI and completes its signalling test with zero peers, creates a Setup URI for the second device through the registered command, and remains online while the second device imports it. The second device must select the expected online source before Fetch can rebuild its local database. The workflow accepts each connection request visibly on the receiving device, verifies the initial A-to-B fetch, checks that the menu for the three persistent per-peer actions remains within the viewport, reconnects both P2P sessions in join order, and verifies the B-to-A return journey. Every started session remains tracked until teardown completes.

View File

@@ -19,6 +19,9 @@ export type ConfiguredSettings = {
endpoint?: string;
bucket?: string;
bucketPrefix?: string;
expectedRepositoryId?: string;
journalFormat?: string;
packReadPolicy?: string;
};
export type CoreReadiness = {
@@ -320,6 +323,9 @@ export async function configureObjectStorage(
"endpoint:current.endpoint,",
"bucket:current.bucket,",
"bucketPrefix:current.bucketPrefix,",
"expectedRepositoryId:current.expectedRepositoryId,",
"journalFormat:current.journalFormat,",
"packReadPolicy:current.packReadPolicy,",
"});",
"})()",
].join(""),

View File

@@ -30,6 +30,7 @@ const testSteps: Step[] = [
args: ["run", "test:e2e:obsidian:cli-to-obsidian-sync"],
},
{ name: "Object Storage upload", args: ["run", "test:e2e:obsidian:minio-upload"] },
{ name: "Adaptive S3 upload", args: ["run", "test:e2e:obsidian:adaptive-s3"] },
{
name: "Object Storage Setup URI workflow",
args: ["run", "test:e2e:obsidian:object-storage-setup-uri-workflow"],

View File

@@ -33,6 +33,7 @@ import {
listObjectStorageObjects,
loadObjectStorageConfig,
makeUniqueBucketPrefix,
readObjectStorageJson,
} from "../runner/objectStorage.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { createTemporaryVault } from "../runner/vault.ts";
@@ -40,9 +41,21 @@ import { REMOTE_ACTIVITY_EXPECTED_STATE, waitForRemoteActivityState } from "../r
process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "30000";
const notePath = "E2E/minio-upload.md";
const adaptive = process.argv.includes("--adaptive");
const unsupportedArguments = process.argv.slice(2).filter((argument) => argument !== "--adaptive");
if (unsupportedArguments.length > 0) {
throw new Error(`Unsupported Object Storage upload argument: ${unsupportedArguments.join(", ")}`);
}
const journalSettings = adaptive
? {
journalFormat: "adaptive-v1",
packReadPolicy: "range",
}
: {};
const scenarioName = adaptive ? "Adaptive S3" : "Object Storage";
const notePath = adaptive ? "E2E/adaptive-s3-upload.md" : "E2E/minio-upload.md";
const noteContent = [
"# Object Storage upload from real Obsidian",
`# ${scenarioName} upload from real Obsidian`,
"",
"This note is created through Obsidian and uploaded by Self-hosted LiveSync to S3-compatible Object Storage.",
"The test is intentionally small, but it crosses the real Obsidian, Journal Sync, and AWS SDK boundary.",
@@ -78,7 +91,7 @@ async function createNoteAndWaitForLocalDb(cliBinary: string, env: NodeJS.Proces
);
}
async function waitForObjectStorageObjects(prefix: string): Promise<string[]> {
async function waitForObjectStorageObjects(prefix: string, requiredKeyPrefix?: string): Promise<string[]> {
const objectStorage = await loadObjectStorageConfig();
const timeoutMs = Number(process.env.E2E_OBSIDIAN_OBJECT_STORAGE_TIMEOUT_MS ?? 20000);
const deadline = Date.now() + timeoutMs;
@@ -86,12 +99,48 @@ async function waitForObjectStorageObjects(prefix: string): Promise<string[]> {
while (Date.now() < deadline) {
const objects = await listObjectStorageObjects(objectStorage, prefix);
keys = objects.flatMap((object) => (object.Key ? [object.Key] : []));
if (keys.length > 0) {
if (keys.length > 0 && (!requiredKeyPrefix || keys.some((key) => key.startsWith(requiredKeyPrefix)))) {
return keys;
}
await new Promise((resolve) => setTimeout(resolve, 500));
}
throw new Error(`Timed out waiting for Object Storage objects under ${prefix}. Last keys: ${keys.join(", ")}`);
throw new Error(
`Timed out waiting for Object Storage objects under ${prefix}${requiredKeyPrefix ? ` with prefix ${requiredKeyPrefix}` : ""}. Last keys: ${keys.join(", ")}`
);
}
async function assertAdaptiveObjects(prefix: string, keys: string[]): Promise<void> {
const objectStorage = await loadObjectStorageConfig();
const manifestKey = `${prefix}a1~manifest.json`;
const requiredPrefixes = [`${prefix}a1~writer~`, `${prefix}a1~commit~`];
if (!keys.includes(manifestKey)) {
throw new Error(`Adaptive Journal manifest is missing. Keys: ${keys.join(", ")}`);
}
for (const requiredPrefix of requiredPrefixes) {
if (!keys.some((key) => key.startsWith(requiredPrefix))) {
throw new Error(`Adaptive Journal object prefix ${requiredPrefix} is missing. Keys: ${keys.join(", ")}`);
}
}
if (keys.includes(`${prefix}_00000000-milestone.json`)) {
throw new Error("Adaptive Journal wrote the legacy Opaque Journal milestone.");
}
const manifest = await readObjectStorageJson<{
format?: unknown;
formatVersion?: unknown;
manifestAuth?: unknown;
objectLayout?: unknown;
repositoryId?: unknown;
}>(objectStorage, manifestKey);
assertEqual(manifest.format, "adaptive-journal", "Unexpected Adaptive Journal manifest format.");
assertEqual(manifest.formatVersion, 1, "Unexpected Adaptive Journal manifest version.");
assertEqual(manifest.objectLayout, "commit-bundle-v1", "Unexpected Adaptive Journal object layout.");
if (typeof manifest.repositoryId !== "string" || manifest.repositoryId.length === 0) {
throw new Error("Adaptive Journal manifest did not contain a repository ID.");
}
if (typeof manifest.manifestAuth !== "string" || manifest.manifestAuth.length === 0) {
throw new Error("Adaptive Journal manifest did not contain its authentication value.");
}
}
async function main(): Promise<void> {
@@ -102,7 +151,7 @@ async function main(): Promise<void> {
}
const objectStorage = await loadObjectStorageConfig();
const bucketPrefix = makeUniqueBucketPrefix("minio-upload");
const bucketPrefix = makeUniqueBucketPrefix(adaptive ? "adaptive-s3-upload" : "minio-upload");
const vault = await createTemporaryVault();
let session: ObsidianLiveSyncSession | undefined;
@@ -119,18 +168,26 @@ async function main(): Promise<void> {
cliBinary: cli.binary,
vault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: createE2eObjectStoragePluginData({
...objectStorage,
bucketPrefix,
}),
pluginData: createE2eObjectStoragePluginData(
{
...objectStorage,
bucketPrefix,
},
journalSettings
),
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
});
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
const configured = await configureObjectStorage(cli.binary, session.cliEnv, {
...objectStorage,
bucketPrefix,
});
const configured = await configureObjectStorage(
cli.binary,
session.cliEnv,
{
...objectStorage,
bucketPrefix,
},
journalSettings
);
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
assertEqual(configured.isConfigured, true, "Self-hosted LiveSync was not marked as configured.");
assertEqual(configured.remoteType, "MINIO", "Remote type was not Object Storage.");
@@ -138,6 +195,11 @@ async function main(): Promise<void> {
assertEqual(configured.bucket, objectStorage.bucket, "Configured Object Storage bucket did not match.");
assertEqual(configured.bucketPrefix, bucketPrefix, "Configured Object Storage bucket prefix did not match.");
assertEqual(configured.liveSync, false, "LiveSync should remain disabled during this one-shot workflow.");
if (adaptive) {
assertEqual(configured.journalFormat, "adaptive-v1", "Adaptive Journal format was not retained.");
assertEqual(configured.packReadPolicy, "range", "Adaptive Journal Pack retrieval was not retained.");
assertEqual(configured.expectedRepositoryId, "", "A new Adaptive repository should not be pre-bound.");
}
await prepareRemote(cli.binary, session.cliEnv);
const activityBeforeUpload = await waitForRemoteActivityState(
@@ -159,10 +221,16 @@ async function main(): Promise<void> {
"Object Storage remote-request counters did not rebalance after synchronisation."
);
const keys = await waitForObjectStorageObjects(bucketPrefix);
const keys = await waitForObjectStorageObjects(
bucketPrefix,
adaptive ? `${bucketPrefix}a1~commit~` : undefined
);
if (adaptive) {
await assertAdaptiveObjects(bucketPrefix, keys);
}
console.log(
`Uploaded ${localEntry.path} through Journal Sync to ${objectStorage.bucket}/${bucketPrefix} (${keys.length} object(s)); tracked requests: ${activityAfterUpload.requestCount - activityBeforeUpload.requestCount}`
`Uploaded ${localEntry.path} through ${scenarioName} Journal Sync to ${objectStorage.bucket}/${bucketPrefix} (${keys.length} object(s)); tracked requests: ${activityAfterUpload.requestCount - activityBeforeUpload.requestCount}`
);
} finally {
if (session) {

View File

@@ -18,6 +18,7 @@ const focusedScenarios = new Set([
"couchdb-manual-setup-workflow",
"cli-to-obsidian-sync",
"minio-upload",
"adaptive-s3",
"object-storage-setup-uri-workflow",
"p2p-setup-uri-workflow",
"startup-scan",

View File

@@ -12,6 +12,12 @@ Earlier releases remain available in the 0.25 release history and the legacy rel
## Unreleased
### Synchronisation and storage
#### Improved
- Object Storage setup can select the experimental Adaptive Journal format and choose complete Pack or verified Range retrieval. Existing Opaque Journal repositories remain the default and require an explicit remote Rebuild before changing formats.
### P2P and experimental browser applications
#### Improved