Add AOF replication for RangeIndex migration#1911
Open
tiagonapoli wants to merge 6 commits into
Open
Conversation
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
tiagonapoli
commented
Jul 6, 2026
ba9689a to
4c56fc9
Compare
tiagonapoli
commented
Jul 12, 2026
tiagonapoli
commented
Jul 12, 2026
88caa4a to
2c75f8d
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end AOF transport for migrated RangeIndex (BF-Tree) keys by streaming the serialized BF-Tree file into the AOF as chunked entries, then reassembling and publishing the full index during replication replay and crash recovery. This closes the “empty tree on replay” gap for migrated range indexes and adds targeted tests to validate both live replica sync and restart recovery paths.
Changes:
- Introduces a new AOF entry type for chunked RangeIndex stream replication and adds replay handling that reassembles/publishes the migrated BF-Tree.
- Suppresses the auto-logged
RICREATEstub during migration publish so the stream becomes the single AOF source of truth. - Expands cluster migration tests (including multi-chunk and interleaving scenarios) and adds unit tests for replay-side reassembly bookkeeping.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/standalone/Garnet.test/RangeIndexStreamReplayTests.cs | Adds unit tests validating replay-side stream reassembly bookkeeping and cleanup behaviors. |
| test/standalone/Garnet.test/RangeIndexChunkedSerializerTests.cs | Updates RangeIndexMigrationReader construction to match the new parameter naming/API. |
| test/standalone/Garnet.test/CapturingLogger.cs | Adds a structured-log capturing test logger to assert on structured fields (e.g., reason). |
| test/cluster/Garnet.test.cluster/ClusterTestContext.cs | Adds a test hook to force small RangeIndex stream chunk sizes across nodes. |
| test/cluster/Garnet.test.cluster.migrate.rangeindex/ClusterRangeIndexMigrateTests.cs | Adds/extends cluster tests to verify migrated RI replication via AOF and recovery across restarts, including multi-chunk/interleaving/failover. |
| libs/server/Storage/Functions/MainStore/PrivateMethods.cs | Skips auto-AOF logging for migration-publish RICREATE using a sentinel argument. |
| libs/server/Resp/RangeIndex/RangeIndexReplicationActivities.cs | Adds structured activity logging for stream replication and reassembly. |
| libs/server/Resp/RangeIndex/RangeIndexMigrationReader.cs | Refactors reader to accept a general Stream, adds sync chunk read, and introduces a default internal read-buffer size. |
| libs/server/Resp/RangeIndex/RangeIndexManager.Replication.cs | Implements AOF chunk streaming, replay reassembly state, and publish-on-complete handling for migrated indexes. |
| libs/server/Resp/RangeIndex/RangeIndexManager.Migration.cs | Publishes migrated indexes by first streaming snapshot into AOF, then suppressing the competing RICREATE AOF entry. |
| libs/server/Resp/RangeIndex/RangeIndexManager.cs | Ensures incomplete reassembly state is dropped during manager disposal. |
| libs/server/AOF/AofProcessor.cs | Adds replay dispatch for the new RangeIndex stream chunk entry type and cleanup on dispose. |
| libs/server/AOF/AofEntryType.cs | Introduces RangeIndexStreamChunk AOF entry type and marks it replayable/store-type. |
| libs/host/GarnetServer.cs | Adds startup validation that AOF page size can accommodate RangeIndex stream chunks when preview is enabled. |
| libs/host/Garnet.host.csproj | Adds InternalsVisibleTo for cluster tests. |
| libs/cluster/Session/RangeIndexMigrationReceiveSession.cs | Passes AOF handle into migrated-index publish so the target primary can stream to its replicas via AOF. |
| libs/cluster/Session/ClusterSession.cs | Wires append-only file into the RangeIndex migration receive state. |
| libs/cluster/Server/Migration/RangeIndex/RangeIndexMigrationActivities.cs | Minor doc comment formatting change in migration activity tracing. |
| libs/cluster/Server/Migration/MigrateSession.RangeIndex.cs | Updates migration transmit to use the updated snapshot+reader API. |
Signed-off-by: Tiago Napoli <tiagonapoli@microsoft.com>
2c75f8d to
8da6d41
Compare
Assert primary AOF stream chunking, replica reassembly, and crash-recovery reassembly in the multi-chunk RangeIndex migration test via a per-test CapturingLogger. Extract repeated endpoint/migrate/ownership patterns into Endpoint, MigrateSlotsAndWaitCleanup, and MigrateSlotsAndWaitOwnership helpers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Tiago Napoli <tiagonapoli@microsoft.com>
Delete tests whose coverage is subsumed by others: ClusterMigrateRangeIndexSlot (by SingleBySlot), ClusterMigrateEmptyRangeIndex (by the Replicates variant), ClusterMigrateRangeIndexBack (by RoundTripThenRestart), ClusterMigrateRangeIndexThenCheckpointRestart (by CheckpointThenRecover + ThenCheckpointRestartAndContinue), and the interleaved-writes test. Also remove ClusterMigrateRangeIndexWithChunkSize, whose chunkSize parameter was never used (all TestCases ran identically) so it exercised no multi-chunk path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds AOF replication for migrated RangeIndex (BF-Tree) keys, so a migrated
index reaches replicas and survives crash recovery with its full data — not just
an empty tree rebuilt from the stub.
Why
When a node receives a migrated RI key,
PublishMigratedIndexcreated the storerecord via a
RICREATERMW that auto-logged only the stub to the AOF. On AOFreplay,
HandleRangeIndexCreateReplayrebuilds an empty tree from the stubconfig, so the migrated data never reached secondaries or survived recovery
(the
TODO(RI)atRangeIndexManager.Migration.cs). A BF-Tree file can alsoexceed a single AOF page, so it must be chunked.
How
RISTREAM— a new internal, AOF-only pseudo-command (never parsed from thenetwork), intercepted in
AofProcessor.StoreRMWalongsideRICREATE/RISET/RIDEL.ReplicateRangeIndexStream): the reassembled BfTree file isstreamed into the AOF as chunked
RISTREAMentries (reusingRangeIndexChunkedSerializer), each entry keyed by the RI key and no larger thanDefaultMigrationChunkSize; the final chunk is flagged.HandleRangeIndexStreamReplay): a per-keyRangeIndexChunkedDeserializerreassembles the file across interleaved AOFentries (all chunks for one key hash to a single virtual sublog, so they arrive
in order on one replay task), then publishes via
PublishMigratedIndex— whichre-streams on a chained replica. Incomplete reassembly (crash mid-stream) is
dropped by
CleanupIncompleteStreamReassembly.RICREATERMW carries theStreamedPublishLogArgsentinel soWriteLogRMWskips its auto-log. TheRISTREAMstream is the single AOF source of truth for a migrated key; keeping acompeting empty-tree
RICREATEentry would cause data loss on replay in eitherordering (it would block the publish via the
KeyExistsgate, or clobber thereassembled tree).
Source-side deletion of the migrated key already replicates via the normal
DELETE-> AOF path (out of scope here).Tests
ClusterMigrateRangeIndexReplicatesToReplicaViaAof— a migrated DISK-backed RIkey replicates to the destination primary's replica via AOF; the replica serves
the full data.
ClusterMigrateRangeIndexRecoversFromAof— the destination primary reconstructsthe migrated key from its own AOF on restart.
migrate.rangeindexsuite (21) green;RangeIndexChunkedSerializerTests(57) and
RangeIndexMigrationReceiveStateTests(2) green; full solution buildswith 0 warnings.
Notes
MIGRATE REPLACEfor RI keys remains unsupported (pre-existingTODO).--enable-range-index-preview).