[feature](ann-index) Support IVF on-disk index type for ANN vector search - #61160
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
zhiqiang-hhhh
force-pushed
the
feat-ivf-on-disk
branch
from
March 13, 2026 06:38
f10199a to
ba24111
Compare
zhiqiang-hhhh
changed the title
zhiqiang-hhhh
force-pushed
the
feat-ivf-on-disk
branch
from
March 22, 2026 10:43
765ee9a to
f3ebf42
Compare
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 27028 ms |
TPC-DS: Total hot run time: 168656 ms |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 27092 ms |
TPC-DS: Total hot run time: 168033 ms |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
zhiqiang-hhhh
marked this pull request as ready for review
March 23, 2026 08:47
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Update the ivf_on_disk regression case to assert successful stream load behavior now that the underlying path succeeds. ### Release note None ### Check List (For Author) - Test: No need to test (test expectation update only; no local test run in this commit) - Behavior changed: No (regression expectation only) - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Clarify that ann_index_ivf_list_cache_limit="70%" is based on process-available memory, and with default mem_limit="90%" it is effectively about 63% of process-visible physical memory (including cgroup constraints). ### Release note None ### Check List (For Author) - Test: No need to test (comment-only change) - Behavior changed: No - Does this need documentation: No
zhiqiang-hhhh
force-pushed
the
feat-ivf-on-disk
branch
from
March 25, 2026 03:52
e312870 to
5e49415
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 27093 ms |
TPC-DS: Total hot run time: 169049 ms |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Mar 31, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 26458 ms |
TPC-DS: Total hot run time: 168874 ms |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
airborne12
approved these changes
Apr 1, 2026
airborne12
left a comment
airborne12
left a comment
Member
There was a problem hiding this comment.
LGTM. Well-designed feature that enables IVF vector search on datasets exceeding available memory, with clean architecture and thorough documentation.
Key strengths:
CachedRandomAccessReaderwith zero-copyborrow()and per-range LRU caching aligns perfectly with IVF access patterns- Excellent lifetime management:
_compound_dirdeclared before_vector_indexwith clear comment explaining C++ reverse destruction order, preventing use-after-free - FAISS exception handling added to both
search()andrange_search()paths IO_FLAG_SKIP_PRECOMPUTE_TABLEcomment provides detailed per-segment memory calculation (512 MiB per segment × 146 segments = 73 GiB)- Comprehensive regression tests covering L2, inner product, error cases, stream load, and range search
Behavior changes to document (non-blocking):
- Default
ivf_nprobechanged from 1 to 32 — improves recall but increases search cost for existing IVF users IO_FLAG_SKIP_PRECOMPUTE_TABLEnow applied to in-memory IVF load as well — saves significant RSS but may affect search latency- Consider adding a config flag if users need the precomputed table for latency-sensitive in-memory IVF workloads
Minor suggestions:
- Unify config.h/config.cpp comments on cache limit percentage base
- Add comment noting thread-local cache stats assumes single-threaded search (
nq=1,parallel_mode=0) - Write path could chunk large list data similar to read path (
kMaxChunkpattern)
morningman
pushed a commit
that referenced
this pull request
Apr 2, 2026
…arch (#61160) ## Summary This PR introduces a new ANN index type `ivf_on_disk` that stores IVF inverted list data on disk instead of fully loading it into memory. This enables vector search on datasets that exceed available memory, with a dedicated LRU cache for frequently accessed IVF list pages. ## Motivation The existing `ivf` index type loads the entire IVF index (including all inverted list codes and IDs) into memory during search. For large-scale vector datasets (billions of vectors), this makes the memory footprint prohibitively expensive. The `ivf_on_disk` approach stores the inverted list data in a separate file (`ann.ivfdata`) and reads only the lists needed for each query, backed by an LRU cache for hot data. ## Changes ### BE - Core IVF On-Disk Implementation **New index type `IVF_ON_DISK`:** - Extended `AnnIndexType` enum with `IVF_ON_DISK` and added string conversion support (`be/src/storage/index/ann/ann_index.h`, `ann_index.cpp`) - Extended `FaissBuildParameter::IndexType` with `IVF_ON_DISK` (`be/src/storage/index/ann/faiss_ann_index.h`) - Added `faiss_ivfdata_file_name` constant (`ann.ivfdata`) for the separate data file (`be/src/storage/index/ann/ann_index_files.h`) **On-disk save/load in `FaissVectorIndex` (`faiss_ann_index.cpp`):** - **Save path**: Converts in-memory `ArrayInvertedLists` to `OnDiskInvertedLists` format, writes list data to `ann.ivfdata` and index metadata to `ann.faiss` - **Load path**: Reads `ann.ivfdata` via a `CachedRandomAccessReader` backed by an LRU cache; replaces the deserialized `PreadInvertedLists` with a cached reader that provides zero-copy `borrow()` for repeated list accesses - Introduced `CachedRandomAccessReader` implementing `faiss::RandomAccessReader` with per-range LRU caching keyed by `(file-prefix, file-size, byte-offset)` **Dedicated IVF list cache (`be/src/storage/cache/ann_index_ivf_list_cache.h/cpp`):** - New `AnnIndexIVFListCache` class — a dedicated LRU cache separated from `StoragePageCache` to avoid contention with column data pages - Configurable capacity via `ann_index_ivf_list_cache_limit` (default: 70% of physical memory) - Registered in `CachePolicy` as `ANN_INDEX_IVF_LIST_CACHE` **Runtime environment integration:** - `ExecEnv` now creates/destroys the `AnnIndexIVFListCache` singleton - Config entries: `ann_index_ivf_list_cache_limit`, `ann_index_ivf_list_cache_stale_sweep_time_sec` **Metrics & profiling:** - Added 6 new metrics: `ann_ivf_on_disk_fetch_page_costs_ms`, `ann_ivf_on_disk_fetch_page_cnt`, `ann_ivf_on_disk_search_costs_ms`, `ann_ivf_on_disk_search_cnt`, `ann_ivf_on_disk_cache_hit_cnt`, `ann_ivf_on_disk_cache_miss_cnt` - Extended `AnnIndexStats` and `OlapReaderStatistics` with IVF on-disk counters - Propagated stats through `AnnIndexReader::query()` and `range_search()` to `SegmentIterator` **Search execution:** - `AnnIndexReader` now handles `IVF_ON_DISK` alongside `IVF` for both top-N and range search paths - `ScopedIoCtxBinding` propagates `IOContext` via `thread_local` so `CachedRandomAccessReader` can attribute file-cache stats to the correct query - `ScopedOmpThreadBudget` now uses `condition_variable` to properly block waiting index builders instead of silently degrading **Index file writer fix:** - `IndexFileWriter::add_into_searcher_cache()` now correctly skips ANN indexes (both single-file HNSW/IVF and two-file IVF_ON_DISK) by checking for `ann.faiss`/`ann.ivfdata` file names **Compound directory lifetime:** - `AnnIndexReader` now holds `_compound_dir` alive to prevent use-after-free when `CachedRandomAccessReader` holds a cloned `CSIndexInput` whose base pointer references the compound reader's stream ### FE - DDL Validation - `AnnIndexPropertiesChecker.java`: Accept `ivf_on_disk` as a valid index type; require `nlist` for both `ivf` and `ivf_on_disk` ### FAISS Submodule - Updated `contrib/faiss` submodule to a version supporting `PreadInvertedLists` with `RandomAccessReader`/`borrow()` interface ### Regression Tests - `ivf_on_disk_index_test.groovy`: Tests for L2 distance, inner product, missing nlist error, insufficient training points, larger datasets, range search - `create_ann_index_test.groovy`: Added `ivf_on_disk` CREATE INDEX test case - `create_tbl_with_ann_index_test.groovy`: Added CREATE TABLE with `ivf_on_disk` (L2, IP, missing nlist error) - Test data files: `ivf_on_disk_stream_load.csv`, `ivf_on_disk_stream_load.json`, `ivf_on_disk_index_test.out` ## Usage ```sql CREATE TABLE tbl ( id INT NOT NULL, embedding ARRAY<FLOAT> NOT NULL, INDEX idx_emb (`embedding`) USING ANN PROPERTIES( "index_type" = "ivf_on_disk", "metric_type" = "l2_distance", "dim" = "128", "nlist" = "128" ) ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 2 PROPERTIES ("replication_num" = "1"); -- Approximate nearest neighbor search SELECT id, l2_distance_approximate(embedding, [1.0, 2.0, 3.0]) AS dist FROM tbl ORDER BY dist LIMIT 10; Known Limitations - Stream load to ivf_on_disk tables currently fails during index building (the FulltextIndexSearcherBuilder path does not support the two-file format yet). This is covered by a regression test that asserts the current failure behavior.
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Apr 8, 2026
…arch (apache#61160) This PR introduces a new ANN index type `ivf_on_disk` that stores IVF inverted list data on disk instead of fully loading it into memory. This enables vector search on datasets that exceed available memory, with a dedicated LRU cache for frequently accessed IVF list pages. The existing `ivf` index type loads the entire IVF index (including all inverted list codes and IDs) into memory during search. For large-scale vector datasets (billions of vectors), this makes the memory footprint prohibitively expensive. The `ivf_on_disk` approach stores the inverted list data in a separate file (`ann.ivfdata`) and reads only the lists needed for each query, backed by an LRU cache for hot data. **New index type `IVF_ON_DISK`:** - Extended `AnnIndexType` enum with `IVF_ON_DISK` and added string conversion support (`be/src/storage/index/ann/ann_index.h`, `ann_index.cpp`) - Extended `FaissBuildParameter::IndexType` with `IVF_ON_DISK` (`be/src/storage/index/ann/faiss_ann_index.h`) - Added `faiss_ivfdata_file_name` constant (`ann.ivfdata`) for the separate data file (`be/src/storage/index/ann/ann_index_files.h`) **On-disk save/load in `FaissVectorIndex` (`faiss_ann_index.cpp`):** - **Save path**: Converts in-memory `ArrayInvertedLists` to `OnDiskInvertedLists` format, writes list data to `ann.ivfdata` and index metadata to `ann.faiss` - **Load path**: Reads `ann.ivfdata` via a `CachedRandomAccessReader` backed by an LRU cache; replaces the deserialized `PreadInvertedLists` with a cached reader that provides zero-copy `borrow()` for repeated list accesses - Introduced `CachedRandomAccessReader` implementing `faiss::RandomAccessReader` with per-range LRU caching keyed by `(file-prefix, file-size, byte-offset)` **Dedicated IVF list cache (`be/src/storage/cache/ann_index_ivf_list_cache.h/cpp`):** - New `AnnIndexIVFListCache` class — a dedicated LRU cache separated from `StoragePageCache` to avoid contention with column data pages - Configurable capacity via `ann_index_ivf_list_cache_limit` (default: 70% of physical memory) - Registered in `CachePolicy` as `ANN_INDEX_IVF_LIST_CACHE` **Runtime environment integration:** - `ExecEnv` now creates/destroys the `AnnIndexIVFListCache` singleton - Config entries: `ann_index_ivf_list_cache_limit`, `ann_index_ivf_list_cache_stale_sweep_time_sec` **Metrics & profiling:** - Added 6 new metrics: `ann_ivf_on_disk_fetch_page_costs_ms`, `ann_ivf_on_disk_fetch_page_cnt`, `ann_ivf_on_disk_search_costs_ms`, `ann_ivf_on_disk_search_cnt`, `ann_ivf_on_disk_cache_hit_cnt`, `ann_ivf_on_disk_cache_miss_cnt` - Extended `AnnIndexStats` and `OlapReaderStatistics` with IVF on-disk counters - Propagated stats through `AnnIndexReader::query()` and `range_search()` to `SegmentIterator` **Search execution:** - `AnnIndexReader` now handles `IVF_ON_DISK` alongside `IVF` for both top-N and range search paths - `ScopedIoCtxBinding` propagates `IOContext` via `thread_local` so `CachedRandomAccessReader` can attribute file-cache stats to the correct query - `ScopedOmpThreadBudget` now uses `condition_variable` to properly block waiting index builders instead of silently degrading **Index file writer fix:** - `IndexFileWriter::add_into_searcher_cache()` now correctly skips ANN indexes (both single-file HNSW/IVF and two-file IVF_ON_DISK) by checking for `ann.faiss`/`ann.ivfdata` file names **Compound directory lifetime:** - `AnnIndexReader` now holds `_compound_dir` alive to prevent use-after-free when `CachedRandomAccessReader` holds a cloned `CSIndexInput` whose base pointer references the compound reader's stream - `AnnIndexPropertiesChecker.java`: Accept `ivf_on_disk` as a valid index type; require `nlist` for both `ivf` and `ivf_on_disk` - Updated `contrib/faiss` submodule to a version supporting `PreadInvertedLists` with `RandomAccessReader`/`borrow()` interface - `ivf_on_disk_index_test.groovy`: Tests for L2 distance, inner product, missing nlist error, insufficient training points, larger datasets, range search - `create_ann_index_test.groovy`: Added `ivf_on_disk` CREATE INDEX test case - `create_tbl_with_ann_index_test.groovy`: Added CREATE TABLE with `ivf_on_disk` (L2, IP, missing nlist error) - Test data files: `ivf_on_disk_stream_load.csv`, `ivf_on_disk_stream_load.json`, `ivf_on_disk_index_test.out` ```sql CREATE TABLE tbl ( id INT NOT NULL, embedding ARRAY<FLOAT> NOT NULL, INDEX idx_emb (`embedding`) USING ANN PROPERTIES( "index_type" = "ivf_on_disk", "metric_type" = "l2_distance", "dim" = "128", "nlist" = "128" ) ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 2 PROPERTIES ("replication_num" = "1"); -- Approximate nearest neighbor search SELECT id, l2_distance_approximate(embedding, [1.0, 2.0, 3.0]) AS dist FROM tbl ORDER BY dist LIMIT 10; Known Limitations - Stream load to ivf_on_disk tables currently fails during index building (the FulltextIndexSearcherBuilder path does not support the two-file format yet). This is covered by a regression test that asserts the current failure behavior.
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Apr 8, 2026
yiguolei
pushed a commit
that referenced
this pull request
Apr 9, 2026
…es (#62215) ## Summary - backport PR #60358, #61160 and #62178 into branch-4.1 as a single commit - add IVF on-disk ANN index support, related cache/runtime changes, and FE session/property updates - bring over ANN regression coverage updates for IVF, IVF on-disk, small-segment and min-train-rows scenarios
76 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a new ANN index type
ivf_on_diskthat stores IVF inverted list data on disk instead of fully loading it into memory. This enables vector search on datasets that exceed available memory, with a dedicated LRU cache for frequently accessed IVF list pages.Motivation
The existing
ivfindex type loads the entire IVF index (including all inverted list codes and IDs) into memory during search. For large-scale vector datasets (billions of vectors), this makes the memory footprint prohibitively expensive. Theivf_on_diskapproach stores the inverted list data in a separate file (ann.ivfdata) and reads only the lists needed for each query, backed by an LRU cache for hot data.Changes
BE - Core IVF On-Disk Implementation
New index type
IVF_ON_DISK:AnnIndexTypeenum withIVF_ON_DISKand added string conversion support (be/src/storage/index/ann/ann_index.h,ann_index.cpp)FaissBuildParameter::IndexTypewithIVF_ON_DISK(be/src/storage/index/ann/faiss_ann_index.h)faiss_ivfdata_file_nameconstant (ann.ivfdata) for the separate data file (be/src/storage/index/ann/ann_index_files.h)On-disk save/load in
FaissVectorIndex(faiss_ann_index.cpp):ArrayInvertedListstoOnDiskInvertedListsformat, writes list data toann.ivfdataand index metadata toann.faissann.ivfdatavia aCachedRandomAccessReaderbacked by an LRU cache; replaces the deserializedPreadInvertedListswith a cached reader that provides zero-copyborrow()for repeated list accessesCachedRandomAccessReaderimplementingfaiss::RandomAccessReaderwith per-range LRU caching keyed by(file-prefix, file-size, byte-offset)Dedicated IVF list cache (
be/src/storage/cache/ann_index_ivf_list_cache.h/cpp):AnnIndexIVFListCacheclass — a dedicated LRU cache separated fromStoragePageCacheto avoid contention with column data pagesann_index_ivf_list_cache_limit(default: 70% of physical memory)CachePolicyasANN_INDEX_IVF_LIST_CACHERuntime environment integration:
ExecEnvnow creates/destroys theAnnIndexIVFListCachesingletonann_index_ivf_list_cache_limit,ann_index_ivf_list_cache_stale_sweep_time_secMetrics & profiling:
ann_ivf_on_disk_fetch_page_costs_ms,ann_ivf_on_disk_fetch_page_cnt,ann_ivf_on_disk_search_costs_ms,ann_ivf_on_disk_search_cnt,ann_ivf_on_disk_cache_hit_cnt,ann_ivf_on_disk_cache_miss_cntAnnIndexStatsandOlapReaderStatisticswith IVF on-disk countersAnnIndexReader::query()andrange_search()toSegmentIteratorSearch execution:
AnnIndexReadernow handlesIVF_ON_DISKalongsideIVFfor both top-N and range search pathsScopedIoCtxBindingpropagatesIOContextviathread_localsoCachedRandomAccessReadercan attribute file-cache stats to the correct queryScopedOmpThreadBudgetnow usescondition_variableto properly block waiting index builders instead of silently degradingIndex file writer fix:
IndexFileWriter::add_into_searcher_cache()now correctly skips ANN indexes (both single-file HNSW/IVF and two-file IVF_ON_DISK) by checking forann.faiss/ann.ivfdatafile namesCompound directory lifetime:
AnnIndexReadernow holds_compound_diralive to prevent use-after-free whenCachedRandomAccessReaderholds a clonedCSIndexInputwhose base pointer references the compound reader's streamFE - DDL Validation
AnnIndexPropertiesChecker.java: Acceptivf_on_diskas a valid index type; requirenlistfor bothivfandivf_on_diskFAISS Submodule
contrib/faisssubmodule to a version supportingPreadInvertedListswithRandomAccessReader/borrow()interfaceRegression Tests
ivf_on_disk_index_test.groovy: Tests for L2 distance, inner product, missing nlist error, insufficient training points, larger datasets, range searchcreate_ann_index_test.groovy: Addedivf_on_diskCREATE INDEX test casecreate_tbl_with_ann_index_test.groovy: Added CREATE TABLE withivf_on_disk(L2, IP, missing nlist error)ivf_on_disk_stream_load.csv,ivf_on_disk_stream_load.json,ivf_on_disk_index_test.outUsage