module Memo::USearchIndex

Overview

USearch HNSW index management for fast approximate nearest neighbor search.

Wraps USearch index lifecycle: file naming, open/save/close, type conversion, and vector operations. All Float64↔Float32 conversion happens here.

One index per service (isolated vector spaces). Index files are named by service format, model, and dimensions: memo.openai--text-embedding-3-small--1536.usearch

Extended Modules

Defined in:

memo/usearch_index.cr

Constant Summary

DEFAULT_INDEX_DIR = begin if xdg = ENV["XDG_DATA_HOME"]? File.join(xdg, "memo", "indices") else home = Path.home.to_s if ((home == "/nonexistent") || home.empty?) || !(Dir.exists?(home)) "/var/lib/memo/indices" else File.join(home, ".local", "share", "memo", "indices") end end end

Default directory for index files when no db_path is available (e.g., PostgreSQL backend). Follows XDG Base Directory spec, with fallback to /var/lib/memo/indices for system users.

Instance Method Summary

Instance Method Detail

def add(index : USearch::Index, key : UInt64, embedding : Array(Float64)) #

Add a vector to the index.

Key is the embedding rowid (SQLite rowid or PostgreSQL eid). Embedding is converted from Float64 to Float32 at this boundary.


[View source]
def checkpoint_path(path : String) : String #

Path of the journal checkpoint saved beside an index file.


[View source]
def close(index : USearch::Index, path : String) #

Save and close the index, freeing resources.


[View source]
def delete_file(path : String) #

Delete the index file (and its journal checkpoint and lock) from disk.


[View source]
def exact_search(index : USearch::Index, query : Array(Float64), keys : Enumerable(UInt64), k : Int32) : Array(USearch::SearchResult) #

Exact nearest neighbors among keys, comparing the query with each.

Faster than filtered_search when the keys are few: HNSW explores much of its graph to find matches that sparse.


[View source]
def filtered_search(index : USearch::Index, query : Array(Float64), k : Int32, &filter : UInt64 -> Bool) : Array(USearch::SearchResult) #

Search with a filter predicate on keys.

Only results where the filter block returns true are included. Use this with a Set of valid rowids from SQL pre-filtering.


[View source]
def get_vector(index : USearch::Index, key : UInt64) : Array(Float64) | Nil #

Retrieve a vector from the index by key.

Returns Float64 array for compatibility with the rest of Memo, or nil if the key doesn't exist.


[View source]
def index_path(db_path : String, format : String, model : String, dimensions : Int32) : String #

Build the index file path from a database path.

Path: {db_dir}/{db_stem}.{format}--{model}--{dimensions}.usearch Path-unsafe characters in format/model are replaced with hyphens.


[View source]
def index_path_in_dir(dir : String, format : String, model : String, dimensions : Int32, stem : String = "memo") : String #

Build the index file path from an explicit directory.

Path: {dir}/{stem}.{format}--{model}--{dimensions}.usearch If no stem is provided, uses "memo" as default.


[View source]
def load_or_create(path : String, dimensions : Int32) : Tuple(USearch::Index, Bool) #

Load the index at path, or create an empty one when the file is missing or can't be loaded (corrupt, wrong dimensions). Returns the index and whether it was loaded from the file.


[View source]
def lock(path : String) : File #

Lock the index at path for this process, or raise InUse. Two holders of one index would each keep their own copy in memory and overwrite each other's saves, so a second open fails instead. The lock lasts until the returned file is closed or the process exits (a crash included).


[View source]
def open(path : String, dimensions : Int32) : USearch::Index #

Open or create a USearch index at a specific path.

If the index file exists on disk, loads it. Otherwise creates a new empty index with cosine metric and f16 quantization.


[View source]
def remove(index : USearch::Index, key : UInt64) #

Remove a vector from the index by key.


[View source]
def save(index : USearch::Index, path : String) #

Save the index to disk.

Writes a temporary file and renames it over path, so a crash mid-save leaves the previous file intact.


[View source]
def save_in_background(index : USearch::Index, path : String) #

Save on a separate system thread, suspending only the calling fiber.

Writing a large index takes seconds (it's one blocking call), and on a single-threaded runtime that would freeze every other fiber. Searches can run meanwhile (they only read the index); the caller must keep writers out until this returns.


[View source]
def search(index : USearch::Index, query : Array(Float64), k : Int32) : Array(USearch::SearchResult) #

Search for k nearest neighbors (unfiltered).

Returns USearch::SearchResult array with keys and distances. Cosine distance = 1 - similarity, so similarity = 1 - distance.


[View source]
def to_f32(embedding : Array(Float64)) : Array(Float32) #

Convert Float64 array to Float32 array for USearch.


[View source]