Semantic (vector) and full-text (keyword) search across indexed corpora. Dual-indexes PDFs, markdown, and code into Qdrant + MeiliSearch.
- version
- 0.11.0
- GitHub stars
- 8
Install
/plugin install arc@arcaneum-marketplaceRun it inside Claude Code after adding the marketplace with /plugin marketplace add cwensel/arcaneum.
Contents
Discovered from sourceSkills 5
Qdrant Collection Management
# List all collections
arc collection list
arc collection list --verbose # Show type and vector details
arc collection list --json # JSON output
# Create collection
arc collection create MyCollection --type pdf # PDF collection (arctic-m model)
arc collection create MyCollection --type code # Code collection (jina-code)
arc collection create MyCollection --type markdown # Markdown collection (arctic-m)
arc collection create MyCollection --model qwen3-embed # Explicit model
# View collection info
arc collection info MyCollection
arc collection info MyCollection --json
# List indexed items
arc collection items MyCollection
arc collection items MyCollection --json
# Verify integrity (fsck-like check)
arc collection verify MyCollection
arc collection verify MyCollection --project myrepo#main # Code collections
arc collection verify MyCollection --verbose --json
# Export collection
arc collection export MyCollection -o backup.arcexp
arc collection export MyCollection -o backup.jsonl --format jsonl
arc collection export MyCollection -o shareable.arcexp --detach # Portable
# Import collection
arc collection import backup.arcexp
arc collection import backup.arcexp --into RestoredCollection
arc collection import shareable.arcexp --attach /new/root/path
# Delete collection
arc collection delete MyCollection --confirm
Collection Types
- pdf: Documents and PDFs (arctic-m model, 768D)
- code: Source code repositories (jina-code, 768D)
- markdown: Documentation and notes (arctic-m model, 768D)
Container Management
Manages Qdrant and MeiliSearch Docker containers for local development.
# Start services
arc container start # Start both Qdrant and MeiliSearch
arc container start --service qdrant # Start only Qdrant
arc container start --service meilisearch # Start only MeiliSearch
# Stop services
arc container stop
arc container stop --service qdrant
arc container stop --service meilisearch
# Restart services
arc container restart
arc container restart --service qdrant
# Check status
arc container status
arc container status --json
# View logs
arc container logs # Tail logs from both services
arc container logs --service qdrant # Qdrant logs only
arc container logs --lines 100 # Show more lines
# Reset (delete all data)
arc container reset --confirm # Reset both services
arc container reset --service qdrant --confirm # Reset only Qdrant
Service URLs
After starting:
- Qdrant: http://localhost:6333 (REST API), http://localhost:6334 (gRPC)
- MeiliSearch: http://localhost:7700
Troubleshooting
# Check if Docker is running
docker info
# Check service health
arc container status
# View recent errors
arc container logs --lines 50
# Full reset if needed
arc container reset --confirm
arc container start
Corpus Management (Dual-Index)
A corpus maintains both a Qdrant collection (semantic search) and a MeiliSearch index (full-text search) in sync.
For adding or updating content, always prefer arc corpus sync over arc index, arc collection, or arc indexes.
The corpus commands manage Qdrant and MeiliSearch together. Single-system commands
(arc index, arc collection, arc indexes) are for advanced workflows that need one system without the other.
# Create corpus (creates both collection and index)
arc corpus create MyCorpus --type pdf
arc corpus create MyCorpus --type code
arc corpus create MyCorpus --type markdown
arc corpus create MyCorpus --type pdf --models qwen3-embed,jina-v3 # Multiple models
# Delete corpus (deletes both collection and index)
arc corpus delete MyCorpus # With confirmation prompt
arc corpus delete MyCorpus --confirm # Skip confirmation
arc corpus delete MyCorpus --confirm --json # JSON output
# Sync files to both systems (preferred command for adding/updating content)
arc corpus sync MyCorpus /path/to/files
arc corpus sync MyCorpus /path/one /path/two /path/three # Multiple directories
arc corpus sync MyCorpus /path/to/files --parity # Detect renames, remove files no longer on disk
arc corpus sync MyCorpus /path/to/files --parity --dry-run # Preview parity changes first
arc corpus sync MyCorpus /path/to/files --force # Force reindex
arc corpus sync MyCorpus /path/to/files --verify # Verify after sync
arc corpus sync MyCorpus /path/to/files --verbose # Show progress
arc corpus sync MyCorpus /path/to/files --gpu # Opt into experimental accelerator (CPU is default)
# Keep a git repo in sync automatically (installs a git hook)
arc corpus hook install # Guided: pick/create corpus, choose hooks
arc corpus hook install MyCorpus # Auto-sync the default branch
arc corpus hook install MyCorpus --branch develop # Follow another branch
arc corpus hook install MyCorpus --service # Plus an OS watcher for reboot recovery
arc corpus hook status # What is installed in this repo
arc corpus jobs # Running syncs/drains and queued hook work
arc corpus hook uninstall MyCorpus # Remove just this corpus's hook
# Sync only what a commit touched, without installing a hook
arc corpus sync MyCorpus --changed-since HEAD # Last commit
arc corpus sync MyCorpus --changed-since ORIG_HEAD..HEAD # Since a pull
# Re-extract unchanged files chosen by stored metadata (after an extractor fix);
# `!fallback` selects only PDFs not yet re-extracted since the column-order fix
arc corpus sync MyCorpus --from-index 'method=pymupdf_normalized,!fallback' --dry-run
arc corpus sync MyCorpus --from-index 'method=pymupdf_normalized,!fallback'
arc corpus sync MyCorpus --from-index 'version<0.10.0'
# Check health: content quality plus Qdrant/MeiliSearch parity
arc corpus verify MyCorpus # Human-readable report
arc corpus verify MyCorpus --json # parity_status: synced | mismatch | unknown
arc corpus verify MyCorpus --strict --json # Exit 1 on any issue, including mismatch
# Sweep every corpus (read-only) and get the commands that fix each one
arc corpus doctor # All corpora; lists commands to run
arc corpus doctor MyCorpus OtherCorpus --json # data.commands / data.optional_commands
# Repair incomplete or garbled files, then restore parity
arc corpus repair MyCorpus # Detect and fix quality and parity issues
arc corpus repair MyCorpus --dry-run # Preview what would be repaired
arc corpus repair MyCorpus --include-stale-policy # Also migrate policy-only files
arc corpus repair MyCorpus --prune-orphans # Also delete docs for files gone everywhere
arc corpus repair MyCorpus --quality-threshold 0.5 # More aggressive detection
arc corpus repair MyCorpus --verbose # Show per-file quality scores
# View corpus info (both systems)
arc corpus info MyCorpus
arc corpus info MyCorpus --json
# List indexed items with parity status
arc corpus items MyCorpus # Table output with Q/M chunk counts
arc corpus items MyCorpus --json # JSON output for automation
# Check and restore parity between systems
arc corpus parity MyCorpus # Check and backfill single corpus
arc corpus parity MyCorpus --dry-run # Preview only
arc corpus parity MyCorpus --verify # Verify chunk counts match
arc corpus parity MyCorpus --repair-metadata # Fix missing git metadata (code corpora)
arc corpus parity MyCorpus --prune-orphans # Delete MeiliSearch docs for files gone everywhere
arc corpus parity MyCorpus --force-prune # Same, even if the file's directory is missing
arc corpus parity MyCorpus --verbose # Detailed progress
# All-corpora mode (no corpus name)
arc corpus parity # Process all corpora
arc corpus parity --dry-run # Preview all
arc corpus parity --confirm # Skip confirmation prompt
# Create missing MeiliSearch indexes for qdrant_only corpora
arc corpus parity --create-missing --dry-run # Preview what would be created
arc corpus parity --create-missing --confirm # Create and sync all
When to Use Corpus vs Collection/Index
- Use Corpus: When you need both semantic search (conceptual queries) AND full-text search (exact phrases)
- Use Collection alone: When you only need semantic/conceptual search
- Use Index alone: When you only need exact keyword/phrase search
Re-indexing by Stored Metadata
arc corpus sync <corpus> --from-index SELECTOR re-extracts the indexed files
whose stored metadata matches SELECTOR, even though they are unchanged on disk.
Use it when an extraction fix should reach files indexed before the fix; run
with --dry-run first to see the selection. Terms are comma-separated and must
all match. Shorthand keys: method (extraction method, matching any part of a
combined method), fallback (why the preferred extractor was skipped), ocr
(why OCR ran), warning (a recorded quality warning), and version (arc
version that indexed the file, with <, <=, >, >=). Any other key is a
dotted payload path such as quality_manifest.page_coverage.page_count=3. A
key no chunk records is an error, not an empty selection. !key selects files
whose record lacks the field (quote the selector so the shell leaves !
alone). Missing files are skipped, never deleted.
arc corpus verify reports stale_extraction_files: PDFs extracted before the
column-order fix, whose two-column pages are indexed as interleaved lines.
arc corpus doctor lists the targeted fix,
arc corpus sync <corpus> --from-index 'method=pymupdf_normalized,!fallback';
arc corpus repair also re-extracts them.
Parity Behavior
The --parity flag on arc corpus sync and the standalone arc corpus parity
command both ensure both systems hold the same content, but operate differently:
arc corpus sync --parity <path>: Runs during a sync. In addition to indexing new/changed files in<path>, it detects renamed/moved files by content hash and removes indexed entries for files that no longer exist on disk (scoped to the directories being synced). Identical PDFs share one canonical searchable chunk set; alias rename/removal preserves it, and canonical removal promotes the lexicographically first live alias after dual-index read-back.arc corpus parity: Standalone cross-system repair with no directory scan. Backfills missing entries between Qdrant and MeiliSearch;--verifyalso fixes per-file chunk-count mismatches. It does not remove missing-from-disk files unless--prune-orphansis passed, and then only MeiliSearch documents that nothing in Qdrant references. Files whose directory is also missing are kept (possibly an unmounted volume) unless--force-prune. Holds the corpus write lock;--dry-rundoes not. With no NAME,--jsonprints one summary document.
Both parity commands check differences against the chunks Qdrant stores: empty files (a file record with no chunks) are not drift, and files whose chunks are stored without a file record get the record rebuilt instead of being re-embedded.
Cross-system backfill direction (both commands):
- Qdrant -> MeiliSearch: Copies metadata (fast, no file access needed)
- MeiliSearch -> Qdrant: Re-chunks and embeds files (requires file access)
Creating Missing Indexes:
Use --create-missing to promote single-sided Qdrant collections into full corpora:
- Creates MeiliSearch indexes for
qdrant_onlycollections - Reads corpus type from Qdrant metadata
- Applies appropriate index settings automatically
- Then proceeds with normal parity sync
Note: meili_only corpora cannot be auto-created (require --type and --model).
Selective Repair
arc corpus repair re-indexes only verifier-selected unhealthy files; it is not
a full-corpus force operation. Use arc corpus repair NAME --dry-run --json to
inspect the exact set first. The selection includes incomplete/duplicate chunks,
corrupt or dropped-out PDF extraction, stale source metadata, missing quality
manifests (PDF and code only), and duplicate PDF sources. Policy-only findings stay visible but are
deferred unless --include-stale-policy is passed; files that also have another
repair reason remain selected. Duplicate sources are consolidated without losing
their physical-path aliases. There is no separate --only-corrupt flag because
the default repair path is already selective.
After the content step, repair runs the arc corpus parity --verify pass, so it
also fixes the mismatch that arc corpus verify reports. Both steps share one
corpus lock, and --dry-run previews both. With --json the output is one
document with the parity result under data.parity; a parity failure exits 1.
Pass --prune-orphans to also clear MeiliSearch documents for files gone from
both Qdrant and disk.
Automatic Sync via Git Hook
Without a hook, an indexed repo drifts behind its source tree between manual
syncs. arc corpus hook install <corpus> closes that gap.
How it works:
- The corpus follows one branch: the remote's default unless
--branchnames another. Commits on other branches and in linked worktrees are not indexed. - Whenever that branch moves while checked out (commit, merge, pull, rebase,
reset, or a tool's
update-ref), the hook asks git which paths changed since the last commit it queued (git diff-tree, not a tree walk), queues them, and starts a background worker. A missed event is caught up by the next one. - The worker drains the queue until empty, so a burst of commits pays one embedding-model load instead of one per commit.
- Deleted and renamed-away files are removed from both Qdrant and MeiliSearch.
- The hook never blocks or fails a git operation; it logs to
~/.local/state/arcaneum/hook.log.
Run arc corpus hook install with no corpus name to be walked through it:
it lists existing corpora or offers to create one (inferring the type from the
repo's contents), suggests hook points, and offers the initial backfill (run at
background priority, and only while the followed branch is checked out). Prefer
this when the user has not said which corpus to use. Pass --yes to take the
defaults without prompting.
A hook only indexes future commits, so files already committed need one real
arc corpus sync <corpus> <repo> to get in.
Installing is safe and reversible: the hook block is delimited by marker
comments and appended to any existing hook, so a user's own hook keeps working,
installing twice does not duplicate it, and uninstall removes only its own
block. core.hooksPath is respected, and one repo can feed several corpora.
The default hook points (reference-transaction, post-checkout,
post-rewrite) already cover pulls, checkouts and rebases; --hook picks
others. Hooks installed before branch pinning should be reinstalled. Use
--service to also register an OS watcher
(launchd on macOS, systemd on Linux) that drains work left over after a reboot
or a failed spawn.
For a one-off equivalent with no hook installed, use
arc corpus sync <corpus> --changed-since HEAD.
Concurrent Syncs
Syncs of one corpus are serialized by a per-corpus write lock, so two runs
cannot duplicate entries or drift the two indexes apart. A second sync waits by
default; --no-wait fails immediately instead, and --lock-timeout <seconds>
bounds the wait. Different corpora never block each other, and --dry-run takes
no lock.
Remote Embedding Workers (Optional)
When remote_embedding endpoints are configured in
~/.config/arcaneum/config.yaml, document embedding for selected models runs
on spare worker machines instead of this one; query embedding always stays
local. By default arctic-m goes to the workers and qwen3-embed only on
request. A run that uses them prints Remote embedding: <models> on <workers>
at the start.
arc corpus sync DevRef ~/papers --remote-embed # also send qwen3-embed to the workers
arc corpus repair PapersFast --local-embed # this run entirely local
arc models workers # worker state, versions, resident model
arc models deploy --build # update workers after code changes
Workers must run the same code as this machine; after an upgrade (or a commit,
for a source checkout) they report mismatch until arc models deploy runs.
See docs/guides/remote-embedding-workers.md.
GPU Acceleration and Apple Silicon
Corpus sync embeds on the CPU by default, the stable path. --gpu opts into an
experimental accelerator (MPS on Apple Silicon or CUDA on NVIDIA); it does not
guarantee GPU placement or a speedup, and no accelerator combination is
currently stable.
Large models on Apple Silicon: with --gpu, models like jina-code-1.5b
(1.5B params) may cause system instability on Macs with limited memory. If you
experience lockups:
# Drop --gpu to return to the stable CPU default
arc corpus sync MyCorpus /path
# Or force CPU even when --gpu is passed (e.g. by a script or hook)
ARC_NO_GPU=1 arc corpus sync MyCorpus /path --gpu
# Or use a smaller model (bge is 0.3B params)
arc corpus sync MyCorpus /path --models bge
Model sizes:
bge,bge-base,bge-small: Safe for all systemsstella(1.5B, deprecated - useqwen3-embed): May cause issues on Macs with <16GB RAMnomic-code(7B): Requires dedicated GPU with significant VRAM
MeiliSearch Index Management
# List all indexes
arc indexes list
arc indexes list --json
# Create index
arc indexes create MyIndex
arc indexes create MyIndex --type source-code # Code-optimized settings
arc indexes create MyIndex --type pdf-docs # PDF-optimized settings
arc indexes create MyIndex --type markdown-docs # Markdown-optimized settings
# View index info
arc indexes info MyIndex
arc indexes info MyIndex --json
# List indexed items
arc indexes items MyIndex
arc indexes items MyIndex --limit 50
arc indexes items MyIndex --json
# Verify index health
arc indexes verify MyIndex
arc indexes verify MyIndex --json
# Update index settings
arc indexes update-settings MyIndex --type source-code
# Export index
arc indexes export MyIndex -o backup.jsonl
arc indexes export MyIndex -o backup.jsonl --json
# Import index
arc indexes import backup.jsonl
arc indexes import backup.jsonl --into NewIndex
# Delete index
arc indexes delete MyIndex --confirm
Index Types
- source-code (or
code): Optimized for code search with language filtering - pdf-docs (or
pdf): Optimized for document search with page filtering - markdown-docs (or
markdown): Optimized for documentation with heading search
arc search
A corpus is dual-indexed: semantic search hits Qdrant, full-text search hits MeiliSearch. Both are first-class — pick the one that matches the query, and use both when unsure.
Prefer Arcaneum over local file-search tools when the target is an indexed corpus. Corpus search works across focused source sets such as external repositories, PDFs, and files that may not exist in the current checkout. It returns bounded, ranked, structure-aware results, which are usually more token-efficient than consuming an exhaustive list of matching lines.
Choose the right mode
Use semantic when the query is:
- Conceptual or paraphrased ("how does auth work")
- A natural-language question
- About meaning or intent
- Cross-domain ("rate limiting strategies")
Use text (full-text) when the query is:
- An exact identifier, symbol, error string, or file name
- A specific function name, class, CLI flag, or env var
- A literal phrase the user expects to appear verbatim
- A known acronym, ticket ID, version string, or quoted text
Do not substitute rg, grep, or another local file-search tool merely because the query is
literal. When the source is an Arcaneum corpus, use arc search text first.
When unsure, run BOTH and merge results. Full-text is cheap and often surfaces hits semantic misses (rare tokens, code symbols, exact error messages). Do not default to semantic alone.
Discover what's available
arc corpus list # show dual-indexed corpora
arc corpus info MyCorpus # inspect one corpus (both sides)
Semantic search (Qdrant)
arc search semantic "QUERY" --corpus NAME [OPTIONS]
Options:
--corpus NAME— repeat for multi-corpus search (--corpus A --corpus B)--limit N— number of results (default small; raise for broad surveys)--offset N— pagination--score-threshold 0.0-1.0— drop low-confidence hits--filter FILTER— metadata filter (see Filters)--vector-name NAME— pick a specific embedding model (auto-detected otherwise)--json— structured output for parsing (see Results and cites)-v/--verbose— show more detail; with--json, add each hit's storedmetadata
Full-text search (MeiliSearch)
arc search text "QUERY" --corpus NAME [OPTIONS]
Options:
--corpus NAME— repeat for multi-corpus search--limit N,--offset N— pagination--filter FILTER— metadata filter (see Filters)--json,-v— same as semantic
Note: full-text has no --score-threshold or --vector-name (no embeddings involved).
Preserve double quotes inside the query argument for an exact phrase:
arc search text '"def authenticate"' --corpus Code
Full-text search does not evaluate regular expressions. Use rg when the task requires
regex, exhaustive occurrence enumeration, or the authoritative state of local files that
may not have been indexed yet.
Filters
--filter takes the same syntax in both modes:
key=value, with terms joined by,(all must match):--filter "filename=db(31).pdf,page_number:gte:80"key:op:value, where op ismatch,in,gte,gt,lte,ltorcontains. Separateinvalues with|:--filter "language:in:python|java"- JSON
{"field": value}, where a list meansin:--filter '{"filename": "db(31).pdf"}'
Use the JSON form when a value contains ,, ", <, >, != or an upper-case AND, OR, NOT,
TO, IN or EXISTS. contains works in semantic only.
A field the corpus can't filter on fails with bad-filter and lists the fields it can filter on.
For PDF corpora these are filename, file_path and page_number, among others. For markdown corpora
they are filename, file_path, section and tags. In code corpora, semantic search names
language, project and branch as programming_language, git_project_name and git_branch.
Advanced: each backend's native syntax also works, but only in its own mode. Text search takes a
MeiliSearch expression ("page_number > 80 AND filename = \"db(31).pdf\""). Semantic search takes
Qdrant JSON with a top-level must, should or must_not.
Results and cites
Every output format prints each hit's cite. Both modes' --json share one schema:
data.hits[] holds corpus, chunk_id, locator, cite, score, snippet and
snippet_truncated. The snippet is 500 characters unless --max-content-chars N says
otherwise (0 is unlimited).
A cite is <corpus>:<path> plus #page=N for a PDF, #<heading-slug> for markdown, or
#L<start>-<end> for a code hit that carries a line range (semantic code hits cite the path
alone). The path is relative to the corpus's sync root, or absolute with ~ for the home
directory until a sync records one. #, % and space in the path appear as %23, %25 and
%20. Example: DevRef:Database-Books/db(31).pdf#page=89. Quote a cite in backticks when
citing a hit; chunk_id names the chunk itself.
Scores compare only within one mode, and text scores only within one corpus: multi-corpus text results interleave each corpus's ranking rather than sorting by score.
Common patterns
# Find a function by name — full-text wins
arc search text "parse_frontmatter" --corpus Code
# Find an exact phrase — preserve the inner double quotes
arc search text '"def authenticate"' --corpus Code
# Conceptual question — semantic wins
arc search semantic "how is the embedding cache invalidated" --corpus Code
# Unknown — run both
arc search semantic "retry backoff" --corpus Docs --limit 5
arc search text "retry backoff" --corpus Docs --limit 5
# Multi-corpus
arc search semantic "auth flow" --corpus Code --corpus Docs
# Filtered (e.g. one PDF, from page 80 on): the same filter in both modes
arc search text "transaction" --corpus DevRef --filter "filename=db(31).pdf,page_number:gte:80" --json
arc search semantic "transaction isolation" --corpus DevRef --filter '{"filename": "db(31).pdf"}' --json
Subcommand placement
The subcommand (semantic | text) MUST come before the query:
- ✅
arc search semantic "query" --corpus X - ❌
arc search "query" --corpus X --semantic - ❌
arc search --corpus X "query"
Commands 10
Manage Qdrant vector collections for storing embeddings.
Note: For most users, /arc:corpus is recommended as it provides both semantic and full-text
search. Use /arc:collection when you only need semantic search.
IMPORTANT: You must specify a subcommand (create, list, info, delete, items, verify, export, or import).
Subcommands (required):
create: Create a new collection with specified embedding modellist: List all collections in Qdrantinfo: Show detailed information about a collectiondelete: Delete a collection permanentlyitems: List indexed files/repos in a collectionverify: Verify collection integrity (fsck-like check)export: Export collection to a portable formatimport: Import collection from an export file
Common Options:
- --json: Output in JSON format for programmatic use
Examples:
/collection create MyDocs --type pdf
/collection create MyDocsQuality --model stella --type pdf
/collection list
/collection info MyDocs
/collection delete MyDocs --confirm
Execution:
arc collection $ARGUMENTS
Collection Types:
Collections should specify a type (pdf or code) to ensure content matches:
- pdf: For document collections (PDFs, text files)
- code: For source code repositories
- markdown: For markdown/document collections
Available Models:
- arctic-m: 768D, stable document/PDF/markdown default
- stella: 1024D, high-quality opt-in document model (requires
arcaneum[sentence-transformers]) - mxbai-large: 1024D, high-quality FastEmbed document model
- jina-code: 768D, stable FastEmbed source-code default
- bge: 1024D, general purpose
Related Commands:
- /arc:corpus create - Create corpus for dual indexing (recommended)
- /arc:corpus sync - Index to both systems (recommended)
- /arc:index pdf - Index PDFs into a collection (semantic only)
- /arc:index code - Index source code into a collection (semantic only)
Implementation:
- Defined in RDR-003 (Collection Creation)
- Enhanced in RDR-006 (Claude Code Integration)
- Type enforcement added in arcaneum-122
Manage Arcaneum configuration and model cache.
Subcommands:
- show-cache-dir: Display cache locations and sizes
- clear-cache: Clear model cache to free disk space
Arguments:
- --confirm: Confirm cache deletion (required for clear-cache)
- --json: Emit a machine-readable JSON envelope on stdout
Examples:
/config show-cache-dir
/config show-cache-dir --json
/config clear-cache --confirm
Execution:
arc config $ARGUMENTS
Note: The model cache stores downloaded embedding models in XDG-compliant locations. First-time indexing downloads ~1-2GB of models which are then reused. I'll show you:
- Current cache directory locations
- Size of each directory
- Free disk space information
Use clear-cache when models are corrupted or to free disk space (models will be re-downloaded on next use).
Directory Locations (XDG-compliant):
- Models (cache):
~/.cache/arcaneum/models - Qdrant data: Docker volume
qdrant-arcaneum-storage
Related:
- Implemented in arcaneum-157 and arcaneum-162
Manage Docker container services for Qdrant and MeiliSearch.
Subcommands:
- start: Start all services
- stop: Stop all services
- status: Show service status and health
- logs: View service logs
- restart: Restart services
- reset: Delete all data and reset (WARNING: destructive)
- backup: Back up Qdrant snapshots and MeiliSearch indexes
- restore: Restore Qdrant snapshots and MeiliSearch indexes from a backup
- upgrade: Back up MeiliSearch, then migrate its data to the pinned image version
Arguments:
- --follow, -f: Follow log output (logs command only)
- --tail : Number of log lines to show (logs command, default: 100)
- --confirm: Confirm data deletion (reset command only)
- --dry-run: Show the upgrade plan without changing anything (upgrade command only)
- --json: Emit a machine-readable JSON envelope on stdout
Examples:
/container start
/container status
/container status --json
/container logs
/container logs --follow
/container stop
/container restart
/container reset --confirm
/container upgrade --dry-run
Execution:
arc container $ARGUMENTS
Note: Container management commands check Docker availability and provide helpful error messages if Docker is not running. I'll show you:
- Service startup confirmation with URLs
- Health check status for Qdrant
- Data directory locations and sizes
- Log output for debugging
Data Locations (Docker Volumes):
- Qdrant storage:
qdrant-arcaneum-storage - Qdrant snapshots:
qdrant-arcaneum-snapshots - All data persists across container restarts
- Use
docker volume ls --filter name=arcaneumto view volumes
Related:
- Implemented in arcaneum-167, renamed in arcaneum-169
- Replaces old scripts/qdrant-manage.sh
- Part of simplified Docker management (arcaneum-158)
Recommended for most users. Manage corpora that combine both vector search (Qdrant) and full-text search (MeiliSearch) for the same content.
IMPORTANT: You must specify a subcommand.
Subcommands:
create: Create both Qdrant collection and MeiliSearch indexlist: List all corpora with parity statusupdate: Update corpus metadata without reindexingdelete: Delete both Qdrant collection and MeiliSearch indexsync: Index directory to both systems simultaneouslyrepair: Selectively repair verifier-selected unhealthy files and PDF duplicates, then restore parityhook: Install a git hook that auto-syncs a repo on every commitjobs: List running syncs, repairs, and hook drains, plus queued hook work (read-only; takes no locks)info: Show corpus details (both systems)items: List indexed items with parity statusparity: Check and restore parity between systemsverify: Verify corpus health across both Qdrant and MeiliSearchdoctor: Check all corpora (or the named ones) read-only and list the commands that fix each problem
Common Options:
- --json: Output in JSON format
- --details: Show extended list columns, including exact item counts
Create Options:
- name: Corpus name (required)
- --type: Corpus type - code, pdf, or markdown (required)
- --models: Embedding models, comma-separated (default inferred from --type: arctic-m for pdf/markdown, jina-code for code)
Delete Options:
- name: Corpus name (required)
- --confirm: Skip confirmation prompt
- --json: Output in JSON format
Sync Options:
- name: Corpus name (required, first positional argument)
- directories: One or more directory paths to index (required)
- --models: Embedding models (default: use corpus metadata)
- --file-types: File extensions to index (e.g., .py,.md)
- --gpu: Opt into accelerator embedding (CPU is the stable default)
- --remote-embed: Also send on-request models (e.g. qwen3-embed) to the configured remote embedding workers for this run
- --local-embed: Embed every model locally for this run, ignoring configured remote embedding workers
- --background: Run below interactive work (compilers, tests) for continuous indexing; nice 10 on macOS, SCHED_IDLE on Linux
- --changed-since REV: Sync only what a git commit or range touched (e.g. HEAD, ORIG_HEAD..HEAD), removing the files it deleted
- --order: Process files by path (default), newest modification first, or oldest modification first; this changes processing order, not search ranking
- --include-stale-policy: Also re-index otherwise unchanged files whose indexing policy is stale or missing
- --from-index SELECTOR: Instead of paths, re-extract the indexed files whose
stored metadata matches SELECTOR (e.g. method=pymupdf_normalized,
version<0.10.0); shorthand keys are method, fallback, ocr, warning, version;
!keyselects files that do not record key - --no-wait: Fail instead of queueing when another sync of this corpus is running
- --lock-timeout: Seconds to wait for the corpus write lock (default: 600)
Repair Options:
- name: Corpus name (required)
- --quality-threshold: Text quality threshold (0.0-1.0, default: 0.9)
- --dry-run: Preview what would be repaired without making changes
- --include-stale-policy: Also re-index files whose only issue is stale or missing indexing policy
- --gpu: Opt into accelerator embedding (CPU is the stable default)
- --remote-embed / --local-embed: Route this repair's document embedding as for sync (remote embedding workers)
- --background: Run below interactive work, as for sync
- --verbose: Show per-file quality scores and details
- --json: Output in JSON format
Hook Options:
hook takes its own subcommand: install, uninstall, or status.
- name: Corpus name. Omit it on
installto be walked through picking or creating a corpus, choosing hook points, and backfilling. - --repo: Repository to act on (default: current directory)
- --branch: (install) Branch to follow - the remote's default branch unless named; other branches and worktrees are never indexed
- --hook: Hook point, repeatable - reference-transaction, post-commit, post-merge, post-checkout, or post-rewrite. Install defaults to reference-transaction, post-checkout and post-rewrite; uninstall defaults to all of them.
- --no-spawn: (install) Queue touched paths but start no background worker
- --yes/-y: (install) Take the defaults instead of prompting
- --service: Also register/remove an OS watcher that drains the spool after a reboot or a failed spawn
- --json: Output in JSON format
Info/Items Options:
- name: Corpus name (required)
- --json: Output in JSON format
Parity Options:
- name: Corpus name (optional - if omitted, processes all corpora)
- --dry-run: Preview what would be backfilled without making changes
- --verify: Verify chunk counts match between systems
- --repair-metadata: Update MeiliSearch docs with missing git metadata (code corpora)
- --create-missing: Create missing MeiliSearch indexes for qdrant_only corpora
- --confirm: Skip confirmation prompt when processing all corpora
- --prune-orphans: Delete MeiliSearch documents for files gone from both Qdrant and disk (files whose directory is missing are kept: possible unmounted volume)
- --force-prune: Like --prune-orphans, but also prune files whose directory is missing
- --verbose: Show detailed progress
- --json: Output in JSON format
Doctor Options:
- names: Corpus names (optional - if omitted, checks all corpora)
- --verbose: Also show informational findings and counts
- --strict: Exit 1 when any corpus needs action
- --json: Output in JSON format (one document;
data.commandslists what to run)
/corpus doctor writes nothing. Required fixes are combined per corpus (for
example arc corpus repair NAME --prune-orphans); policy migration and pruning
files whose directory is missing are listed separately as optional. PDFs whose
only defect is extraction from before the column-order fix get a targeted
arc corpus sync NAME --from-index 'method=pymupdf_normalized,!fallback'.
Repair Behavior:
/corpus repair NAME re-indexes only files selected by corpus verification,
including corrupt PDF extraction, incomplete or duplicate chunks, and duplicate
PDF sources. Files whose only finding is stale or missing indexing policy are
reported but deferred by default; pass --include-stale-policy to migrate them.
Files that also have an integrity or quality defect remain selected without the
flag. Preview the exact set with /corpus repair NAME --dry-run --json.
Identical PDFs retain every physical path as an alias of one searchable canonical
document; parity cleanup promotes a live alias if the canonical path disappears.
Repair then restores Qdrant/MeiliSearch parity (as /corpus parity NAME --verify
does), so it also fixes the mismatch that /corpus verify reports. Under
--json the parity result is nested at data.parity. MeiliSearch documents for
files gone from both Qdrant and disk are only reported; pass --prune-orphans
(or --force-prune) to delete them.
Examples:
/corpus create MyDocs --type pdf
/corpus create MyDocsQuality --type pdf --models stella
/corpus sync MyDocs ~/Documents
/corpus create CodeBase --type code
/corpus sync CodeBase ~/projects --file-types .py,.js,.md
/corpus sync CodeBase ~/project1 ~/project2 ~/project3
/corpus repair PapersFast
/corpus repair PapersFast --dry-run
/corpus repair PapersFast --include-stale-policy
/corpus repair PapersFast --quality-threshold 0.5
/corpus sync CodeBase --changed-since HEAD
/corpus sync PapersFast --from-index method=pymupdf_normalized --dry-run
/corpus sync CodeBase ~/projects --order newest
/corpus hook install
/corpus hook install CodeBase
/corpus hook install CodeBase --branch develop
/corpus hook status
/corpus hook uninstall CodeBase
/corpus info MyDocs
/corpus items CodeBase
/corpus parity CodeBase --verify
/corpus parity CodeBase --repair-metadata
/corpus parity --create-missing --dry-run
/corpus parity --create-missing --confirm
/corpus delete OldCorpus
/corpus delete OldCorpus --confirm
Execution:
arc corpus $ARGUMENTS
What Is a Corpus?
A corpus combines two search systems:
- Vector search (Qdrant): Semantic similarity, concept matching
- Full-text search (MeiliSearch): Keyword, phrase, boolean operators
This enables hybrid search strategies:
- Broad semantic discovery (vector search)
- Precise keyword refinement (full-text search)
- Combined results for best of both worlds
When to Use Corpus vs Collection:
Use Corpus (Recommended):
- Default choice for most use cases
- Need both semantic and keyword search
- Users search different ways (concepts vs exact terms)
- Want fast keyword filtering of semantic results
- Building search UIs with multiple search modes
Use Collection (Advanced):
- Only need semantic search (no full-text)
- Working with embeddings/vectors directly
- Integrating with existing vector workflows
- MeiliSearch not available/needed
How Sync Works:
- Discovers files in directory (respects .gitignore for code)
- Chunks content appropriately (PDFs vs code)
- Generates embeddings with specified models
- Uploads to Qdrant (vector search)
- Indexes to MeiliSearch (full-text search)
- Both indexes share same document IDs and metadata
Keeping a Repo in Sync Automatically:
/corpus hook install NAME installs git hooks so an indexed repo stops
drifting behind its source tree between manual syncs. The corpus follows one
branch (the remote's default unless --branch names another). Whenever that
branch moves while checked out (commit, merge, pull, rebase, reset, or a tool's
update-ref), the hook asks git which paths changed since the last commit it
queued, queues them, and indexes them in the background, so a burst of commits
pays one embedding-model load instead of one per commit. It never blocks or
fails a git operation.
Run /corpus hook install with no corpus name for a guided setup: it lists
existing corpora or offers to create one (inferring the type from the repo's
contents), suggests hook points, and offers the initial backfill at background
priority. A hook only sees future commits, so already-committed files need one
real sync.
/corpus sync NAME --changed-since HEAD does the same thing on demand, without
installing anything.
Changed paths are written to a durable spool, so a failed or interrupted worker
does not lose queued work. Git operations never wait for indexing. Sync writes
to one corpus are serialized, while different corpora can proceed concurrently.
Hook and drain outcomes are recorded in
~/.local/state/arcaneum/hook.log.
Install the optional arcaneum[proctitle] extra to make background workers
appear as readable arc ... commands in ps and top.
Performance:
Corpus sync is approximately 2x slower than single-system indexing due to dual upload, but still efficient:
- PDFs: ~5-15/minute
- Source files: 50-100 files/second
Related Commands:
- /arc:search semantic - Search vector index
- /arc:search text - Search full-text index
- /arc:collection create - Create vector-only collection (advanced)
- /arc:indexes create - Create full-text index only (advanced)
- /arc:index pdf - Index PDFs to vector only (advanced)
- /arc:index code - Index code to vector only (advanced)
Implementation:
- RDR-009: Dual indexing strategy
- RDR-006: Claude Code integration
Check that all Arcaneum prerequisites are met and the system is ready for use.
Arguments:
- --verbose: Show detailed diagnostic information
- --json: Output JSON format
Checks Performed:
- Python version (>= 3.12 required)
- Required Python dependencies installed
- Qdrant server connectivity and health
- MeiliSearch server connectivity (if configured)
- Embedding model availability
- Write permissions for temporary files
- Environment variable configuration
Examples:
/doctor
/doctor --verbose
/doctor --json
Execution:
arc doctor $ARGUMENTS
Note: This diagnostic command helps troubleshoot setup issues. I'll run all checks and present a summary showing which requirements are met (✅) and which need attention (❌), along with specific guidance for fixing any problems found.
These checks cover setup only, not indexed content. To check corpora for
needed repairs and get the commands that fix them, use /corpus doctor.
Index PDFs, markdown, or source code into Qdrant collections for semantic search.
Note: For most users, /arc:corpus sync is recommended as it indexes to both semantic and
full-text search. Use /arc:index when you only need semantic search in a single collection.
IMPORTANT: You must specify a subcommand (pdf, code, markdown, or text) before the path.
Subcommands (required):
pdf: Index PDF documents into Qdrant (with OCR support)markdown: Index plain or zstd-compressed markdown into Qdrant (with frontmatter extraction)code: Index source code repositories into Qdrant (git-aware)text: Index content to MeiliSearch for full-text search (has its ownpdf/code/markdownsubcommands; advanced — prefer/arc:corpus syncor/arc:indexesfor full-text)
Common Options:
- --collection: Target collection (required)
- --from-file: Read file paths from list (one per line, or "-" for stdin)
- --model: Embedding model (auto-selected by content type)
- --workers: Parallel workers (default: 4)
- --force: Force reindex all files
- --randomize: Randomize file processing order (useful for parallel indexing)
- --gpu: Opt into accelerator embedding (CPU is the stable default)
- --streaming: Stream embeddings to Qdrant immediately (lower memory usage)
- --verbose: Show detailed progress (suppress library warnings)
- --debug: Show all library warnings including transformers
- --json: Output in JSON format
PDF Indexing Options:
- --no-ocr: Disable OCR (enabled by default for scanned PDFs)
- --ocr-language: OCR language code (default: eng)
- --ocr-workers: Parallel OCR workers (default: cpu_count)
- --normalize-only: Skip markdown conversion, only normalize whitespace
- --preserve-images: Extract images for multimodal search
- --process-priority: Process scheduling priority (low, normal, high)
- --embedding-batch-size: Batch size for embeddings (auto-tuned if not specified)
- --offline: Use cached models only (no network)
Markdown Indexing Options:
- --chunk-size: Target chunk size in tokens (overrides model default)
- --chunk-overlap: Overlap between chunks in tokens
- --recursive/--no-recursive: Search subdirectories recursively (default: recursive)
- --exclude: Patterns to exclude (e.g., node_modules, .obsidian)
- --offline: Use cached models only (no network)
Source Code Indexing Options:
- --depth: Git discovery depth (traverse subdirectories)
Examples:
# Basic indexing (CPU-first stable defaults)
/index pdf ~/Documents/Research --collection PDFs --model arctic-m
/index markdown ~/notes --collection Notes --model arctic-m
/index code ~/projects/myapp --collection MyCode --model jina-code
# Index from file list
/index pdf --from-file /path/to/pdf_list.txt --collection PDFs
/index markdown --from-file /path/to/md_list.txt --collection Notes
# Index from stdin (pipe file paths)
find ~/Documents -name "*.pdf" | /index pdf --from-file - --collection PDFs
ls ~/notes/*.md | /index markdown --from-file - --collection Notes
# With options
/index markdown ~/docs --collection Docs --chunk-size 512 --verbose
/index pdf ~/scanned-docs --collection Scans --no-ocr --offline
# Opt into accelerator embedding
/index pdf ~/Documents/Research --collection PDFs --model stella --gpu
# Streaming mode (lower memory for large collections)
/index pdf ~/Documents/Research --collection PDFs --streaming
/index code ~/projects/myapp --collection MyCode --streaming
# Parallel indexing from multiple terminals (randomize order)
/index pdf ~/Documents/Research --collection PDFs --randomize
/index markdown ~/notes --collection Notes --randomize
# Debug mode (show all warnings)
/index pdf ~/Documents/Research --collection PDFs --model stella --debug
Execution:
arc index $ARGUMENTS
File List Format (--from-file):
When using --from-file, provide a text file with one file path per line:
# Comments are supported (lines starting with #)
/absolute/path/to/file1.pdf
relative/path/to/file2.md
/another/file3.pdf
# Empty lines are ignored
Features:
- Supports both absolute and relative paths
- Relative paths resolved from current directory
- Comments (lines starting with #) and empty lines are skipped
- Non-existent files are warned about but processing continues
- Wrong file extensions are filtered with warnings
- Use "-" to read from stdin
How It Works:
PDF Indexing:
- Extract text from PDFs (PyMuPDF + pdfplumber fallback)
- Auto-trigger OCR for scanned PDFs (< 100 chars extracted)
- Chunk text with 15% overlap for context
- Generate embeddings (arctic-m default: 768D for documents)
- Upload to Qdrant with metadata (file path, page numbers)
- Incremental: Skips unchanged files (file hash metadata check)
Markdown Indexing:
- Discover markdown files (
.md,.markdown,.mdown, and compressed.md.zstvariants) - Extract YAML frontmatter (title, author, tags, category, etc.)
- Semantic chunking preserving document structure (headers, code blocks) and merging sub-200-character residue into neighboring chunks
- Generate embeddings (arctic-m default: 768D for documents)
- Upload to Qdrant with metadata (file path, frontmatter fields, header context)
- Incremental: Skips unchanged files (SHA256 content hash check)
Source Code Indexing:
- Discover git repositories in directory tree
- Extract git metadata (project, branch, commit)
- Parse code with tree-sitter (AST-aware chunking, 15+ languages)
- Generate embeddings (jina-code default: 768D for code)
- Upload to Qdrant with metadata (git info, file path, language)
- Multi-branch support: project#branch identifier
- Incremental: Skips unchanged commits (metadata-based sync)
Default Models:
- PDFs: arctic-m (768D, stable FastEmbed document model)
- Markdown: arctic-m (768D, stable FastEmbed document model)
- Source: jina-code (768D, code-optimized)
Performance: Measure the selected model, corpus, extraction workload, and
hardware with the repository benchmark tools. --gpu requests an experimental
eligible backend; it does not guarantee accelerator placement or improved
throughput.
GPU Acceleration:
CPU embedding is the stable default. --gpu requests an eligible experimental
backend: PyTorch MPS or CUDA for SentenceTransformers, or ONNX Runtime CoreML for
FastEmbed on Apple Silicon. CoreML placement may be hybrid CPU/CoreML. MLX is not
available. PDF layout inference runs in its own process and is unrelated to the
embedding backend. See docs/guides/accelerators.md for current capability states,
worker timeout/fallback behavior, verbose diagnostics, and benchmark evidence.
Offline Mode:
Use --offline for corporate proxies or SSL issues:
- Requires the embedding model already cached. Models download automatically on first use, so run any index/sync once while online to populate the cache
- No network calls during indexing
- Fails if model not cached
Related Commands:
- /arc:corpus sync - Index to both systems (recommended)
- /arc:corpus create - Create corpus for dual indexing
- /arc:collection create - Create collection before indexing (semantic only)
- /arc:search semantic - Search indexed content
Debug Mode:
Use --debug to troubleshoot indexing issues:
- Shows all library warnings (including HuggingFace transformers)
- Displays detailed stack traces
- Helps diagnose model loading or GPU issues
- Use --verbose for user-facing progress without library warnings
Implementation:
- RDR-004: PDF bulk indexing
- RDR-005: Git-aware source code indexing
- RDR-013: Performance optimization with GPU acceleration
- RDR-014: Markdown content indexing
- RDR-006: Claude Code integration
Manage MeiliSearch full-text indexes (mirrors arc collection for Qdrant).
Note: For most users, /arc:corpus is recommended as it provides both semantic and full-text
search. Use /arc:indexes when you only need full-text search.
IMPORTANT: You must specify a subcommand (create, list, info, delete, items, verify, export, or import).
Subcommands (required):
create: Create a new index with type-specific settingslist: List all indexes in MeiliSearchinfo: Show detailed information about an indexdelete: Delete an index permanentlyitems: List indexed files/documents in an indexverify: Verify index health and integrityexport: Export index to JSONL fileimport: Import index from JSONL fileupdate-settings: Update index settings from preset typelist-projects: List git projects in code indexdelete-project: Delete a git project from code index
Common Options:
- --json: Output in JSON format for programmatic use
Examples:
/indexes create MyDocs --type pdf
/indexes list
/indexes info MyDocs
/indexes items MyDocs
/indexes verify MyDocs
/indexes export MyDocs -o backup.jsonl
/indexes import backup.jsonl --into MyDocs-restored
/indexes delete MyDocs --confirm
Execution:
arc indexes $ARGUMENTS
Index Types:
pdf(alias:pdf-docs): PDF documents with stop wordscode(alias:source-code): Source code with higher typo thresholdsmarkdown(alias:markdown-docs): Markdown with headings search
Git Project Commands (for code indexes):
/indexes list-projects MyCode
/indexes delete-project arcaneum#main --index MyCode
Related Commands:
- /arc:corpus create - Create corpus for dual indexing (recommended)
- /arc:corpus sync - Index to both systems (recommended)
- /arc:collection - Manage Qdrant collections (semantic search only)
- /arc:search text - Full-text search in MeiliSearch indexes
Implementation:
- Defined in RDR-008 (Full-Text Search Server Setup)
- Enhanced in RDR-011 (Source Code Full-Text Indexing)
- Enhanced in RDR-012 (Full-Text Search Claude Integration)
Manage and view available embedding models for vector search, and the optional remote embedding workers that embed documents on spare machines.
IMPORTANT: You must specify a subcommand.
Subcommands (required):
list: List all available embedding models with detailsworkers: Show each configured remote embedding worker's state, version, code fingerprint, resident model and dependency versionsdeploy [HOSTS...]: Install, upgrade or reconfigure remote embedding workers over SSH, then check their vectors match this workstation'sstop [HOSTS...]: Stop remote embedding workers over SSH, cancelling any encode in progress and freeing their CPU and memory untilstartstart [HOSTS...]: Start stopped remote embedding workers and wait until they answerwake [HOSTS...]: Wake sleeping remote embedding workers with a Wake-on-LAN magic packet and wait until they answer (syncs also wake them)serve: Run a remote embedding worker (normally inside the containerdeploycreates; not for use on the workstation)
Options:
- --json: Output in JSON format (
list,workers) - --dry-run: Show each worker's current and desired state without changes
(
deploy), or which workers would stop, start or be woken (stop,start,wake) - --no-wait: Send the wake packets without waiting (
wake) - --timeout SECONDS: How long to wait for each woken worker (
wake) - --build: Deploy a wheel built from this source tree (
deploy; needed for source checkouts and unreleased builds) - --wheel PATH: Deploy a given wheel (
deploy) - --refresh: Rebuild the worker image without cache (
deploy) - --no-verify: Skip the vector parity check (
deploy)
Examples:
/models list
/models list --json
/models workers
/models deploy --dry-run
/models deploy --build
/models stop dev-0
/models start
/models wake dev-0
Remote embedding workers are optional and configured under remote_embedding
in ~/.config/arcaneum/config.yaml; see
docs/guides/remote-embedding-workers.md. Query embedding always stays local.
Execution:
arc models $ARGUMENTS
Available Models:
The list command shows:
- Model name (for --model flags)
- Dimensions (vector size)
- Backend (fastembed, sentence-transformers)
- Best use case (PDFs, code, general)
- Model ID (HuggingFace identifier)
Current Models:
For Documents/PDFs:
- arctic-m (768D): DEFAULT - stable FastEmbed retrieval model
- stella (1024D): Highest-quality opt-in document model, requires
arcaneum[sentence-transformers] - mxbai-large (1024D): High-quality FastEmbed document model
- bge-large (1024D): Legacy BGE document model
For Source Code:
- jina-code (768D): DEFAULT - stable FastEmbed code model
- jina-code-st (768D): Legacy SentenceTransformers code path, requires
arcaneum[sentence-transformers] - jina-code-0.5b (896D): Higher-quality opt-in code model, 32K context, requires
arcaneum[sentence-transformers] - jina-code-1.5b (1536D): SOTA Sept 2025, 32K context, highest quality, requires
arcaneum[sentence-transformers] - codesage-large (1024D): CodeSage V2, Dec 2024, 9 languages, requires
arcaneum[sentence-transformers] - nomic-code (3584D): 7B params, 6 languages, slower but comprehensive, requires
arcaneum[sentence-transformers]
For General Use:
- bge (1024D): High-quality general embeddings
- bge-small (384D): Faster, smaller, lower quality
Model Selection Tips:
-
Match content type:
- PDFs/docs → arctic-m (stable default), stella (quality), or mxbai-large (FastEmbed quality)
- Source code → jina-code (stable FastEmbed default), jina-code-0.5b or jina-code-1.5b (quality extra)
- Mixed → arctic-m or mxbai-large
-
Consider dimensions:
- Higher dimensions (1024D) = better quality, more storage
- Lower dimensions (384D, 768D) = faster, less storage
-
Backend matters:
- fastembed: Faster, optimized, limited models
- sentence-transformers: More models, HuggingFace ecosystem
-
Collection consistency:
- Use same model for all documents in a collection
- Cannot mix dimensions in one vector space
Downloading Models:
Models auto-download on first use. FastEmbed defaults are part of the core install;
SentenceTransformers models require arcaneum[sentence-transformers].
- Cached in
~/.cache/arcaneum/models(XDG-compliant) - Reused across indexing operations
- Use --offline flag to require cached models
Pre-download for offline use:
# Download an opt-in higher-quality code model
python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('jinaai/jina-code-embeddings-0.5b')"
# Or the legacy v2 SentenceTransformers model
python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('jinaai/jina-embeddings-v2-base-code')"
Related Commands:
- /collection create - Create collection with specific model
- /index pdf - Index with model selection
- /index code - Index with model selection
Implementation:
- RDR-002: Embedding client architecture
- RDR-006: Model listing CLI
- arcaneum-142: Multi-backend support
Search indexed content using semantic or full-text search. Both modes are first-class: choose the one that matches the query, and use both when unsure.
Quick Start:
# Conceptual or paraphrased query
arc search semantic "your query here" --corpus CorpusName
# Exact phrase (the inner quotes are part of the query)
arc search text '"def authenticate"' --corpus CorpusName
IMPORTANT: The subcommand (semantic or text) comes BEFORE the query.
Subcommands (required):
semantic: Vector-based semantic search (Qdrant)text: Keyword-based full-text search (MeiliSearch)
Examples:
# Conceptual search
/arc:search semantic "identity proofing" --corpus Standards
# Exact identifier
/arc:search text "authenticate_user" --corpus MyCode
# Exact phrase
/arc:search text '"def authenticate"' --corpus MyCode
# Unsure: run both and combine the useful results
/arc:search semantic "retry backoff" --corpus Docs --limit 5
/arc:search text "retry backoff" --corpus Docs --limit 5
Common Options:
- --corpus: Corpus/collection to search (can specify multiple times)
- --limit: Number of results to return (default: 10)
- --offset: Number of results to skip for pagination (default: 0)
- --filter: Metadata filter (key=value or JSON)
- --json: Output in JSON format
- --verbose: Show detailed information
Semantic Search Options:
- --vector-name: Vector name (auto-detected if not specified)
- --score-threshold: Minimum similarity score
Execution:
arc search $ARGUMENTS
When to Use Each:
Semantic Search (vector-based):
- Finding conceptually similar code/documents
- Cross-language semantic matching
- "What does this" or "How to" questions
- Fuzzy concept matching
Full-Text Search (keyword-based):
- Keyword and identifier lookup
- Function/variable name search
- Quoted phrase search
- Typo-tolerant keyword retrieval
Why use full-text corpus search instead of grep?
- It searches focused corpora, reducing unrelated matches.
- Sources may be PDFs, external repositories, or files absent from the local checkout.
- Results are bounded, ranked, and structure-aware, which is usually more token-efficient than consuming exhaustive matching lines.
- Use
rgor grep instead for regular expressions, exhaustive occurrence enumeration, or the authoritative state of local files that may not have been indexed yet.
Result Format:
Both commands show, for each hit in ranked order:
- A cite,
<corpus>:<path>plus#page=N(PDF),#<heading-slug>(markdown) or#L<start>-<end>(code with a line range), e.g.DevRef:Database-Books/db(31).pdf#page=89. Quote it in backticks to cite a hit. - The score: semantic similarity, or MeiliSearch's ranking score for text (text search prints it in compact and JSON output only)
- A bounded content snippet
--json gives both modes one schema: data.hits[] with corpus, chunk_id,
locator, cite, score, snippet and snippet_truncated; --verbose adds
the stored metadata. See docs/reference/cli-output-format.md.
Related Commands:
- /arc:corpus create - Create corpus for dual indexing (recommended)
- /arc:corpus sync - Index content to both systems
- /arc:corpus list - List available corpora
- /arc:collection list - See available collections (semantic only)
Implementation:
- RDR-007: Semantic search via Qdrant
- RDR-012: Full-text search via MeiliSearch
- RDR-006: Claude Code integration
Store agent-generated content (research, analysis, synthesized information) with rich metadata. Content is persisted to disk for re-indexing and full-text retrieval, then indexed to Qdrant for semantic search.
Tip: For full search capabilities (semantic + full-text), first create a corpus with
/arc:corpus create Memory --type markdown, then use this command with --collection Memory.
The stored content will be searchable via both /arc:search semantic and /arc:search text.
Storage Location: ~/.local/share/arcaneum/agent-memory/{collection}/
Options:
- --collection: Target collection (required)
- --model: Embedding model (default: arctic-m for documents)
- --title: Document title (added to frontmatter)
- --category: Document category (e.g., research, security, analysis)
- --tags: Comma-separated tags
- --metadata: Additional metadata as JSON
- --chunk-size: Target chunk size in tokens (overrides model default)
- --chunk-overlap: Overlap between chunks in tokens
- --verbose: Show detailed progress
- --json: Output in JSON format
Examples:
/store analysis.md --collection Memory --title "Security Analysis" --category security
/store - --collection Research --title "Findings" --tags "research,important"
Execution:
arc store $ARGUMENTS
How It Works:
- Accept content from file or stdin (
-) - Extract/add rich metadata (title, category, tags, custom fields)
- Semantic chunking preserving document structure
- Generate embeddings (arctic-m default: 768D for stable document retrieval)
- Upload to Qdrant with metadata
- Persist to disk:
~/.local/share/arcaneum/agent-memory/{collection}/{date}_{agent}_{slug}.md - Generate YAML frontmatter with injection metadata (injection_id, injected_at, injected_by)
Persistence:
Content is always persisted for durability. This enables:
- Re-indexing: Update embeddings without losing original content
- Full-text retrieval: Access complete original documents
- Audit trail: Track what was stored and when (injection_id, timestamps)
Filename Format:
YYYYMMDD_agent_slug.md (e.g., 20251030_claude_security-analysis.md)
Use Cases:
- AI agents storing research findings
- Preserving analysis results
- Collecting synthesized information
- Building knowledge bases from agent workflows
Default Model:
- arctic-m (768D, stable FastEmbed document retrieval)
Related Commands:
- /arc:corpus create - Create corpus for dual search (recommended:
--type markdown) - /arc:collection create - Create collection for semantic search only
- /arc:search semantic - Search stored content semantically
- /arc:search text - Search stored content with full-text (requires corpus)
- /arc:index markdown - For indexing existing markdown directories (different use case)
Implementation:
- RDR-014: Markdown content indexing
- arcaneum-204: Direct injection persistence module
Sponsors
Details
- Marketplace
- cwensel/arcaneum
- Author
- Arcaneum Contributors
- Category
- Search
- License
- MIT
- Keywords
- semantic-search, full-text-search, hybrid-search, qdrant, meilisearch, agent-memory
