Embedding Orrery

Visual exploration of vector spaces and SAE features.

Embedding Orrery is an open-source platform for interactive 3D visualisation of embedding spaces, with native Sparse Autoencoder (SAE) support. Orrery turns any textual, image, or vector dataset into a 3D constellation searchable visually, semantically, lexically, or by SAE feature name, with filtering and in-interface probe training. SAE feature spaces themselves can be visualised as collections, and individual features inspected in a dashboard or injected into the model for causal steering.

Try the live demo on Hugging Face Spaces  — read-only, with a guided tour and preset views. No install needed.

What it does

Embedding visualisation — Embed from the HuggingFace Hub, local files (CSV/JSON/Parquet), images, or pre-computed vectors, then explore in WebGL 2D/3D scatter plots. Eight embedding providers; one dataset can carry multiple embeddings without duplication. Tested up to 500k points on 8 GB of RAM.

Topic extraction — A BERTopic-style pipeline: HDBSCAN clustering with c-TF-IDF keywords and optional LLM labels, plus hierarchical reduction with nested colouring.

SAE feature analysis — Live inference on Gemma 3 (Gemma Scope 2) and Qwen (Qwen-Scope) with from-scratch JumpReLU/TopK SAE implementations. Capture per-token activations, highlight activated features on the scatter plot, apply additive steering, and chat with the steered model.

Search and probing — Semantic, text, and SAE-feature search: type “poetry” and Orrery ranks documents by how strongly the matching SAE features fire on them. Probes (mass-mean, ridge, SVR, MLP) train on any metadata field directly in the interface, and linear probes double as directions to colour the plot by.

EMNLP abstracts 2014–2025: category-coloured density contours with the analytics panel	Lacan corpus (153k sentences): cross-lingual semantic search

Find a feature, inspect it, steer the model: with the “poetry” feature injected, Gemma-3-4b-it answers “What is your favourite job?” in verse, side by side with the baseline.

Where to start

• Getting Started — clone-to-running in a few minutes
• Docker — containerised setup
• Database Architecture — the DuckDB + ChromaDB design
• SAE & Interpretability — feature storage, pipelines, live inference
• Case Studies — probing and steering validated against published results

Source code on GitHub , licensed Apache 2.0.