Documentation

kurn is an in-memory search engine for fast fuzzy and exact retrieval over application-owned collections. It combines declared analyzers, typed payload filters, atomic refresh, and deterministic data-and-configuration versions in a Go library or sidecar. A single kurnd is the simplest starting point. kurn search provides example adapter implementations with TypeScript and Python clients as reference designs for building on the engine.

Need live updates? Run kurnd -data to add, update and delete entries while serving queries. Prepared .kurn artifacts use a separate read-only mode.

Install

Use Go 1.27 for a shared engine and Search development setup. The engine alone requires Go 1.26 or newer; kurn search requires Go 1.27 and pins the go1.27.0 toolchain. The commands below install only the engine binaries; see the Search guide for its separate setup.

go install github.com/kurn-dev/kurn/cmd/kurnd@latest   # the server
go install github.com/kurn-dev/kurn/cmd/kurn@latest    # the CLI: build, ingest, bench, calibrate

Or from a clone — the repo also carries a distroless Dockerfile if you would rather build an image:

git clone https://github.com/kurn-dev/kurn && cd kurn
go build -o kurnd ./cmd/kurnd
docker build -t kurnd .        # optional

Two static binaries, no runtime dependencies. The engine is also a Go library — see embedding.

Build → verify → serve a prepared artifact

Run these commands from the current kurn source checkout. The supplied examples/prepared-artifact files provide a small collection and its mapping.

Build the index offline, verify it, then serve the immutable .kurn file without the original feed or mapping. Run the following from the compatible engine checkout with its Go toolchain. Use a fresh output directory and an unused loopback port; each artifact target must be absent.

export ENGINE_PREPARED_ROOT="$(mktemp -d)"
go build -o "$ENGINE_PREPARED_ROOT/kurn" ./cmd/kurn
go build -o "$ENGINE_PREPARED_ROOT/kurnd" ./cmd/kurnd
"$ENGINE_PREPARED_ROOT/kurn" build \
  -mapping ./examples/prepared-artifact/ngram-mapping.json \
  -in ./examples/prepared-artifact/feed.ndjson \
  -artifact "$ENGINE_PREPARED_ROOT/people.kurn" -name people
"$ENGINE_PREPARED_ROOT/kurn" verify -semantic "$ENGINE_PREPARED_ROOT/people.kurn"
"$ENGINE_PREPARED_ROOT/kurnd" -artifact "$ENGINE_PREPARED_ROOT/people.kurn" \
  -addr 127.0.0.1:8080

Require successful build and verification before starting the foreground server. In another terminal:

curl --fail --include http://127.0.0.1:8080/readyz
curl --fail -X POST http://127.0.0.1:8080/v1/query \
  -H 'Content-Type: application/json' \
  -d '{"q":"lovelce ada","lists":["people"],"filter":{"country":"GB"}}'

Readiness must be HTTP 200 with Kurn-Artifact-Verification: full. The example query retrieves p1, Ada Lovelace. Stop the foreground server with Ctrl-C and wait for exit; restart with the same artifact command. Prepare replacements at new paths and coordinate process replacement yourself. This is not hot reload or a zero-downtime guarantee.

• Read-only runtime: mutation routes refuse. The mutable-store commands below are a separate mode, not instructions for modifying a prepared artifact.
• Verification: full verification is the default. Explicit -artifact-verify trusted skips content-digest recomputation: its identity is declared, not proof of the served bytes. Use it only after prior verification and trusted build, delivery and storage; structurally valid content corruption can escape its checks. Neither policy authenticates the publisher. There is no fallback after failure: rejected or incompatible artifacts refuse startup, without a rebuild from source.
• Resources: collections remain resident on one node. Preparation does not establish a hard process-memory cap, production sizing or distributed sharding. Keep authoritative source separately; the artifact is not its replacement.
• Access: pass API keys at runtime with -api-keys-file; keys are not artifact contents. Probes and metrics remain unauthenticated, so restrict listener access.

An n-gram list can include an exact companion for whole-key lookups alongside fuzzy retrieval. For source projection, clients and version-bound credentials, continue with prepared Search suggestions.

The exact_any_of query accepts 1–64 terms for normalized whole-key lookup on an eligible exact list or configured exact companion. Matching entries are unique by ID, score 100, sorted by entry ID and limited by top-K; ordinary typed filters still apply. It is not a fuzzy OR query, and threshold is refused. See the source checkout's API documentation for this query; the website OpenAPI schema describes the ordinary query API.

Mutable-store quickstart — five minutes

First install the engine; this example needs only kurnd and curl, not a Search checkout. Use an unused loopback port and a fresh data directory. This path starts a writable data directory. Its bundle, journal and reload operations do not apply to the read-only prepared mode above.

1. Start the server

kurnd -data ./data     # creates the directory if missing; listens on :8080

2. Create a list and add entries

A list is declared once with an analyzer (how strings are normalized) and a match mode. The person-name preset handles case, accents, punctuation, titles and word order.

curl -s -X PUT localhost:8080/v1/lists/people \
  -d '{"analyzer":{"preset":"person-name"},
       "match":{"mode":"ngram","strip_spaces":true}}'

curl -s -X POST localhost:8080/v1/lists/people/entries \
  -d '[{"id":"p1","keys":["Elena Vasquez","Vasquez, Elena M."]},
       {"id":"p2","keys":["Marcus Chen"]}]'
{"dropped_keys":0,"keyless_entries":0,"upserted":2}

3. Query with a misspelled, reordered name

curl -s localhost:8080/v1/query -d '{"q":"vasquez elna","lists":["people"]}'

{
  "candidates": [
    { "list": "people", "entry_id": "p1",
      "score": 100, "key": "Elena Vasquez" }
  ],
  "versions": { "people": "empty@0+j140.3c8269322e4ba679545907064411634085ae0a4503dd963a442f5f005595ee39+ca44e884c53f256cce53698926d338dfa8a7f1770ced219968985a8fc030f6015" },
  "took_us": 44
}

Misspelled and reversed — found, scored, and attributed: the answer names which of the entry's keys matched. It also carries versions, the content-addressed identity of the data that answered. Keep the request and the response and that result can be reproduced exactly, later, by anyone.

4. Thirty seconds more: a different shape of list

Names are the fuzzy flagship, but exact mode answers identifier lists in microseconds — here a domain blocklist where listing a parent covers every subdomain:

curl -s -X PUT localhost:8080/v1/lists/blocked-domains \
  -d '{"analyzer":{"preset":"domain"},
       "match":{"mode":"exact","fallback":"parent_domain"}}'

printf '{"id":"d1","keys":["tempmail.com"]}\n' | \
  curl -s -X POST 'localhost:8080/v1/lists/blocked-domains/entries?replace=true' \
    -H 'Content-Type: application/x-ndjson' --data-binary @-

curl -s localhost:8080/v1/query -d '{"q":"smtp.tempmail.com","lists":["blocked-domains"]}'
{"candidates":[{"list":"blocked-domains","entry_id":"d1","score":90,
  "key":"tempmail.com"}], …}    # 100 = exact hit, 90 = parent match

Two lists, two matching shapes, one process — and a single query can span both.

Build a list from your database — mutable-store path

This recipe builds a mutable-store bundle with -out. Its updates, compaction and reload steps are not for read-only .kurn artifacts.

The most common reason to reach for kurn: your data already lives in Postgres, MySQL or a warehouse, something in your request path needs a typo-tolerant lookup over it, and the options are all bad — a LIKE '%…%' scan on the primary, a trigram index competing with production traffic, or a search cluster to stand up and keep alive.

The pattern instead: your database stays the system of record, and kurn is a derived, disposable read path. A query produces a list, a build turns it into an artifact, and the artifact ships to every instance that needs it.

1. Export the rows you want to match

Any query that produces one JSON object per line works. Postgres:

psql -At -c "
  SELECT json_build_object('sku', sku, 'title', title, 'brand', brand)
  FROM products WHERE NOT discontinued" > products.ndjson

MySQL is the same shape with JSON_OBJECT(…) and mysql -N -B; a warehouse export or a CSV works too — the mapping in the next step handles the format.

2. Declare the mapping once

Which field is the identity, which fields are searchable, which ride along as payload — plus how the list should match. Committed to git, versioned with your code.

{
  "format": "ndjson",
  "id": "sku",
  "keys": [{"path": "title"}, {"path": "brand"}],
  "payload": {"brand": "brand"},
  "list": {
    "analyzer": {"preset": "free-text"},
    "match": {"mode": "ngram", "strip_spaces": true, "threshold": 0.5}
  }
}

3. Build, serve, query

kurn build -mapping products.mapping.json -in products.ndjson \
  -out bundles/products -source "products $(date -u +%FT%TZ)"
bundle bundles/products: 3 entries, 6 keys, base fa502064497c (mode ngram)

cp -r bundles/products ./data/products && kurnd -data ./data

curl -s localhost:8080/v1/query -d '{"q":"makita impakt drivr","lists":["products"]}'
{"candidates":[{"list":"products","entry_id":"A-2","score":100,
  "key":"Makita impact driver","payload":{"brand":"Makita"}}],
 "versions":{"products":"fa502064497ce8ce1ed03db536d8c8746511432e5f3b5783ba5e196db9efd67c@3+j0+ce9dd5b03c98d113fee839e4457a98902f91df39765f16f5f4483fc8a7f147490"}, "took_us":38}

4. Put it on a timer

Data changes, so rebuild on whatever cadence your staleness budget allows — a cron entry, a systemd timer, a CI job. Unchanged rows under an unchanged resolved mapping produce the same complete served version, so frequent publishes do not change query identity.

psql -At -c "SELECT json_build_object('sku',sku,'title',title,'brand',brand)
             FROM products WHERE NOT discontinued" > /tmp/products.ndjson

kurn build -mapping products.mapping.json -in /tmp/products.ndjson \
  -out /tmp/products.bundle -source "products $(date -u +%FT%TZ)"

BASE=$(sed -n 's/.*"sha256": "\([0-9a-f]*\)".*/\1/p' /tmp/products.bundle/manifest.json)
CFG=$(shasum -a 256 /tmp/products.bundle/config.json | cut -d' ' -f1)
BUNDLE_ID="$BASE-$CFG"
aws s3 sync /tmp/products.bundle "s3://artifacts/products/$BUNDLE_ID/"   # storage key only; not the served +c digest

Each serving instance then pulls the current version and reloads it — POST /v1/lists/products/reload is atomic and golden-gated, and a failed load keeps the previous version serving. The reported version hashes the complete base, journal, and resolved configuration, so “has anything changed?” is a collision-resistant string comparison.

What you trade

Freshness for load. Lookups answer from the last built version, so the staleness bound is your timer interval — five minutes is nothing for a product catalogue and too much for a fraud denylist. When some changes cannot wait, the two paths compose: keep the periodic rebuild for the bulk, and push urgent rows straight to the serving instances as ordinary mutations (POST …/entries, DELETE …/entries/{id}), which are journaled and versioned like everything else.

Rebuilds reflect the source

A row removed from the source is absent from the next build. The version stamp on each answer identifies the collection that replied. Older artifacts still hold their original records, so choose how many versions to retain for reproduction and rollback.

Build from a published feed

A publisher-owned feed is one useful ingestion recipe, not kurn's product category. kurn ships mappings — small, versioned JSON recipes — for several public reference feeds in docs/examples: OFAC SDN, OIG LEIE, the UN Consolidated List, the US Consolidated Screening List and EU financial sanctions. You fetch the publisher's own file; kurn build turns it into a servable, content-addressed bundle.

# 1. the official file, from the official source
curl -fsSL -o sdn.xml \
  https://sanctionslistservice.ofac.treas.gov/api/PublicationPreview/exports/SDN.XML

# 2. build a bundle with the shipped mapping
kurn build -mapping docs/examples/ofac.mapping.json -in sdn.xml \
  -out bundles/sdn -source "sdn 2026-08-04"

# 3. full base evidence plus its short display prefix
grep -E '"(sha256|version_id)"' bundles/sdn/manifest.json
"sha256": "45ee1c2ec11be8ce1ed03db536d8c8746511432e5f3b5783ba5e196db9efd67c"
"version_id": "45ee1c2ec11b"    # display prefix only; never the semantic key

# 4. install and serve
cp -r bundles/sdn ./data/sdn && kurnd -data ./data

Re-run the build on a timer and nothing changes until the publisher's file or resolved mapping changes: equal base bytes and configuration produce the same complete served version. The 12-hex version_id is only a readable prefix of the full base hash. Because bundles are plain directories, keeping every version you ever served is a copy — which is what makes an old answer reproducible years later.

Supported retrieval shapes

The engine retrieves short strings. The table separates named measurements from mechanics the implementation supports; it does not infer suitability for an unevaluated domain.

Entries, keys, payloads

One entry is one thing you might match: an id, its searchable keys, and an opaque payload that rides along.

{"id": "p1",
 "keys": ["Elena Vasquez", "Vasquez, Elena M.", "E. Vasquez"],
 "payload": {"region": "west", "active": true}}

Mappings, analyzers, matching

Three layers do three different jobs. Keeping them straight explains almost everything about how kurn behaves.

Analyzer steps

Presets are exactly these step lists — person-name, free-text, identifier, domain. An analyzer change alters what the index means, so kurn records a digest of the pipeline in every index artifact: change the steps and the index rebuilds itself once, rather than silently answering with stale normalization.

Scores and thresholds

N-gram results are ordered by IDF-weighted string similarity on a 0–100 scale. Rare fragments count for more than common ones, which is why a distinctive surname carries a match and a common filler word does not. Exact-mode results represent normalized whole-key or configured parent-domain matches.

Word order is free when the analyzer sorts tokens — the person-name preset does, so smith jane and jane smith normalize to the same string and score 100. Token fusion (janesmith) is handled separately, by strip_spaces at index time. (The grams themselves do encode order within their own span; it is the analyzer that makes word order irrelevant.)

Scores are comparable within a list, not across lists of different sizes: the weighting is computed over the corpus that answered.

Calibrate on your corpus

kurn calibrate sweeps 2–32 caller-chosen fuzzy thresholds over one immutable n-gram list snapshot and a human-labelled file corpus. For each threshold it reports truth-ID recall and misses, returned-candidate count, and truth rank. The command provides calibration evidence, not automatic threshold selection; threshold selection belongs to the evaluator's loss function. Exact matching has no fuzzy-threshold sweep.

Returned-candidate burden is bounded by the resolved top-K: it counts the candidates actually returned within that cut, never every score-qualifying entry beyond it. Deterministic identity and result fields stay separate from hardware-dependent timing and approximate HeapAlloc observations, and corpus results describe only the measured labels.

Use the calibration walkthrough and kurn-calibration/v1 report schema in the current source for the command, input contract and report fields.

This command measures direct engine queries. A Search adapter can add acceptance rules and fallback lookups, so evaluate that end-to-end path separately. The prepared identifier example checks edit distance and uses exact lookups to recover adjacent transpositions; these are adapter rules, not changes to the engine's similarity score.

Every tagged release runs a pinned synthetic kurn bench command in CI. Rerunning that command regenerates its deterministic identity and result fields; CI retains the complete kurn-eval/v1 report for 90 days as a convenience, not an archive.

Versions and reproducibility

For prepared artifacts, retain the serving binary, artifact and verification mode with the response. The byte-identity claims below require recomputed identity: trusted mode reports declared digests and does not independently establish that the served bytes match them.

Every answer carries versions — the content-addressed identity of each list it consulted:

"catalog": "4f107b9652014c…@83639+j0+c8062333a3e9ae…"
         └ sha256 of the built list: WHICH data
                        └ entry count
                              └ journal applied on top: byte depth, plus
                                a content hash once mutations land
                                ("+j174.d2e39af703dbcc…")
                                     └ sha256 of the RESOLVED config:
                                       WHICH lens read the data

The version and the candidates come from one atomic snapshot — the stamp is the identity of exactly the data AND configuration that answered, never of a mutation that landed mid-query. All three hashes are complete sha256, not prefixes: the stamp is evidence, and truncated digests invite deliberate collisions. The candidates are a pure function of the query, those versions, and the kurn version: same question, same data, same matches — byte for byte, on any machine, on any CPU architecture. There is no query planner making runtime decisions, so nothing drifts as statistics change. (The response envelope also carries took_us, which is wall time: deliberately not part of the reproducible answer.)

Keeping an audit trail

Store three things beside each other: the request (q, lists, threshold, topk, and the filter with its echo when you sent one — an empty filtered result describes the scoped slice, not the whole list), the response's candidates and versions (everything but the timing field), and your own timestamp. That tuple is complete evidence — for empty results as much as for hits, since “we searched this string against version 4f107b… and found nothing” is exactly as reproducible as a match. Keep the bundle directory for each version you publish and any past answer can be re-derived from first principles; an archived base.jsonl re-hashes to the stamp's base half, and a living list's journal re-hashes to its journal half, so the evidence verifies itself without trusting anyone.

Update a live collection

With kurnd -data ./data, collections are writable. Add or update entries by ID, delete entries, or replace a collection's contents while serving queries. Each successful write changes the collection's version. On disk, a base snapshot and append-only journal retain the changes for restart.

These commands continue from the mutable-store quickstart, with its server running and the people list already created. The first updates p2; the second removes it. Prepared kurnd -artifact mode does not accept these writes.

curl --fail --silent --show-error -X POST http://127.0.0.1:8080/v1/lists/people/entries \
  -H 'Content-Type: application/json' \
  -d '[{"id":"p2","keys":["Marcus T. Chen","Marcus Chen"]}]'

curl --fail --silent --show-error -X DELETE http://127.0.0.1:8080/v1/lists/people/entries/p2

Time enters as mutations, never as engine state: kurn has no TTL. An entry that should expire is deleted when it expires — which makes the expiry itself a journaled, versioned, auditable event rather than a silent disappearance. A ten-line reaper beside your service is the whole pattern.

Sidecar over HTTP

The pattern for any language: run kurnd next to your service — same host, same pod — and POST to it over loopback with a reused connection. There is no SDK to install because the client is your HTTP client.

import requests
kurn = requests.Session()                    # connection reuse matters

def search(query: str) -> dict:
    r = kurn.post("http://127.0.0.1:8080/v1/query",
                  json={"q": query, "lists": ["catalog", "customers"]})
    r.raise_for_status()
    return r.json()          # candidates plus exact collection versions

Reuse the loopback connection and measure latency on your own collection and query mix. Gate deployment on /readyz — it answers 503 until every list is loaded and each list's golden probes pass.

Embed in Go

The engine is a plain library; in-process there is no HTTP at all and a check is a function call.

import "github.com/kurn-dev/kurn/engine"

l, _ := engine.NewList("people", engine.ListConfig{
    Analyzer: engine.AnalyzerConfig{Preset: "person-name"},
    Match:    engine.MatchConfig{Mode: "ngram", StripSpaces: true},
})
l.Replace([]engine.Entry{
    {ID: "p1", Keys: []string{"Elena Vasquez", "Vasquez, Elena M."}},
})
for _, c := range l.Query("vasquez elna", engine.QueryOpts{}) {
    fmt.Printf("%s score=%.0f key=%q\n", c.EntryID, c.Score, c.Key)
}
// p1 score=100 key="Elena Vasquez"

Query is lock-free and safe under concurrent mutation; mutations serialize internally. For a durable multi-list store on disk use engine.Open(dir). Note the licence: kurn is AGPL-3.0, so embedding it in a closed-source product needs a commercial licence — ops@kurn.it.

Batch queries

POST /v1/batch-query takes 1–100 independent lookups in one round trip. Each carries its own lists and overrides, results come back in order, and a failed lookup yields {"error": "…"} in its place — so one bad row never fails the batch.

-d '{"checks":[
      {"q":"makita impakt drivr","lists":["products"],"topk":1},
      {"q":"smtp.tempmail.com","lists":["blocked-domains"]},
      {"q":"vasquez elna","lists":["customers"],"threshold":0.7}
    ]}'

Running a whole file through — deduplicating an import, enriching a catalogue, re-checking a customer table — is this in a loop: stream it in batches of 100 and store each response with its version stamp, and the pass is re-runnable later with its evidence attached.

Crash-only operation

For the mutable data-directory store, mutations are journaled before they are acknowledged and list swaps are atomic renames. A journal that cannot be replayed is quarantined rather than half-applied, a damaged index artifact is discarded and rebuilt from the base data, and a failed publish leaves the previous list serving. kill -9 is the tested path, not the feared one — recovery is simply the next start.

These rebuild and restart measurements concern mutable-store index caches, not prepared artifacts. A prepared artifact that fails verification refuses startup without rebuilding or falling back to trusted mode.

That is the recovery-time objective: recovery time is open time. Journal durability is a knob — -journal-fsync=none (default; survives process crash, not power loss), every, or interval for group commit.

Memory and admission

Index cost is about 114 bytes per key end-to-end for fuzzy lists (roughly 69 B of postings plus the id map) and 133 B/key for exact lists. Process memory is a larger, separate number, because the entries and payloads are resident too: five public name-list corpora — 148,837 entries, 191,166 keys — index in about 22 MB and settle at about 128 MB resident. Ten million synthetic keys take about 2.2 GB.

Queries also borrow scratch memory: roughly 8 bytes per ordinal per list touched (scan accumulators, returned to a pool afterwards) plus a bounded per-hit working set. Admission control charges each query a conservative ceiling for that shape against the scratch budget. This is not a total process-memory limit or an out-of-memory guarantee. -query-mem-budget-mb (default 1024) bounds in-flight scratch and -query-queue-timeout (default 2 s) bounds waiting; a query that cannot be admitted in time gets a 503 naming the budget. Resident data, HTTP work and other allocations still need separate capacity planning.

Metrics and probes

In prepared mode, readiness means the chosen startup policy completed and queries can be served. Check Kurn-Artifact-Verification on /readyz and /v1/*; /artifactz separates verified from declared identities, and query version_bases distinguishes verified-base-id from declared-base-id. These unauthenticated management endpoints share the serving listener. The golden-probe description below concerns mutable stores.

Golden probes make “did this deployment load the right data?” part of the readiness check.

Auth and tenants

Auth is off by default — a loopback or private-network kurnd needs none, and probes and metrics always stay open so orchestration keeps working.

HTTP API

The website OpenAPI schema describes the ordinary mutable-store API. For prepared-artifact extensions, use the source checkout's API documentation. In prepared mode the mutation operations below refuse. Errors are always a JSON object with one error field and an honest status code.

400  {"error": "q must be non-empty"}
404  {"error": "unknown list \"nope\""}
401  {"error": "missing or invalid API key"}      # only when auth is on
503  {"error": "query admission timed out: scratch budget exhausted (3 queued, 41943040 bytes in flight)"}

Command line

Prepared-artifact mode uses kurn build -artifact FILE, kurn verify -semantic FILE and kurnd -artifact FILE. Do not confuse the single-file artifact with the mutable bundle produced by kurn build -out DIR.

Retrieval and operating model

kurn keeps application-owned short-string collections resident in memory on one node by default. Run independent replicas manually when needed. N-gram results are ordered by IDF-weighted string similarity, while exact-mode results represent normalized whole-key or configured parent-domain matches. Typed filters apply exact predicates to declared payload paths before top-K without changing scores on kurnd.

Evaluate retrieval quality on the collection and threshold you will use. The published figures are limited to the named 148,837-entry corpus, one idle 4 vCPU server, 5 Aug 2026, and threshold 0.60; the per-list breakdowns and false-positive caveats remain in bench/README.md.

Design philosophy

The client is curl. JSON in, JSON out, over plain HTTP — one endpoint answers every lookup, and the handful of others manage lists. No SDK, no session, no handshake: anything that can POST can query, and everything the engine does is visible in a terminal.

Dependencies are debt. The engine is the Go standard library, the Go team's golang.org/x packages, and exactly one chosen third-party library — Roaring bitmaps, for the posting lists. When a feature seems to need a framework, the feature gets redesigned: the metrics are hand-written Prometheus text and the feed parsers are stdlib streaming parsers.

One binary to start. Direct kurnd use needs one binary and your list. The Search examples demonstrate a separate adapter service and TypeScript and Python clients for applications built on the engine.

Crash-only mutable store. Journaled mutations, atomic renames, an unreplayable journal quarantined instead of half-applied, and a failed publish that leaves the previous list serving. Kill −9 is the tested path. Prepared-artifact verification instead fails closed at startup.

Measured, not asserted. Performance and matching claims ship with the harness that produced them, and the awkward numbers get published beside the flattering ones.

Back to kurn.it · Search guide · GitHub · openapi.yaml · llms.txt · ops@kurn.it