Documentation

kurn is an in-memory search engine for fast fuzzy and exact retrieval over application-owned collections. It combines declared analyzers, typed payload filters, atomic refresh, and deterministic data-and-configuration versions in a Go library or sidecar. Application-owned short-string collections stay resident in memory on one node; independent replicas provide availability or throughput without a sharded or coordinated search cluster.

Install

go install github.com/kurn-dev/kurn/cmd/kurnd@latest   # the server
go install github.com/kurn-dev/kurn/cmd/kurn@latest    # the CLI: build, ingest, bench, calibrate

Or from a clone — the repo also carries a distroless Dockerfile if you would rather build an image:

git clone https://github.com/kurn-dev/kurn && cd kurn
go build -o kurnd ./cmd/kurnd
docker build -t kurnd .        # optional

Two static binaries, no runtime dependencies. The engine is also a Go library — see embedding.

Quickstart — five minutes

1. Start the server

kurnd -data ./data     # creates the directory if missing; listens on :8080

2. Create a list and add entries

A list is declared once with an analyzer (how strings are normalized) and a match mode. The person-name preset handles case, accents, punctuation, titles and word order.

curl -s -X PUT localhost:8080/v1/lists/people \
  -d '{"analyzer":{"preset":"person-name"},
       "match":{"mode":"ngram","strip_spaces":true}}'

curl -s -X POST localhost:8080/v1/lists/people/entries \
  -d '[{"id":"p1","keys":["Elena Vasquez","Vasquez, Elena M."]},
       {"id":"p2","keys":["Marcus Chen"]}]'
{"dropped_keys":0,"keyless_entries":0,"upserted":2}

3. Query with a misspelled, reordered name

curl -s localhost:8080/v1/query -d '{"q":"vasquez elna","lists":["people"]}'

{
  "candidates": [
    { "list": "people", "entry_id": "p1",
      "score": 100, "key": "Elena Vasquez" }
  ],
  "versions": { "people": "empty@0+j140.3c8269322e4ba679545907064411634085ae0a4503dd963a442f5f005595ee39+ca44e884c53f256cce53698926d338dfa8a7f1770ced219968985a8fc030f6015" },
  "took_us": 44
}

Misspelled and reversed — found, scored, and attributed: the answer names which of the entry's keys matched. It also carries versions, the content-addressed identity of the data that answered. Keep the request and the response and that result can be reproduced exactly, later, by anyone.

4. Thirty seconds more: a different shape of list

Names are the fuzzy flagship, but exact mode answers identifier lists in microseconds — here a domain blocklist where listing a parent covers every subdomain:

curl -s -X PUT localhost:8080/v1/lists/blocked-domains \
  -d '{"analyzer":{"preset":"domain"},
       "match":{"mode":"exact","fallback":"parent_domain"}}'

printf '{"id":"d1","keys":["tempmail.com"]}\n' | \
  curl -s -X POST 'localhost:8080/v1/lists/blocked-domains/entries?replace=true' \
    -H 'Content-Type: application/x-ndjson' --data-binary @-

curl -s localhost:8080/v1/query -d '{"q":"smtp.tempmail.com","lists":["blocked-domains"]}'
{"candidates":[{"list":"blocked-domains","entry_id":"d1","score":90,
  "key":"tempmail.com"}], …}    # 100 = exact hit, 90 = parent match

Two lists, two matching shapes, one process — and a single query can span both.

Build a list from your database

The most common reason to reach for kurn: your data already lives in Postgres, MySQL or a warehouse, something in your request path needs a typo-tolerant lookup over it, and the options are all bad — a LIKE '%…%' scan on the primary, a trigram index competing with production traffic, or a search cluster to stand up and keep alive.

The pattern instead: your database stays the system of record, and kurn is a derived, disposable read path. A query produces a list, a build turns it into an artifact, and the artifact ships to every instance that needs it.

1. Export the rows you want to match

Any query that produces one JSON object per line works. Postgres:

psql -At -c "
  SELECT json_build_object('sku', sku, 'title', title, 'brand', brand)
  FROM products WHERE NOT discontinued" > products.ndjson

MySQL is the same shape with JSON_OBJECT(…) and mysql -N -B; a warehouse export or a CSV works too — the mapping in the next step handles the format.

2. Declare the mapping once

Which field is the identity, which fields are searchable, which ride along as payload — plus how the list should match. Committed to git, versioned with your code.

{
  "format": "ndjson",
  "id": "sku",
  "keys": [{"path": "title"}, {"path": "brand"}],
  "payload": {"brand": "brand"},
  "list": {
    "analyzer": {"preset": "free-text"},
    "match": {"mode": "ngram", "strip_spaces": true, "threshold": 0.5}
  }
}

3. Build, serve, query

kurn build -mapping products.mapping.json -in products.ndjson \
  -out bundles/products -source "products $(date -u +%FT%TZ)"
bundle bundles/products: 3 entries, 6 keys, base fa502064497c (mode ngram)

cp -r bundles/products ./data/products && kurnd -data ./data

curl -s localhost:8080/v1/query -d '{"q":"makita impakt drivr","lists":["products"]}'
{"candidates":[{"list":"products","entry_id":"A-2","score":100,
  "key":"Makita impact driver","payload":{"brand":"Makita"}}],
 "versions":{"products":"fa502064497ce8ce1ed03db536d8c8746511432e5f3b5783ba5e196db9efd67c@3+j0+ce9dd5b03c98d113fee839e4457a98902f91df39765f16f5f4483fc8a7f147490"}, "took_us":38}

4. Put it on a timer

Data changes, so rebuild on whatever cadence your staleness budget allows — a cron entry, a systemd timer, a CI job. Unchanged rows under an unchanged resolved mapping produce the same complete served version, so frequent publishes do not change query identity.

psql -At -c "SELECT json_build_object('sku',sku,'title',title,'brand',brand)
             FROM products WHERE NOT discontinued" > /tmp/products.ndjson

kurn build -mapping products.mapping.json -in /tmp/products.ndjson \
  -out /tmp/products.bundle -source "products $(date -u +%FT%TZ)"

BASE=$(sed -n 's/.*"sha256": "\([0-9a-f]*\)".*/\1/p' /tmp/products.bundle/manifest.json)
CFG=$(shasum -a 256 /tmp/products.bundle/config.json | cut -d' ' -f1)
BUNDLE_ID="$BASE-$CFG"
aws s3 sync /tmp/products.bundle "s3://artifacts/products/$BUNDLE_ID/"   # storage key only; not the served +c digest

Each serving instance then pulls the current version and reloads it — POST /v1/lists/products/reload is atomic and golden-gated, and a failed load keeps the previous version serving. The reported version hashes the complete base, journal, and resolved configuration, so “has anything changed?” is a collision-resistant string comparison.

What you trade

Freshness for load. Lookups answer from the last built version, so the staleness bound is your timer interval — five minutes is nothing for a product catalogue and too much for a fraud denylist. When some changes cannot wait, the two paths compose: keep the periodic rebuild for the bulk, and push urgent rows straight to the serving instances as ordinary mutations (POST …/entries, DELETE …/entries/{id}), which are journaled and versioned like everything else.

A side effect worth knowing about

Because each build is derived from a query, a row that is gone from the source is gone from the next artifact by construction — there is no separate deletion path to remember, and no delete-by-query to hope succeeded. For erasure requests that property is worth a lot: you can show that version N+1 does not contain a record, and the version stamp on every answer since then says which data replied. The other side of it: any older artifacts you kept still contain that row, so if the list holds personal data, decide your archive retention deliberately rather than keeping everything by default. Public reference lists and product catalogues are the easy cases; your customer table is not.

Build from a published feed

A publisher-owned feed is one useful ingestion recipe, not kurn's product category. kurn ships mappings — small, versioned JSON recipes — for several public reference feeds in docs/examples: OFAC SDN, OIG LEIE, the UN Consolidated List, the US Consolidated Screening List and EU financial sanctions. You fetch the publisher's own file; kurn build turns it into a servable, content-addressed bundle.

# 1. the official file, from the official source
curl -fsSL -o sdn.xml \
  https://sanctionslistservice.ofac.treas.gov/api/PublicationPreview/exports/SDN.XML

# 2. build a bundle with the shipped mapping
kurn build -mapping docs/examples/ofac.mapping.json -in sdn.xml \
  -out bundles/sdn -source "sdn 2026-08-04"

# 3. full base evidence plus its short display prefix
grep -E '"(sha256|version_id)"' bundles/sdn/manifest.json
"sha256": "45ee1c2ec11be8ce1ed03db536d8c8746511432e5f3b5783ba5e196db9efd67c"
"version_id": "45ee1c2ec11b"    # display prefix only; never the semantic key

# 4. install and serve
cp -r bundles/sdn ./data/sdn && kurnd -data ./data

Re-run the build on a timer and nothing changes until the publisher's file or resolved mapping changes: equal base bytes and configuration produce the same complete served version. The 12-hex version_id is only a readable prefix of the full base hash. Because bundles are plain directories, keeping every version you ever served is a copy — which is what makes an old answer reproducible years later.

Supported retrieval shapes

The engine retrieves short strings. The table separates named measurements from mechanics the implementation supports; it does not infer suitability for an unevaluated domain.

Entries, keys, payloads

One entry is one thing you might match: an id, its searchable keys, and an opaque payload that rides along.

{"id": "p1",
 "keys": ["Elena Vasquez", "Vasquez, Elena M.", "E. Vasquez"],
 "payload": {"region": "west", "active": true}}

Mappings, analyzers, matching

Three layers do three different jobs. Keeping them straight explains almost everything about how kurn behaves.

Analyzer steps

Presets are exactly these step lists — person-name, free-text, identifier, domain. An analyzer change alters what the index means, so kurn records a digest of the pipeline in every index artifact: change the steps and the index rebuilds itself once, rather than silently answering with stale normalization.

Scores and thresholds

N-gram results are ordered by IDF-weighted string similarity on a 0–100 scale. Rare fragments count for more than common ones, which is why a distinctive surname carries a match and a common filler word does not. Exact-mode results represent normalized whole-key or configured parent-domain matches.

Word order is free when the analyzer sorts tokens — the person-name preset does, so smith jane and jane smith normalize to the same string and score 100. Token fusion (janesmith) is handled separately, by strip_spaces at index time. (The grams themselves do encode order within their own span; it is the analyzer that makes word order irrelevant.)

Scores are comparable within a list, not across lists of different sizes: the weighting is computed over the corpus that answered.

Calibrate on your corpus

kurn calibrate sweeps 2–32 caller-chosen fuzzy thresholds over one immutable n-gram list snapshot and a human-labelled file corpus. For each threshold it reports truth-ID recall and misses, returned-candidate count, and truth rank. The command provides calibration evidence, not automatic threshold selection; threshold selection belongs to the evaluator's loss function. Exact matching has no fuzzy-threshold sweep.

Returned-candidate burden is bounded by the resolved top-K: it counts the candidates actually returned within that cut, never every score-qualifying entry beyond it. Deterministic identity and result fields stay separate from hardware-dependent timing and approximate HeapAlloc observations, and corpus results describe only the measured labels.

Use the version-pinned v0.6.0 walkthrough and kurn-calibration/v1 schema for the complete command, input contract, report fields, privacy boundary, and atomic write behavior.

Every tagged release runs a pinned synthetic kurn bench command in CI. Rerunning that command regenerates its deterministic identity and result fields; CI retains the complete kurn-eval/v1 report for 90 days as a convenience, not an archive.

Versions and reproducibility

Every answer carries versions — the content-addressed identity of each list it consulted:

"catalog": "4f107b9652014c…@83639+j0+c8062333a3e9ae…"
         └ sha256 of the built list: WHICH data
                        └ entry count
                              └ journal applied on top: byte depth, plus
                                a content hash once mutations land
                                ("+j174.d2e39af703dbcc…")
                                     └ sha256 of the RESOLVED config:
                                       WHICH lens read the data

The version and the candidates come from one atomic snapshot — the stamp is the identity of exactly the data AND configuration that answered, never of a mutation that landed mid-query. All three hashes are complete sha256, not prefixes: the stamp is evidence, and truncated digests invite deliberate collisions. The candidates are a pure function of the query, those versions, and the kurn version: same question, same data, same matches — byte for byte, on any machine, on any CPU architecture. There is no query planner making runtime decisions, so nothing drifts as statistics change. (The response envelope also carries took_us, which is wall time: deliberately not part of the reproducible answer.)

Keeping an audit trail

Store three things beside each other: the request (q, lists, threshold, topk, and the filter with its echo when you sent one — an empty filtered result describes the scoped slice, not the whole list), the response's candidates and versions (everything but the timing field), and your own timestamp. That tuple is complete evidence — for empty results as much as for hits, since “we searched this string against version 4f107b… and found nothing” is exactly as reproducible as a match. Keep the bundle directory for each version you publish and any past answer can be re-derived from first principles; an archived base.jsonl re-hashes to the stamp's base half, and a living list's journal re-hashes to its journal half, so the evidence verifies itself without trusting anyone.

Bundles, journals, living lists

A list on disk is a base snapshot plus an append-only journal.

Time enters as mutations, never as engine state: kurn has no TTL. An entry that should expire is deleted when it expires — which makes the expiry itself a journaled, versioned, auditable event rather than a silent disappearance. A ten-line reaper beside your service is the whole pattern.

Sidecar over HTTP

The pattern for any language: run kurnd next to your service — same host, same pod — and POST to it over loopback with a reused connection. There is no SDK to install because the client is your HTTP client.

import requests
kurn = requests.Session()                    # connection reuse matters

def search(query: str) -> dict:
    r = kurn.post("http://127.0.0.1:8080/v1/query",
                  json={"q": query, "lists": ["catalog", "customers"]})
    r.raise_for_status()
    return r.json()          # candidates plus exact collection versions

Reuse the loopback connection and measure latency on your own collection and query mix. Gate deployment on /readyz — it answers 503 until every list is loaded and each list's golden probes pass.

Embed in Go

The engine is a plain library; in-process there is no HTTP at all and a check is a function call.

import "github.com/kurn-dev/kurn/engine"

l, _ := engine.NewList("people", engine.ListConfig{
    Analyzer: engine.AnalyzerConfig{Preset: "person-name"},
    Match:    engine.MatchConfig{Mode: "ngram", StripSpaces: true},
})
l.Replace([]engine.Entry{
    {ID: "p1", Keys: []string{"Elena Vasquez", "Vasquez, Elena M."}},
})
for _, c := range l.Query("vasquez elna", engine.QueryOpts{}) {
    fmt.Printf("%s score=%.0f key=%q\n", c.EntryID, c.Score, c.Key)
}
// p1 score=100 key="Elena Vasquez"

Query is lock-free and safe under concurrent mutation; mutations serialize internally. For a durable multi-list store on disk use engine.Open(dir). Note the licence: kurn is AGPL-3.0, so embedding it in a closed-source product needs a commercial licence — ops@kurn.it.

Batch queries

POST /v1/batch-query takes 1–100 independent lookups in one round trip. Each carries its own lists and overrides, results come back in order, and a failed lookup yields {"error": "…"} in its place — so one bad row never fails the batch.

-d '{"checks":[
      {"q":"makita impakt drivr","lists":["products"],"topk":1},
      {"q":"smtp.tempmail.com","lists":["blocked-domains"]},
      {"q":"vasquez elna","lists":["customers"],"threshold":0.7}
    ]}'

Running a whole file through — deduplicating an import, enriching a catalogue, re-checking a customer table — is this in a loop: stream it in batches of 100 and store each response with its version stamp, and the pass is re-runnable later with its evidence attached.

Behind a load balancer

One writer per list — the load balancer balances readers. Each kurnd owns its data directory, and there is deliberately no replication between instances. Round-robin mutations across instances and you do not get stale reads, you get divergent lists with no merge story. Route by path instead; the API separates the namespaces precisely so an edge rule can split them.

:8080 {
    @writes path /v1/lists/*
    handle @writes {
        reverse_proxy kurn-writer:8080          # the one writer
    }
    handle {
        reverse_proxy kurn-writer:8080 kurn-replica-1:8080 kurn-replica-2:8080 {
            lb_policy round_robin
            health_uri /readyz                  # golden probes included
        }
    }
}

Replicas do not watch the writer; they receive publications. On whatever cadence your staleness budget allows, the writer compacts and ships, and each replica reloads:

curl -s -X POST kurn-writer:8080/v1/lists/blocked/compact
rsync -a kurn-writer:/data/blocked/ replica-1:/data/blocked/
curl -s -X POST replica-1:8080/v1/lists/blocked/reload   # atomic, fail-keeps-serving

Divergence is detectable by design: every answer stamps its versions, so monitoring is one comparison — alert when replicas disagree on a list's version for longer than the shipping interval. What not to build is real-time agreement between instances; that is consensus, it is out of kurn's scope, and an hour spent simulating it with sticky sessions is an hour spent building a worse etcd.

Crash-only operation

There is no graceful-shutdown choreography to get wrong. Mutations are journaled before they are acknowledged and list swaps are atomic renames. A journal that cannot be replayed is quarantined rather than half-applied, a damaged index artifact is discarded and rebuilt from the base data, and a failed publish leaves the previous list serving. kill -9 is the tested path, not the feared one — recovery is simply the next start.

That is the recovery-time objective: recovery time is open time. Journal durability is a knob — -journal-fsync=none (default; survives process crash, not power loss), every, or interval for group commit.

Memory and admission

Index cost is about 114 bytes per key end-to-end for fuzzy lists (roughly 69 B of postings plus the id map) and 133 B/key for exact lists. Process memory is a larger, separate number, because the entries and payloads are resident too: five public name-list corpora — 148,837 entries, 191,166 keys — index in about 22 MB and settle at about 128 MB resident. Ten million synthetic keys take about 2.2 GB.

Queries also borrow scratch memory: roughly 8 bytes per ordinal per list touched (scan accumulators, returned to a pool afterwards) plus a bounded per-hit working set. Admission control charges each query a conservative ceiling for that shape against the budget, so admitted concurrency stays inside it instead of becoming an out-of-memory event. -query-mem-budget-mb (default 1024) bounds in-flight scratch and -query-queue-timeout (default 2 s) bounds waiting; a query that cannot be admitted in time gets a 503 naming the budget, rather than the machine thrashing.

Metrics and probes

Golden probes are the useful trick: they turn “did this deployment load the right data?” into a health check your load balancer already understands.

Auth and tenants

Auth is off by default — a loopback or private-network kurnd needs none, and probes and metrics always stay open so orchestration keeps working.

HTTP API

The machine-readable contract is openapi.yaml. Errors are always a JSON object with one error field and an honest status code.

400  {"error": "q must be non-empty"}
404  {"error": "unknown list \"nope\""}
401  {"error": "missing or invalid API key"}      # only when auth is on
503  {"error": "query admission timed out: scratch budget exhausted (3 queued, 41943040 bytes in flight)"}

Command line

Retrieval and operating model

kurn keeps application-owned short-string collections resident in memory on one node. Run independent replicas for availability or throughput; kurn does not shard collections or coordinate a search cluster. N-gram results are ordered by IDF-weighted string similarity, while exact-mode results represent normalized whole-key or configured parent-domain matches. Typed filters apply exact predicates to declared payload paths before top-K without changing scores.

Evaluate retrieval quality on the collection and threshold you will use. The published figures are limited to the named 148,837-entry corpus, one idle 4 vCPU server, 5 Aug 2026, and threshold 0.60; the per-list breakdowns and false-positive caveats remain in bench/README.md.

Design philosophy

The client is curl. JSON in, JSON out, over plain HTTP — one endpoint answers every lookup, and the handful of others manage lists. No SDK, no session, no handshake: anything that can POST can query, and everything the engine does is visible in a terminal.

Dependencies are debt. The engine is the Go standard library, the Go team's golang.org/x packages, and exactly one chosen third-party library — Roaring bitmaps, for the posting lists. When a feature seems to need a framework, the feature gets redesigned: the metrics are hand-written Prometheus text and the feed parsers are stdlib streaming parsers.

One binary. The engine compiles to a single static binary; running it is that binary and your list. No cluster to form, no sidecar to feed.

Crash-only. Journaled mutations, atomic renames, an unreplayable journal quarantined instead of half-applied, and a failed publish that leaves the previous list serving. Kill −9 is the tested path.

Measured, not asserted. Performance and matching claims ship with the harness that produced them, and the awkward numbers get published beside the flattering ones.

Back to kurn.it · GitHub · openapi.yaml · llms.txt · ops@kurn.it