Search and export (WS4)

Search and export are government-side productivity features. Search narrows the ledger without requiring a specialized search service. Export creates portable files for enterprise administrators who need to analyze or hand off data outside the application.

Search walkthrough

GET /api/search trims the query, caps it at 255 characters, accepts limit 1–50, and validates the comma-separated types list. An empty query is invalid at the API boundary.

Search semantics

The default backend is a hybrid of two stages. PostgreSQL full-text search (websearch_to_tsquery/to_tsquery with English stemming) runs first over organization name/description/tags, submission title/summary/tags, engagement title/summary/location, and call title/description/tags. Queries support boolean operators (words are AND-ed, OR disjoins, -word excludes), "quoted phrase" matching, and prefix* wildcards, with operator-free queries getting an implicit prefix match on the last word so type-ahead keeps matching partially typed words. Results are ordered by ts_rank relevance with updated/created/date fields as a recency tiebreaker, limited per group, and backed by expression GIN indexes. English stop words are not indexed, so stop-word-only queries fall through to the fuzzy stage.

When full-text search finds nothing — typically a misspelled query — the search degrades to typo-tolerant pg_trgm trigram fuzzy matching, unless the query uses operator syntax (quotes, OR, -/! negation, or * wildcards): the fuzzy stage is operator-blind and would otherwise match the very terms a query excluded, so operator queries with no full-text matches return empty results. The fuzzy stage runs word_similarity() / %> over GIN trigram indexes against the same fields. Misspelled queries like autonmy still match "autonomy" records, and a term buried inside a long summary still matches because word similarity scores the best-matching extent within a field. Fuzzy results are ranked by descending similarity (ties broken by updated/created/date fields) and limited per group. The match threshold defaults to 0.2 and is tunable via the SEARCH_SIMILARITY_THRESHOLD env var (strictly between 0 and 1; a value outside that range logs a warning and the default is kept). Raising it trades recall for precision on both axes: it tightens typo tolerance and also drops short-fragment hits with modest trigram overlap (e.g. ai scores about 0.33 against "artificial intelligence", so it disappears at 0.35).

Each stage is also selectable on its own via SEARCH_BACKEND: hybrid (default), fts (full-text only), trigram (fuzzy only), or prisma (the original case-insensitive substring fallback). If the pg_trgm extension is missing from the database, the trigram stage logs an error and degrades to that substring backend until the process restarts.

Snippets normalize whitespace and center around the first literal query occurrence when present (fuzzy-only matches fall back to the leading text), with a maximum length of 160 characters. Neither stage is vector similarity. In the fuzzy stage, mid-word fragment matching is not guaranteed: a fragment matches only when its trigram overlap clears the threshold (e.g. onom scores 0.4 against "autonomy" and matches at the default 0.2, while other fragments may fall below it).

Export walkthrough

Export routes require GOV_ADMIN or GOV_SYSTEM_ADMIN; a command lead cannot export the enterprise dataset. Formats are:

Data handling

Exports may include organization names, submission summaries, engagement details, and internal operational context. The role gate is necessary but not a substitute for handling policy. Store downloaded files in approved locations, avoid forwarding them to industry contacts, and delete temporary copies when no longer needed.

Backend seam

src/lib/search.ts defines SearchBackend, searchAll, and setSearchBackend. The default is hybridBackend (full-text first, trigram fuzzy fallback); ftsBackend, trigramBackend, and the original Prisma contains backend (prismaBackend) remain individually selectable, and a future vector backend can implement the same result groups without changing the route contract.

Search examples

Search is not a substitute for scope. A result can be found by an authorized government user and still require a destination route to enforce narrower command or record policy.

Export shape and responsibility

JSON is convenient for integrations, CSV is convenient for spreadsheets, and iCalendar is convenient for calendar applications. None should be treated as a redaction layer. The export route selects data for enterprise admins; recipients are responsible for further handling, retention, and deletion.

Search walkthrough

1. Start with a specific term such as Pelagic or NAVSEA, not an empty query.
2. The browser calls /api/search?q=Pelagic with the signed-in session.
3. The server validates the query length and optional entity type, then searches the supported record families.
4. Review the result type, title, snippet, and URL; the response is a compact index, not a full record dump.
5. Open the destination URL and let that page perform its own audience and command/organization authorization.

Search matching is intentionally useful for discovery, but it is not an authorization shortcut. A result appearing in a scoped search must never be interpreted as permission to call an unrelated mutation route.

Export walkthrough

1. Sign in as admin@navy.mil or sysadmin@navy.mil; a command lead is intentionally not sufficient.
2. Choose the smallest format for the task: JSON bundle for machine processing, CSV for tabular review, or iCalendar for engagement dates.
3. Request the route with the browser session and inspect the response content-type and content-disposition.
4. Save the file in a controlled location and verify that the downloaded rows match the requested scope.
5. Remove the temporary copy when the approved review ends.

The routes do not accept a client-supplied role to broaden access. Export authority comes from the current database-backed session.