> ## Documentation Index
> Fetch the complete documentation index at: https://docs.questarr.app/llms.txt
> Use this file to discover all available pages before exploring further.

# RELEASE CLASSIFICATION

# Release classification

Before Questarr grabs a release unattended, it can check that the release is the game and not a
patch, a DLC, a crack, a demo or a soundtrack. This page describes the layers, how to pick a
mode, how to install the local model and what changes for existing installs.

## Layers

```text theme={null}
release name
  → rules          deterministic, always on, no model
  → local model    tf-idf + logistic regression, in-process, works offline (hybrid/local)
  → cloud model    TypeSafe Jev or a compatible model, only if still unresolved (hybrid/cloud)
  → classification (full_game | patch | dlc | unknown, plus flags such as demo or crack_only)
```

Classification is separate from download authorization. The pipeline only answers "what is
this?". The auto-download hold in `server/cron.ts` decides what to do with the answer.

A release that no layer can classify is `unknown`. It is never turned into `full_game`.

## Settings page

Everything below can be changed in Settings > Discovery & Downloads > Release classification,
without restarting: the classification type, the local model confidence, holding unclassifiable
releases, and the active-learning options (collection, cloud suggestions, the low-confidence
threshold and the candidate limit). The card also shows whether a local model is installed (and
its version) and whether a cloud provider is configured.

**An environment variable wins.** When one of the variables in this page is set, the matching
setting is shown read-only with the variable's name, and the API refuses to change it. Unset the
variable to manage the setting from the page. The low-confidence threshold and the candidate
limit have no variable: only the page changes them. Changing the type or the local confidence
rebuilds the pipeline and clears its memory cache at once.

## Modes (`CLASSIFICATION_MODE`)

| Mode | Behaviour |
| - | - |
| `hybrid` (default) | Rules, then the local model. The cloud provider is asked only when the local model abstains (below its threshold), fails, times out or is not installed. The cloud model decides alone: it does not see the local result, and it can still answer `unknown`. |
| `local` | Rules, then the local model. No request ever leaves the server. |
| `cloud` (legacy) | The historical Jev-only check, unchanged. Without a TypeSafe key nothing is checked. Needs a TypeSafe key, so it is not the default. |

The cloud provider is the TypeSafe client already configured in Settings. Its model id
(`typesafe.model`, for example `jev-latest` or an OpenRouter id) is the only thing that changes
to use another compatible model.

## One pipeline for search and auto-download

The manual search results go through the same pipeline as the auto-download check. Each result
keeps the existing categories (Main, Update, DLC, Extra), but its type now comes from the
rules, the local model and, in `hybrid` mode, the cloud layer. So the type you see in a search
is the one auto-download would act on. In `hybrid` and `cloud` mode only the top 15 results of a
search are sent through a layer that may call a provider, to bound the calls; `local` mode types
every result. In `cloud` mode the search keeps the Jev-only call it always made.

## What the auto-download hold does

In `local` and `hybrid` mode an unattended download is held for review when the pipeline says:

* `patch`, `dlc` or `expansion`, or
* a flag `crack_only`, `demo`, `soundtrack` or `other_artifact`, or
* (cloud layer only) the file size is judged implausible.

A result of `unknown` goes through, as it does today. Set `CLASSIFICATION_HOLD_UNKNOWN=true` to
hold those too. That is a behaviour change, so it is off by default. Any error in the pipeline
lets the download proceed, like the existing Jev check.

## The local model

A model trained on 964 labeled releases ships with the application, in
`shared/models/release-classifier.json` (about 1.4 MB of plain JSON, also inside the Docker
image). Nothing needs installing: `hybrid` and `local` mode use it out of the box.

### Performance

In short: out of 10 releases the local model puts about 8 in the right category (full game, patch,
DLC or unknown), against about 6 for the keyword rules alone. Its accuracy is close to the cloud model's (the cloud figure comes from an earlier split),
runs in under a millisecond and needs no account.

How it was measured: the 964 labeled releases (123 games, including console releases, software,
movies and series) were split by game, so the model is tested on games it has never seen. 187 test
releases were scored from the release name alone. The shipped file is retrained on all 964 releases, so it has no test score of its own; these
numbers come from the same method on the split. The full method and the other candidates are in
[`scripts/release-classifier-training/DECISION.md`](../scripts/release-classifier-training/DECISION.md).

| Layer | Accuracy | Macro F1 | Time per release | Size |
| - | -: | -: | -: | - |
| Rules only | 55.6% | 49.7% | 0.06 ms | none |
| Local model (tf-idf + logistic reg.) | 79.1% | 75.1% | 0.06 ms | \~1.4 MB |
| Cloud model (Jev, for comparison) | 79.5% | 74.8% | \~840 ms (network) | none |

The cloud row comes from the earlier 88-release test and was not re-measured on the new split.

* **Accuracy** is the share of releases given the right category.
* **Macro F1** also scores each category separately and then averages them, so a model cannot
  look good just by getting the most common category right. It is also a percentage, and higher
  is better. For example, always answering "patch" can score a decent accuracy but a low macro F1.
* **Cross-validation** repeats the test five times on different groups of games. On the earlier
  459-release dataset it gave 78.9% for the local model, give or take 10.9 points depending on
  which games are tested. With this little data, differences under about 5 points are not
  significant. Because the cloud figure comes from a different split, these numbers do not show
  that the two models are equivalent.
* **The 70% confidence threshold** is how sure the model must be before its answer is used.
  Below it, the release is passed on to the cloud model (`hybrid`) or left `unknown` (`local`)
  instead of being guessed.
* **The main mistake** is a full game read as a DLC; patches and unknown releases are the most
  reliable categories. More labeled games is what improves it most; the earlier 459-release
  evaluation showed accuracy still rising with its 49-game dataset.

To use your own, for example after retraining on more data (the dataset is not published),
generate the file and place it where the server looks first:

```powershell theme={null}
npx tsx scripts/release-benchmark/export-model.ts --dataset releases-labeled.csv --out release-classifier.json
```

Copy the exported file to `<database folder>/models/release-classifier.json`, or keep it anywhere
and set `CLASSIFIER_MODEL_PATH` to its location.

Lookup order: `CLASSIFIER_MODEL_PATH`, then `<database folder>/models/release-classifier.json`
(in Docker: `/app/data/models/`), then the bundled model. Restart after changing the file. The
file is validated on load; a missing or damaged one is logged and the next in the list
is tried. To replace the bundled model for everyone, overwrite `shared/models/release-classifier.json` in a pull request.

### Share your dataset

The bundled model is only as good as the 964 releases it learned from (123 games), and more labeled
games is what improves it most. If you have a dataset of your own, for example the verified labels
exported from **Settings → Release classification**, you are invited to share it so the public
classifier can be retrained on more data. Two ways, both your choice:

* **From the app (opt-in).** In the review card, tick **Share my verified labels
  automatically** (off by default). While it is ticked, a task ("Label sharing", every 6 hours,
  visible in System Tasks) sends the labels you verified since the last run. **See what will be
  sent** shows the exact rows, and **Send now** sends them without waiting. Only verified rows go,
  and each carries four fields and nothing else: the release name, the game title, the label you
  chose and the version of the local model. No account, path, address or library data. Rows
  already sent are not sent again, and a request carries at most 200 rows. The address is fixed in
  the code and the request needs no key; it follows no redirect. If the collection point cannot
  be reached, the rows stay unsent and the next run tries again. Unticking the box stops all
  sending; rows already received cannot be recalled from the app.
* **By hand.** Open an issue on [Doezer/Questarr](https://github.com/Doezer/Questarr/issues) and
  attach the CSV from **Export verified**.

Nothing is shared unless you tick the box. A shared label is a proposal: the maintainer only trains on a
class that several independent senders agree on, or after checking it, and never retrains
automatically. Only release names and their labels are needed, so with the manual route remove
anything you do not want to publish before attaching it.

Run `scripts/release-benchmark/run.ts` on a held-out split before trusting a new model (see
`scripts/release-benchmark/README.md`). Do not expect a small model to reach high automatic
precision: that is why the local layer abstains below `CLASSIFIER_LOCAL_THRESHOLD` (0.7) and why
`unknown` is never turned into `full_game`. Whether an `unknown` release is downloaded is the
hold's decision, not the classifier's: it goes through unless `CLASSIFICATION_HOLD_UNKNOWN=true`.

## Active learning: label the releases the classifier is unsure about

Off by default. When `ACTIVE_LEARNING_COLLECT=true`, the unattended auto-download check keeps
the releases it could not classify with confidence, so you can label the ones that teach the
model the most. Behaviour is unchanged when it is off: nothing is collected, no extra request is
made and the review card does not appear.

What is collected (only in `local` and `hybrid` mode, and never while a layer failed):

| Reason | Meaning |
| - | - |
| `unresolved` | No layer could say what the release is. |
| `low_confidence` | A layer decided, but below 85 % confidence. |
| `disagreement` | The rules and the local model's best guess name different classes. |
| `model_disagreement` | The cloud model answered a different class than the local model's best guess. |

Each release is stored once per game (case and spacing ignored) with its name, game, platform,
size, what each layer answered, and a count of how often it came back. A release you already
reviewed is never reset to pending. At most 5,000 are kept.

**Four kinds of label stay apart.** Rule output and the local model's guess are recorded as
signals for you to read. A cloud suggestion is stored with its model, confidence and time and is
shown as "not verified". Only a label you give in the review card (status `verified`) is a
training example.

### Review

Settings > Discovery & Downloads > Release classification review lists pending candidates, most
useful first: unresolved and disagreeing releases come before low-confidence ones, and a release
whose likely class has few verified examples gets a bonus so the thin classes fill up. For each
one you can verify a label (the local model's guess or the cloud suggestion is prefilled, not
applied), mark it ambiguous, or mark it as not an example. Ambiguous and rejected releases are
never exported.

Every time the waiting list reaches a multiple of 25 releases, each user gets one notification,
"Releases need your review". Clicking it opens this card.

`ACTIVE_LEARNING_CLOUD_SUGGEST=true` adds an "Ask the cloud model" button. It sends one release
name (with its size and platform) to the configured provider, costs what one Jev call costs, and
only when you press it. If its answer differs from the local model's best guess, the release is flagged `model_disagreement` and moves up the list. In `hybrid` mode the same flag is set without an extra request when the cloud was already asked because the local model was unsure. It needs a TypeSafe key; without one the button is hidden. If the
provider fails, the candidate is left as it was.

The API behind the card is under `/api/classification-candidates` and uses the same
authentication as the rest of `/api`.

### Export, retrain, re-evaluate

Nothing retrains or deploys a model by itself. When you have enough verified examples:

```powershell theme={null}
# 1. Download the verified examples (the "Export verified" button, same file as the API):
#    release_name,query,human_label,notes

# 2. Merge them into your labeled dataset. Duplicates are skipped, a different label for a
#    release already in the dataset is reported as a conflict and never overwrites it, and
#    games that appear in the held-out file are not added.
npx tsx scripts/release-benchmark/merge-candidates.ts --dataset releases-labeled.csv --candidates verified-candidates.csv --heldout releases-heldout.csv --out releases-labeled.csv

# 3. Re-evaluate on the held-out set, then compare with the previous model
npx tsx scripts/release-benchmark/run.ts --dataset releases-heldout.csv --classifiers "deterministic,tfidf"

# 4. Only if it is better, generate the model and install it (see "Install the local model")
npx tsx scripts/release-benchmark/export-model.ts --dataset releases-labeled.csv --out release-classifier.json
```

#### Labels shared by other installs (collector)

If you run the label collector (Doezer/Questarr-mgmt#60, repository `questarr-label-collector`), step 1
above is replaced by an export from its database. `consolidate.sql` keeps only labels confirmed by at
least 3 different senders with no contradicting majority. In PowerShell, from the collector's folder
(this snippet is untested on real data, check the CSV before merging):

```powershell theme={null}
$rows = (npx wrangler d1 execute questarr-labels --remote --file consolidate.sql --json | ConvertFrom-Json)[0].results
# Neutralize spreadsheet formulas in any cell
foreach ($r in $rows) { foreach ($p in 'release_name','query','human_label') { if ($r.$p -match '^[=+\-@]') { $r.$p = "'" + $r.$p } } }
$rows | Select-Object release_name, query, human_label | Export-Csv verified-candidates.csv -NoTypeInformation -Encoding utf8
```

Then follow steps 2 to 4 above with `verified-candidates.csv`. Nothing from the collector reaches a
model by itself: the merge reports conflicts, and you decide what to retrain and ship.

Keep the **held-out evaluation set in its own file** and never merge candidates into it: the
benchmark splits by game, and a game that is both in the held-out file and in the training file
would leak. The merge script enforces that for games already in the held-out file. Growing the
held-out file is a deliberate step you do by hand.

### Delete

"Delete all candidates" in the card (or `DELETE /api/classification-candidates`, optionally
`?status=rejected`) removes the collected releases. Turn collection off in the Settings page (or by unsetting
`ACTIVE_LEARNING_COLLECT`); existing candidates stay until you delete them.

## Operations

* Timeouts: each model layer is bounded to 10 s, a failing layer counts as "unresolved".
* Cache: two levels, shared by every mode. Results are kept in memory for 24 h (2,000 entries)
  and in the `classification_cache` table (SQLite or Postgres) for 7 days (20,000 rows), so a
  restart or a Docker update no longer repeats paid cloud calls. The key includes the pipeline
  version, the mode, the local model version and confidence bar, the cloud model id and a
  fingerprint of the provider endpoint and model, and the release name, size, platform and game
  title. Installing a new model, changing the provider endpoint or model, the mode or the confidence bar, or bumping
  the pipeline version therefore never serves an older answer; those rows just age out. Expired
  and surplus rows are pruned at most once an hour. Results from a failing layer (error, timeout,
  unusable answer) are never stored, a stored verdict is stored as it was (an `unknown` stays
  `unknown`), and a table failure only means a cache miss. Active-learning candidates stay in
  their own table: the cache holds pipeline verdicts, not labels. **Clear classification cache**
  on the Release classification card empties both levels.
* Cloud mode: the historical Jev call (`analyzeRelease`, also used by search and auto-search)
  uses the same two levels, keyed by provider fingerprint, release name, size and platform. A
  failed call is never cached.
* Logs: one `debug` line per release in the `classification` module with the layers consulted,
  class, confidence, source and model version.
* Docker: no new container and no native dependency. The model file ships in the image; an
  optional one in the data volume overrides it.

## Compatibility

* Default `CLASSIFICATION_MODE=hybrid`. It needs no key: without a TypeSafe key and without a
  model file only the rules run. Releases the rules identify as a patch, DLC, crack, demo or
  soundtrack are held for review before an unattended download, which the Jev-only check never did
  without a key. Set `CLASSIFICATION_MODE=cloud` to get the previous behaviour back.
* API and UI are unchanged unless `ACTIVE_LEARNING_COLLECT` is set (see above).
* Search-result AI badges (`enrichWithAiAnalysis`) still use Jev only, in every mode.
* One new table, `classification_candidates` (SQLite and Postgres migrations), used only by active learning. Settings are saved in the existing system configuration (`classification.settings`), so no migration.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.