Skip to main content

Release classification

Before Questarr grabs a release unattended, it can check that the release is the game and not a patch, a DLC, a crack, a demo or a soundtrack. This page describes the layers, how to pick a mode, how to install the local model and what changes for existing installs.

Layers

Classification is separate from download authorization. The pipeline only answers “what is this?”. The auto-download hold in server/cron.ts decides what to do with the answer. A release that no layer can classify is unknown. It is never turned into full_game.

Settings page

Everything below can be changed in Settings > Discovery & Downloads > Release classification, without restarting: the classification type, the local model confidence, holding unclassifiable releases, and the active-learning options (collection, cloud suggestions, the low-confidence threshold and the candidate limit). The card also shows whether a local model is installed (and its version) and whether a cloud provider is configured. An environment variable wins. When one of the variables in this page is set, the matching setting is shown read-only with the variable’s name, and the API refuses to change it. Unset the variable to manage the setting from the page. The low-confidence threshold and the candidate limit have no variable: only the page changes them. Changing the type or the local confidence rebuilds the pipeline and clears its memory cache at once.

Modes (CLASSIFICATION_MODE)

The cloud provider is the TypeSafe client already configured in Settings. Its model id (typesafe.model, for example jev-latest or an OpenRouter id) is the only thing that changes to use another compatible model.

One pipeline for search and auto-download

The manual search results go through the same pipeline as the auto-download check. Each result keeps the existing categories (Main, Update, DLC, Extra), but its type now comes from the rules, the local model and, in hybrid mode, the cloud layer. So the type you see in a search is the one auto-download would act on. In hybrid and cloud mode only the top 15 results of a search are sent through a layer that may call a provider, to bound the calls; local mode types every result. In cloud mode the search keeps the Jev-only call it always made.

What the auto-download hold does

In local and hybrid mode an unattended download is held for review when the pipeline says:
  • patch, dlc or expansion, or
  • a flag crack_only, demo, soundtrack or other_artifact, or
  • (cloud layer only) the file size is judged implausible.
A result of unknown goes through, as it does today. Set CLASSIFICATION_HOLD_UNKNOWN=true to hold those too. That is a behaviour change, so it is off by default. Any error in the pipeline lets the download proceed, like the existing Jev check.

The local model

A model trained on 964 labeled releases ships with the application, in shared/models/release-classifier.json (about 1.4 MB of plain JSON, also inside the Docker image). Nothing needs installing: hybrid and local mode use it out of the box.

Performance

In short: out of 10 releases the local model puts about 8 in the right category (full game, patch, DLC or unknown), against about 6 for the keyword rules alone. Its accuracy is close to the cloud model’s (the cloud figure comes from an earlier split), runs in under a millisecond and needs no account. How it was measured: the 964 labeled releases (123 games, including console releases, software, movies and series) were split by game, so the model is tested on games it has never seen. 187 test releases were scored from the release name alone. The shipped file is retrained on all 964 releases, so it has no test score of its own; these numbers come from the same method on the split. The full method and the other candidates are in scripts/release-classifier-training/DECISION.md. The cloud row comes from the earlier 88-release test and was not re-measured on the new split.
  • Accuracy is the share of releases given the right category.
  • Macro F1 also scores each category separately and then averages them, so a model cannot look good just by getting the most common category right. It is also a percentage, and higher is better. For example, always answering “patch” can score a decent accuracy but a low macro F1.
  • Cross-validation repeats the test five times on different groups of games. On the earlier 459-release dataset it gave 78.9% for the local model, give or take 10.9 points depending on which games are tested. With this little data, differences under about 5 points are not significant. Because the cloud figure comes from a different split, these numbers do not show that the two models are equivalent.
  • The 70% confidence threshold is how sure the model must be before its answer is used. Below it, the release is passed on to the cloud model (hybrid) or left unknown (local) instead of being guessed.
  • The main mistake is a full game read as a DLC; patches and unknown releases are the most reliable categories. More labeled games is what improves it most; the earlier 459-release evaluation showed accuracy still rising with its 49-game dataset.
To use your own, for example after retraining on more data (the dataset is not published), generate the file and place it where the server looks first:
Copy the exported file to <database folder>/models/release-classifier.json, or keep it anywhere and set CLASSIFIER_MODEL_PATH to its location. Lookup order: CLASSIFIER_MODEL_PATH, then <database folder>/models/release-classifier.json (in Docker: /app/data/models/), then the bundled model. Restart after changing the file. The file is validated on load; a missing or damaged one is logged and the next in the list is tried. To replace the bundled model for everyone, overwrite shared/models/release-classifier.json in a pull request.

Share your dataset

The bundled model is only as good as the 964 releases it learned from (123 games), and more labeled games is what improves it most. If you have a dataset of your own, for example the verified labels exported from Settings → Release classification, you are invited to share it so the public classifier can be retrained on more data. Two ways, both your choice:
  • From the app (opt-in). In the review card, tick Share my verified labels automatically (off by default). While it is ticked, a task (“Label sharing”, every 6 hours, visible in System Tasks) sends the labels you verified since the last run. See what will be sent shows the exact rows, and Send now sends them without waiting. Only verified rows go, and each carries four fields and nothing else: the release name, the game title, the label you chose and the version of the local model. No account, path, address or library data. Rows already sent are not sent again, and a request carries at most 200 rows. The address is fixed in the code and the request needs no key; it follows no redirect. If the collection point cannot be reached, the rows stay unsent and the next run tries again. Unticking the box stops all sending; rows already received cannot be recalled from the app.
  • By hand. Open an issue on Doezer/Questarr and attach the CSV from Export verified.
Nothing is shared unless you tick the box. A shared label is a proposal: the maintainer only trains on a class that several independent senders agree on, or after checking it, and never retrains automatically. Only release names and their labels are needed, so with the manual route remove anything you do not want to publish before attaching it. Run scripts/release-benchmark/run.ts on a held-out split before trusting a new model (see scripts/release-benchmark/README.md). Do not expect a small model to reach high automatic precision: that is why the local layer abstains below CLASSIFIER_LOCAL_THRESHOLD (0.7) and why unknown is never turned into full_game. Whether an unknown release is downloaded is the hold’s decision, not the classifier’s: it goes through unless CLASSIFICATION_HOLD_UNKNOWN=true.

Active learning: label the releases the classifier is unsure about

Off by default. When ACTIVE_LEARNING_COLLECT=true, the unattended auto-download check keeps the releases it could not classify with confidence, so you can label the ones that teach the model the most. Behaviour is unchanged when it is off: nothing is collected, no extra request is made and the review card does not appear. What is collected (only in local and hybrid mode, and never while a layer failed): Each release is stored once per game (case and spacing ignored) with its name, game, platform, size, what each layer answered, and a count of how often it came back. A release you already reviewed is never reset to pending. At most 5,000 are kept. Four kinds of label stay apart. Rule output and the local model’s guess are recorded as signals for you to read. A cloud suggestion is stored with its model, confidence and time and is shown as “not verified”. Only a label you give in the review card (status verified) is a training example.

Review

Settings > Discovery & Downloads > Release classification review lists pending candidates, most useful first: unresolved and disagreeing releases come before low-confidence ones, and a release whose likely class has few verified examples gets a bonus so the thin classes fill up. For each one you can verify a label (the local model’s guess or the cloud suggestion is prefilled, not applied), mark it ambiguous, or mark it as not an example. Ambiguous and rejected releases are never exported. Every time the waiting list reaches a multiple of 25 releases, each user gets one notification, “Releases need your review”. Clicking it opens this card. ACTIVE_LEARNING_CLOUD_SUGGEST=true adds an “Ask the cloud model” button. It sends one release name (with its size and platform) to the configured provider, costs what one Jev call costs, and only when you press it. If its answer differs from the local model’s best guess, the release is flagged model_disagreement and moves up the list. In hybrid mode the same flag is set without an extra request when the cloud was already asked because the local model was unsure. It needs a TypeSafe key; without one the button is hidden. If the provider fails, the candidate is left as it was. The API behind the card is under /api/classification-candidates and uses the same authentication as the rest of /api.

Export, retrain, re-evaluate

Nothing retrains or deploys a model by itself. When you have enough verified examples:

Labels shared by other installs (collector)

If you run the label collector (Doezer/Questarr-mgmt#60, repository questarr-label-collector), step 1 above is replaced by an export from its database. consolidate.sql keeps only labels confirmed by at least 3 different senders with no contradicting majority. In PowerShell, from the collector’s folder (this snippet is untested on real data, check the CSV before merging):
Then follow steps 2 to 4 above with verified-candidates.csv. Nothing from the collector reaches a model by itself: the merge reports conflicts, and you decide what to retrain and ship. Keep the held-out evaluation set in its own file and never merge candidates into it: the benchmark splits by game, and a game that is both in the held-out file and in the training file would leak. The merge script enforces that for games already in the held-out file. Growing the held-out file is a deliberate step you do by hand.

Delete

“Delete all candidates” in the card (or DELETE /api/classification-candidates, optionally ?status=rejected) removes the collected releases. Turn collection off in the Settings page (or by unsetting ACTIVE_LEARNING_COLLECT); existing candidates stay until you delete them.

Operations

  • Timeouts: each model layer is bounded to 10 s, a failing layer counts as “unresolved”.
  • Cache: two levels, shared by every mode. Results are kept in memory for 24 h (2,000 entries) and in the classification_cache table (SQLite or Postgres) for 7 days (20,000 rows), so a restart or a Docker update no longer repeats paid cloud calls. The key includes the pipeline version, the mode, the local model version and confidence bar, the cloud model id and a fingerprint of the provider endpoint and model, and the release name, size, platform and game title. Installing a new model, changing the provider endpoint or model, the mode or the confidence bar, or bumping the pipeline version therefore never serves an older answer; those rows just age out. Expired and surplus rows are pruned at most once an hour. Results from a failing layer (error, timeout, unusable answer) are never stored, a stored verdict is stored as it was (an unknown stays unknown), and a table failure only means a cache miss. Active-learning candidates stay in their own table: the cache holds pipeline verdicts, not labels. Clear classification cache on the Release classification card empties both levels.
  • Cloud mode: the historical Jev call (analyzeRelease, also used by search and auto-search) uses the same two levels, keyed by provider fingerprint, release name, size and platform. A failed call is never cached.
  • Logs: one debug line per release in the classification module with the layers consulted, class, confidence, source and model version.
  • Docker: no new container and no native dependency. The model file ships in the image; an optional one in the data volume overrides it.

Compatibility

  • Default CLASSIFICATION_MODE=hybrid. It needs no key: without a TypeSafe key and without a model file only the rules run. Releases the rules identify as a patch, DLC, crack, demo or soundtrack are held for review before an unattended download, which the Jev-only check never did without a key. Set CLASSIFICATION_MODE=cloud to get the previous behaviour back.
  • API and UI are unchanged unless ACTIVE_LEARNING_COLLECT is set (see above).
  • Search-result AI badges (enrichWithAiAnalysis) still use Jev only, in every mode.
  • One new table, classification_candidates (SQLite and Postgres migrations), used only by active learning. Settings are saved in the existing system configuration (classification.settings), so no migration.