Release classification
Before Questarr grabs a release unattended, it can check that the release is the game and not a patch, a DLC, a crack, a demo or a soundtrack. This page describes the layers, how to pick a mode, how to install the local model and what changes for existing installs.Layers
server/cron.ts decides what to do with the answer.
A release that no layer can classify is unknown. It is never turned into full_game.
Settings page
Everything below can be changed in Settings > Discovery & Downloads > Release classification, without restarting: the classification type, the local model confidence, holding unclassifiable releases, and the active-learning options (collection, cloud suggestions, the low-confidence threshold and the candidate limit). The card also shows whether a local model is installed (and its version) and whether a cloud provider is configured. An environment variable wins. When one of the variables in this page is set, the matching setting is shown read-only with the variable’s name, and the API refuses to change it. Unset the variable to manage the setting from the page. The low-confidence threshold and the candidate limit have no variable: only the page changes them. Changing the type or the local confidence rebuilds the pipeline and clears its memory cache at once.Modes (CLASSIFICATION_MODE)
The cloud provider is the TypeSafe client already configured in Settings. Its model id
(
typesafe.model, for example jev-latest or an OpenRouter id) is the only thing that changes
to use another compatible model.
One pipeline for search and auto-download
The manual search results go through the same pipeline as the auto-download check. Each result keeps the existing categories (Main, Update, DLC, Extra), but its type now comes from the rules, the local model and, inhybrid mode, the cloud layer. So the type you see in a search
is the one auto-download would act on. In hybrid and cloud mode only the top 15 results of a
search are sent through a layer that may call a provider, to bound the calls; local mode types
every result. In cloud mode the search keeps the Jev-only call it always made.
What the auto-download hold does
Inlocal and hybrid mode an unattended download is held for review when the pipeline says:
patch,dlcorexpansion, or- a flag
crack_only,demo,soundtrackorother_artifact, or - (cloud layer only) the file size is judged implausible.
unknown goes through, as it does today. Set CLASSIFICATION_HOLD_UNKNOWN=true to
hold those too. That is a behaviour change, so it is off by default. Any error in the pipeline
lets the download proceed, like the existing Jev check.
The local model
A model trained on 964 labeled releases ships with the application, inshared/models/release-classifier.json (about 1.4 MB of plain JSON, also inside the Docker
image). Nothing needs installing: hybrid and local mode use it out of the box.
Performance
In short: out of 10 releases the local model puts about 8 in the right category (full game, patch, DLC or unknown), against about 6 for the keyword rules alone. Its accuracy is close to the cloud model’s (the cloud figure comes from an earlier split), runs in under a millisecond and needs no account. How it was measured: the 964 labeled releases (123 games, including console releases, software, movies and series) were split by game, so the model is tested on games it has never seen. 187 test releases were scored from the release name alone. The shipped file is retrained on all 964 releases, so it has no test score of its own; these numbers come from the same method on the split. The full method and the other candidates are inscripts/release-classifier-training/DECISION.md.
The cloud row comes from the earlier 88-release test and was not re-measured on the new split.
- Accuracy is the share of releases given the right category.
- Macro F1 also scores each category separately and then averages them, so a model cannot look good just by getting the most common category right. It is also a percentage, and higher is better. For example, always answering “patch” can score a decent accuracy but a low macro F1.
- Cross-validation repeats the test five times on different groups of games. On the earlier 459-release dataset it gave 78.9% for the local model, give or take 10.9 points depending on which games are tested. With this little data, differences under about 5 points are not significant. Because the cloud figure comes from a different split, these numbers do not show that the two models are equivalent.
- The 70% confidence threshold is how sure the model must be before its answer is used.
Below it, the release is passed on to the cloud model (
hybrid) or leftunknown(local) instead of being guessed. - The main mistake is a full game read as a DLC; patches and unknown releases are the most reliable categories. More labeled games is what improves it most; the earlier 459-release evaluation showed accuracy still rising with its 49-game dataset.
<database folder>/models/release-classifier.json, or keep it anywhere
and set CLASSIFIER_MODEL_PATH to its location.
Lookup order: CLASSIFIER_MODEL_PATH, then <database folder>/models/release-classifier.json
(in Docker: /app/data/models/), then the bundled model. Restart after changing the file. The
file is validated on load; a missing or damaged one is logged and the next in the list
is tried. To replace the bundled model for everyone, overwrite shared/models/release-classifier.json in a pull request.
Share your dataset
The bundled model is only as good as the 964 releases it learned from (123 games), and more labeled games is what improves it most. If you have a dataset of your own, for example the verified labels exported from Settings → Release classification, you are invited to share it so the public classifier can be retrained on more data. Two ways, both your choice:- From the app (opt-in). In the review card, tick Share my verified labels automatically (off by default). While it is ticked, a task (“Label sharing”, every 6 hours, visible in System Tasks) sends the labels you verified since the last run. See what will be sent shows the exact rows, and Send now sends them without waiting. Only verified rows go, and each carries four fields and nothing else: the release name, the game title, the label you chose and the version of the local model. No account, path, address or library data. Rows already sent are not sent again, and a request carries at most 200 rows. The address is fixed in the code and the request needs no key; it follows no redirect. If the collection point cannot be reached, the rows stay unsent and the next run tries again. Unticking the box stops all sending; rows already received cannot be recalled from the app.
- By hand. Open an issue on Doezer/Questarr and attach the CSV from Export verified.
scripts/release-benchmark/run.ts on a held-out split before trusting a new model (see
scripts/release-benchmark/README.md). Do not expect a small model to reach high automatic
precision: that is why the local layer abstains below CLASSIFIER_LOCAL_THRESHOLD (0.7) and why
unknown is never turned into full_game. Whether an unknown release is downloaded is the
hold’s decision, not the classifier’s: it goes through unless CLASSIFICATION_HOLD_UNKNOWN=true.
Active learning: label the releases the classifier is unsure about
Off by default. WhenACTIVE_LEARNING_COLLECT=true, the unattended auto-download check keeps
the releases it could not classify with confidence, so you can label the ones that teach the
model the most. Behaviour is unchanged when it is off: nothing is collected, no extra request is
made and the review card does not appear.
What is collected (only in local and hybrid mode, and never while a layer failed):
Each release is stored once per game (case and spacing ignored) with its name, game, platform,
size, what each layer answered, and a count of how often it came back. A release you already
reviewed is never reset to pending. At most 5,000 are kept.
Four kinds of label stay apart. Rule output and the local model’s guess are recorded as
signals for you to read. A cloud suggestion is stored with its model, confidence and time and is
shown as “not verified”. Only a label you give in the review card (status
verified) is a
training example.
Review
Settings > Discovery & Downloads > Release classification review lists pending candidates, most useful first: unresolved and disagreeing releases come before low-confidence ones, and a release whose likely class has few verified examples gets a bonus so the thin classes fill up. For each one you can verify a label (the local model’s guess or the cloud suggestion is prefilled, not applied), mark it ambiguous, or mark it as not an example. Ambiguous and rejected releases are never exported. Every time the waiting list reaches a multiple of 25 releases, each user gets one notification, “Releases need your review”. Clicking it opens this card.ACTIVE_LEARNING_CLOUD_SUGGEST=true adds an “Ask the cloud model” button. It sends one release
name (with its size and platform) to the configured provider, costs what one Jev call costs, and
only when you press it. If its answer differs from the local model’s best guess, the release is flagged model_disagreement and moves up the list. In hybrid mode the same flag is set without an extra request when the cloud was already asked because the local model was unsure. It needs a TypeSafe key; without one the button is hidden. If the
provider fails, the candidate is left as it was.
The API behind the card is under /api/classification-candidates and uses the same
authentication as the rest of /api.
Export, retrain, re-evaluate
Nothing retrains or deploys a model by itself. When you have enough verified examples:Labels shared by other installs (collector)
If you run the label collector (Doezer/Questarr-mgmt#60, repositoryquestarr-label-collector), step 1
above is replaced by an export from its database. consolidate.sql keeps only labels confirmed by at
least 3 different senders with no contradicting majority. In PowerShell, from the collector’s folder
(this snippet is untested on real data, check the CSV before merging):
verified-candidates.csv. Nothing from the collector reaches a
model by itself: the merge reports conflicts, and you decide what to retrain and ship.
Keep the held-out evaluation set in its own file and never merge candidates into it: the
benchmark splits by game, and a game that is both in the held-out file and in the training file
would leak. The merge script enforces that for games already in the held-out file. Growing the
held-out file is a deliberate step you do by hand.
Delete
“Delete all candidates” in the card (orDELETE /api/classification-candidates, optionally
?status=rejected) removes the collected releases. Turn collection off in the Settings page (or by unsetting
ACTIVE_LEARNING_COLLECT); existing candidates stay until you delete them.
Operations
- Timeouts: each model layer is bounded to 10 s, a failing layer counts as “unresolved”.
- Cache: two levels, shared by every mode. Results are kept in memory for 24 h (2,000 entries)
and in the
classification_cachetable (SQLite or Postgres) for 7 days (20,000 rows), so a restart or a Docker update no longer repeats paid cloud calls. The key includes the pipeline version, the mode, the local model version and confidence bar, the cloud model id and a fingerprint of the provider endpoint and model, and the release name, size, platform and game title. Installing a new model, changing the provider endpoint or model, the mode or the confidence bar, or bumping the pipeline version therefore never serves an older answer; those rows just age out. Expired and surplus rows are pruned at most once an hour. Results from a failing layer (error, timeout, unusable answer) are never stored, a stored verdict is stored as it was (anunknownstaysunknown), and a table failure only means a cache miss. Active-learning candidates stay in their own table: the cache holds pipeline verdicts, not labels. Clear classification cache on the Release classification card empties both levels. - Cloud mode: the historical Jev call (
analyzeRelease, also used by search and auto-search) uses the same two levels, keyed by provider fingerprint, release name, size and platform. A failed call is never cached. - Logs: one
debugline per release in theclassificationmodule with the layers consulted, class, confidence, source and model version. - Docker: no new container and no native dependency. The model file ships in the image; an optional one in the data volume overrides it.
Compatibility
- Default
CLASSIFICATION_MODE=hybrid. It needs no key: without a TypeSafe key and without a model file only the rules run. Releases the rules identify as a patch, DLC, crack, demo or soundtrack are held for review before an unattended download, which the Jev-only check never did without a key. SetCLASSIFICATION_MODE=cloudto get the previous behaviour back. - API and UI are unchanged unless
ACTIVE_LEARNING_COLLECTis set (see above). - Search-result AI badges (
enrichWithAiAnalysis) still use Jev only, in every mode. - One new table,
classification_candidates(SQLite and Postgres migrations), used only by active learning. Settings are saved in the existing system configuration (classification.settings), so no migration.