Local LLM: Task-specific model shortlist

Detailed practical guide

Prepare your workspace and inputs

Choose two installed model names from /api/tags and check their license and hardware fit first. Save the script as shortlist.py. Run python3 shortlist.py MODEL_A MODEL_B using actual installed names. Each model receives the same three cases; this tiny fixture is a smoke test, not an adequate production evaluation set.

Platform context: Python 3.10+ and a running local Ollama service at 127.0.0.1:11434. At least one approved model must already be installed; the script does not pull models.

Scenario to reproduce or investigate

A model is selected from a general benchmark, then fails the exact classification format required by an application. Measure the task contract directly and retain artifact identity.

1. Verify the local endpoint and fixed answer key

   List installed models and record their names and digests. The task has three labels: positive, negative and neutral. Require a JSON object with only a label field. The script uses the same English sentences and expected labels for every candidate and requests nonstreamed output.

   Command, configuration or input

   curl http://127.0.0.1:11434/api/tags
   
   Fixtures:
   The replacement works perfectly. → positive
   The replacement failed again. → negative
   The replacement arrived on Tuesday. → neutral

   Expected result · ผลที่ควรเห็น

   Every candidate is present in the local inventory and has a recorded digest. No model download is started by the benchmark.

   If it fails · เมื่อไม่ผ่าน

   If the endpoint is unreachable, start or connect to the approved local runtime. Do not expose the runtime publicly to work around a local connection failure.

2. Run and retain exact per-case results

   This complete script calls the real local HTTP API and prints JSON Lines. It checks the returned structure outside the model, records raw output for inspection and avoids importing a third-party SDK. Cold-start effects are included in each measured request and must be interpreted separately from steady-state throughput.

   Command, configuration or input

   import json
   import sys
   from time import perf_counter
   from urllib.request import Request, urlopen
   
   BASE = "http://127.0.0.1:11434"
   with urlopen(BASE + "/api/tags", timeout=10) as response:
       inventory = {item["name"]: item for item in json.load(response)["models"]}
   cases = [("The replacement works perfectly.", "positive"),
            ("The replacement failed again.", "negative"),
            ("The replacement arrived on Tuesday.", "neutral")]
   names = sys.argv[1:]
   if not names:
       raise SystemExit("Usage: python3 shortlist.py INSTALLED_MODEL [INSTALLED_MODEL...]")
   for name in names:
       if name not in inventory:
           raise SystemExit("Model is not installed: " + name)
       for index, (text, expected) in enumerate(cases, 1):
           payload = {"model": name, "stream": False, "format": "json",
                      "options": {"temperature": 0},
                      "prompt": "Classify the sentiment. Return only a JSON object with one key label, "
                                "using positive, negative or neutral. Text: " + text}
           request = Request(BASE + "/api/generate", data=json.dumps(payload).encode(),
                             headers={"Content-Type": "application/json"})
           started = perf_counter()
           with urlopen(request, timeout=180) as response:
               result = json.load(response)
           elapsed = perf_counter() - started
           raw = result.get("response", "")
           try:
               parsed = json.loads(raw)
               passed = isinstance(parsed, dict) and set(parsed) == {"label"} and parsed["label"] == expected
           except (ValueError, TypeError):
               passed = False
           print(json.dumps({"model": name, "digest": inventory[name]["digest"],
                 "case": index, "expected": expected, "passed": passed,
                 "elapsed_seconds": round(elapsed, 4), "raw": raw,
                 "load_duration_ns": result.get("load_duration"),
                 "eval_count": result.get("eval_count"),
                 "eval_duration_ns": result.get("eval_duration")}), flush=True)
   

   Expected result · ผลที่ควรเห็น

   There are three result rows per candidate, with its digest and independent passed flag. A model can fail a case even when the API request succeeds.

   If it fails · เมื่อไม่ผ่าน

   HTTP errors or a timeout mean that case did not complete; record them separately and do not silently discard the model. If output is malformed, inspect raw and keep the failed format check instead of accepting an approximate answer.

3. Build the shortlist from a representative held-out set

   Repeat with your own approved, labeled task fixtures, including ambiguous, multilingual and unsupported inputs. Save runtime version, hardware and configuration next to the JSON Lines. Run a separate warm-up if comparing steady-state latency and label it excluded. Select only models that meet the required quality gate before comparing time or memory; inspect actual runtime memory separately because this script does not measure it.

   Command, configuration or input

   Candidate | digest | required cases passed | format failures | cold/warm policy | latency | measured memory | license decision

   Expected result · ผลที่ควรเห็น

   The shortlist can be reproduced from the same fixtures and artifacts, with failed cases and missing measurements visible.

   If it fails · เมื่อไม่ผ่าน

   A perfect three-sentence score is insufficient for deployment. Expand the held-out set before selecting a production model and repeat after any model or runtime change.

Recovery and cleanup

The script sends local inference requests and prints output but does not alter model configuration or download artifacts. Stop the run if it exceeds the approved resource window and retain partial rows labeled incomplete.

Apply to actual work and known limits

Replace only the task fixtures and contract for another use case. This code has been syntax-checked; a real Ollama runtime and model were not available for execution here, so model accuracy and latency remain untested.

Authored against the cited documentation. Execution on the stated vendor product or physical equipment remains to be verified.