AI Engineering

Over the past several months I have operated a small fleet of software agents performing a narrowly scoped but operationally demanding task: given a photographic scan of a trading card, determine its identity, assign a price, and record it in an inventory system. The task’s narrowness is, in my view, precisely what makes it instructive. Card cataloging belongs to a comparatively uncommon class of practical agentic tasks in which a ground-truth answer exists and is independently checkable — each card possesses exactly one correct name, set, and collector number, verifiable against public catalogs. This property shaped nearly every design decision described below, and motivates my writing up the system for practitioners building comparable agentic pipelines.

1. System overview

The system operates on Multica [1], an open-source platform for managing software agents as task-assignable collaborators — issues, assignments, and comments function analogously to a human team’s workflow. Each agent executes through Pi [2], a minimal open-source agent harness, against a language model served on locally controlled hardware rather than a hosted commercial API (§6 characterizes this configuration in detail). Ground-truth identity information is drawn from public sources — Scryfall for Magic: The Gathering, TCGCSV for other trading card games — together with a privately maintained vector index of annotated card images accumulated over the course of operation. The complete stack thus comprises five components: an issue-tracking board, an agent harness, a locally served open-weight model, an image index, and a small number of external reference catalogs.

The unit of work is deliberately minimal: one card corresponds to exactly one issue. A scan is attached to the issue; the card’s identity is withheld, and exactly one agent is assigned. I consider fine-grained task decomposition of this kind to be underappreciated in agentic system design. It confers four properties simultaneously: retries become inexpensive, failures remain isolated to a single unit, progress is directly observable, and each completed task’s transcript constitutes a well-formed training example. Coarser decomposition — assigning an agent to “process a batch” — forfeits all four.

flowchart TD
  A[Scan filed as one issue] --> B[Card Analyst]
  B --> C[Identify: image read + vector match + catalog cross-check]
  C --> D[Record card and cite every source]
  D --> E{Validator re-fetches and verifies}
  E -->|all claims sourced| F[Done]
  E -->|substance fail, under 2 rounds| B
  E -->|still failing after 2 rounds| G[Blocked for a human]
  H[Queue TTL expires an unclaimed task] -.->|watchdog re-enqueues| B

Fig 1. The complete pipeline. The Card Analyst hands off to the Validator, which returns one of three outcomes — pass, revision request, or escalation to a human reviewer — and a Watchdog agent re-enqueues issues silently dropped by the task queue (§5).

2. Grounding procedure

The first agent, the Card Analyst, is responsible for identification and recording. The identification step itself is straightforward; the more consequential design choices concern the constraints imposed on it, described below.

• Two independent reads, cross-checked prior to acceptance. The agent reads the image directly and, separately, embeds the scan and performs a nearest-neighbor lookup against the annotated corpus (cosine similarity ≈1.0 indicates an identical image; ≥0.85 indicates probable identity). This read is then confirmed against an external catalog by set code and collector number. Acceptance requires agreement among all three signals.
• Every factual claim must cite an independently re-fetchable source. A Scryfall URI or a TCGCSV product identifier (with category and group) is required, rather than an unsupported assertion of recognition. Where no source can be located, the agent records only what is directly legible from the scan and states explicitly that the remainder could not be sourced. I take a confidently stated incorrect fact to be strictly worse than an honestly reported failure to identify.
• Prices are never estimated. A price is recorded only when a genuine market datapoint exists (a Scryfall prices.usd field, a TCGplayer price row); extrapolation from comparable cards, corpus averages, or rarity class is disallowed. An unpriced record is an acceptable outcome; a fabricated price constitutes a failure by definition.

A subtler design principle underlies this procedure and, in my view, generalizes beyond this application: the model should not be asked to infer a quantity that a cheap deterministic input already determines. Foil status and condition are not inferred from the image; both are supplied by the operator, since cards are physically pre-sorted by these attributes prior to scanning, and the agent is explicitly instructed not to infer foil status from surface glare. Substituting a known input for a model judgment eliminates an entire class of error at no cost.

3. Verification as an independent procedure

I consider the following design pattern to be the most broadly generalizable finding of this work: the agent performing a task should not be the agent that determines whether the task was performed correctly. A second, independently instantiated agent — the Validator — is triggered upon handoff from the Card Analyst. Its function is restricted to confirming that every claim made by the analyst is supported by the source cited for it: each citation is re-fetched independently and checked against the claimed name, collector number, card text, and price. A claim lacking a citation fails; a citation that does not support the claim fails; an estimated price fails.

Two further constraints are necessary for this arrangement to function in practice.

First, failure is restricted to substantive grounds only, never presentation. The Validator is explicitly prohibited from failing a ticket for formatting, wording, layout, or stylistic preference. Exactly four failure conditions are defined — an uncited claim, a claim contradicted by its cited source, a fabricated price, or a missing traceability tag — and no others. This constraint is more consequential than it may initially appear: the default failure mode of an LLM-based critic is to identify presentational deficiencies rather than substantive ones, and restricting the checker to substance is what prevents it from degenerating into a pedantic gate rather than a useful one.

def validate(claim, source):
    if claim.citation is None:            # uncited: fail
        return FAIL
    if not source.confirms(claim.value):  # source contradicts: fail
        return FAIL
    if claim.is_price and not source.is_market_datapoint:
        return FAIL                       # estimated or fabricated price: fail
    return PASS   # never fail on wording, layout, or formattingCode language: Python (python)

Second, a bounded retry mechanism. Each rejection by the Validator increments a counter; after two rejections, the system escalates to a human reviewer rather than permitting indefinite iteration. Adversarially paired agents require an explicit termination condition, absent which they will iterate without converging on precisely the cases that most warrant human attention.

The rationale for separating execution from evaluation is that self-assessment is unreliable: a model tends toward leniency with respect to its own output and shares its own systematic blind spots. A second agent operating under a narrowly distinct mandate — verification against external sources, rather than cataloging — detects a measurable fraction of errors precisely because its judgments are anchored in re-fetched ground truth rather than in the same priors that produced the original claim.

4. Structured output and its limits

In an earlier iteration, both agents reported findings as unstructured free-text comments. Content converged reliably across runs, but layout did not — headings in one instance, a table in the next, the verdict positioned inconsistently. I subsequently standardized on a fixed template comprising required fields (identity, operator-supplied inputs, price, sources, record identifier) and a verdict-first format for the Validator’s output.

## Catalog: CARD_NAME — SET-NUMBER

### Identity
- Game / Name / Set / Number / Rarity

### Operator inputs
- Condition · Foil · Batch      (deterministic — from the pre-sorted piles)

### Price
- $X.XX USD   —or—   Unpriced (no market datapoint)

### Sources
- every claim cites a Scryfall URI or a TCGCSV product id

### binder
- card_id: CARD_IDCode language: Markdown (markdown)

A predictable failure mode follows from this standardization: the temptation to have the Validator enforce the template. This should be avoided. Doing so directly contradicts the substance-only failure criterion (§3) and converts the quality gate into a formatting linter capable of rejecting correct work over an incorrectly labeled heading. I instead treat the template as a strong expectation for the producing agent and explicitly exclude it from the checking agent’s failure conditions — yielding consistency where it benefits human readers and downstream parsing, without introducing new failure modes.

A final practice worth noting: agent instructions are maintained as version-controlled source. Instruction files reside in a git repository and are synchronized verbatim into the platform, such that prompt modifications are subject to review, diffing, and reversion in the same manner as application code. One consequence of this synchronization step deserves mention: the platform strips angle-bracket tokens as though they were HTML markup, causing placeholder tokens of the form <card_id> to be silently deleted; I resolved this by adopting uppercase placeholder tokens instead. Prompts, in my view, constitute software in the fullest sense — subject to defects and requiring a build step.

5. Operational failure modes

The most instructive failure observed in this system’s operation was unrelated to model quality. Following submission of a large batch (several hundred cards), the majority of issues remained unprocessed, with both agents idle.

The underlying cause was a queue time-to-live constraint internal to Multica. Issue assignment creates a task; a task remaining unclaimed in the queue beyond a fixed threshold (two hours, in the version deployed here) is marked “expired” by a background sweeper. Given the analyst’s configured concurrency limit, a 300-card batch drained more slowly than this threshold, such that the queue’s tail expired silently and the corresponding issues were left in an orphaned state — assigned, but without an active task and without automatic retry. This behavior was identified only by inspecting Multica’s scheduler source following the observed symptom; it is not documented in any user-facing reference.

The Watchdog agent exists solely as a consequence of this platform-level constraint; it addresses a limitation of Multica’s scheduler rather than a defect in my own agents’ behavior. Operating on an hourly schedule, its sole function is to identify orphaned issues — assigned to an agent, in a pre-execution state, whose most recent task terminated with a queue- or platform-level error, and lacking any currently active task — and re-enqueue them. This constitutes a maintenance procedure rather than an intelligent process; however, the underlying failure mode is intrinsic to asynchronous agent queues in general, and to the specific time-to-live constraint enforced by Multica, such that the Watchdog is retained as a permanent component of the system.

Three general observations follow, none of which are novel individually but which I consider collectively underappreciated:

• Sustained throughput must exceed the platform’s queue time-to-live, or work accumulates and silently expires. Both quantities should be known explicitly, not assumed.
• Trigger mechanisms require idempotency and deduplication. Re-execution of a task must not produce a duplicate record, and a recovery process must never re-enqueue a task that remains genuinely in flight.
• A dedicated maintenance process should be planned for from the outset. Any sufficiently long-running agentic system accumulates stuck state, and a process for detecting and repairing it should be regarded as a required component rather than an optional addition.

6. The inference engine

Every agent in the pipeline is served against a self-hosted model rather than a hosted commercial API, using vLLM [3] on a single NVIDIA DGX Spark unit (a GB10 Blackwell system with 128 GB unified memory). The model is Qwen3.6-35B-A3B, a hybrid Mamba-attention mixture-of-experts architecture (approximately 35B total parameters, approximately 3B active per forward pass), quantized using Unsloth’s [4] dynamic-precision NVFP4 scheme — a mixed-precision compressed-tensors quantization retaining attention, output, and a subset of late-layer experts at FP8 while the majority of mixture-of-experts parameters are quantized to NVFP4. This scheme is reported to benchmark meaningfully faster than a uniform NVFP4 quantization at comparable output quality, a property of practical relevance when attempting to meet a 35B-parameter model’s throughput requirements on a single GPU.

vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4 \
  --served-model-name nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --gpu-memory-utilization 0.7 \
  --max-model-len auto \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true}'Code language: Bash (bash)

The engine is deployed via Docker (the official vllm/vllm-openai image), a decision that proved more consequential than anticipated. On this GB10/Blackwell hardware, several of vLLM’s optimized code paths — speculative decoding and certain quantization kernels among them — were observed to silently deadlock the engine (characterized by sustained 100% GPU utilization at an anomalously low power draw, with no tokens emitted) or to produce syntactically well-formed but semantically incorrect output on specific nightly builds. Neither failure mode is attributable to the Qwen architecture or the Unsloth quantization; both arise from the intersection of recently released hardware with recently released inference code. Remediation was unglamorous: pinning to a known-stable image tag rather than tracking the rolling nightly release, disabling speculative decoding, and verifying each configuration change against an active health probe rather than a clean startup log. I note that none of these failure modes are specific to the trading-card application; they represent a general cost of deploying inference workloads on newly released accelerator hardware.

7. Economic characterization

I report measured, rather than estimated, operating characteristics. Throughput: real timestamps recorded by the issue-tracking board indicate that a batch of 323 cards, in progress at the time of writing, completed at a rate of approximately 16 cards per hour, a figure bounded by the Card Analyst’s configured concurrency limit. Token usage: the platform records per-issue token consumption directly; a sample of 43 completed cards drawn from this batch yields a mean of approximately 26,000 fresh input tokens, approximately 150,000 cache-read tokens (a hit rate exceeding 85%), and approximately 3,200 output tokens per card. The high cache-hit rate reflects substantial context sharing between the Card Analyst and Validator — identity information, research tooling, and the comment format are largely invariant across cards. Power: GPU power draw was measured directly via nvidia-smi while both agents were actively processing tasks, yielding a mean of approximately 50 W on the GB10 system-on-chip power rail, against a manufacturer-specified [5] 140 W SoC thermal design power and a 240 W total system power supply rating.

Applying this identical per-card token profile to Claude Sonnet 5’s published pricing [6] yields the comparison in Table 1.

Table 1. Comparative per-unit cost, self-hosted inference versus a hosted commercial API, computed from the measured token profile described above.

I emphasize that the $0.15/kWh figure is a stated assumption rather than a measurement of actual utility billing — power draw was measured at the GPU only, not at the wall outlet or utility meter — and the local-cost figures should accordingly be treated as illustrative rather than exact. Nor do I consider this an equitable comparison in the sense of “the hosted API is categorically worse”: one configuration is a fully managed service requiring no operational burden, with the failure modes described in §5 handled transparently; the other is hardware already owned by the operator, otherwise idle. The comparison does, however, constitute the substantive economic argument underlying this approach. At the token volumes observed here, self-hosting a comparatively small model converts bulk cataloging from a viable-but-metered activity into one sufficiently inexpensive that consumption need not be rationed — a property that permits the fleet to be applied uniformly across an entire inventory rather than reserving hosted-API calls for a difficult subset.

8. Discussion: verification as reward signal

I consider this the most consequential implication of the system described above, and it is precisely the checkable-answer property (§1) that makes it possible. The pipeline produces, as an unavoidable byproduct of ordinary operation, the following artifacts for every processed card:

• a complete transcript of the analyst’s attempted solution to a well-defined task;
• an independently verifiable pass/fail signal from the Validator, grounded in re-fetched external sources;
• and, for a subset requiring adjudication, human-applied labels (identity correct or incorrect; research sourced or hallucinated).

This is structurally identical to a reward signal. A production quality-assurance step that verifies output against ground truth constitutes, in effect, a reward model that required no separate training procedure, and its outputs form a stream of (task, response, verified-reward) tuples. This admits reward-filtered supervised fine-tuning and RLVR-style approaches for continued improvement of a small, computationally inexpensive, open-weight model on this specific task, with frontier models employed only as a bootstrapping teacher and as a fallback for difficult cases. The economic argument follows directly: volume is directed toward a model with substantially lower marginal cost per call, one that improves incrementally with each human correction, rather than incurring hosted-API rates for a task a specialized model in the 4–8B parameter range can perform adequately.

I note explicitly that the described training flywheel represents a direction of ongoing work rather than a completed result. The underlying claim, however, holds independent of the extent to which it is realized: for an agentic workflow whose outputs admit verification, verification and training-data generation are the same artifact.

9. Generalizable principles

Abstracting away the specific application, I summarize the following principles as applicable to agentic systems generally:

1. Minimal task granularity. One task should correspond to one clean transcript, yielding inexpensive retries, isolated failures, and training examples requiring no further preparation.
2. Grounding of every claim in an independently re-fetchable source, with an honestly reported failure to identify preferred over a confidently incorrect assertion.
3. Substitution of deterministic inputs for model judgments wherever a low-cost known value exists (§2 illustrates this with foil status and physical condition).
4. Separation of execution from evaluation, with the evaluating agent restricted to a narrow verify-against-ground-truth mandate, a substance-only failure criterion, and an explicit termination condition.
5. Structured output formats to ensure consistency, without permitting format compliance to constitute an independent failure criterion.
6. Treatment of agent instructions as version-controlled source, subject to review, diffing, and reversion.
7. Explicit operational planning: monitoring throughput against platform-imposed time-to-live constraints, ensuring trigger idempotency, and constructing a dedicated recovery process.
8. Design for verifiability from the outset, such that production quality assurance doubles as a reward signal and the workflow generates its own training data as an operational byproduct.

None of these principles is individually novel. The trading-card domain, however, compelled their consistent application, owing to a task structure that is unforgiving of incorrect output while being generous in signaling when output is incorrect — a combination that, in my assessment, constitutes an effective teacher for agentic system design more broadly.

References

1. Multica: an open-source platform for managing agents. https://multica.ai (source: github.com/multica-ai/multica).
2. Pi: a minimal open-source agent harness. https://pi.dev.
3. vLLM: a high-throughput inference engine for large language models. github.com/vllm-project/vllm.
4. Unsloth: quantization and fine-tuning tooling for open-weight models. https://unsloth.ai.
5. NVIDIA. DGX Spark User Guide — Hardware Overview. docs.nvidia.com/dgx/dgx-spark/hardware.html.
6. Anthropic. Claude models overview and pricing. platform.claude.com/docs/en/about-claude/models/overview.