Prompt Engineering & Structured Output

Precision in a Claude-powered reviewer or classifier comes from explicit, operational criteria, not from softening language. The exam's canonical contrast is 'flag comments only when the claimed behavior contradicts the actual code behavior' (precise, testable) versus 'check that comments are accurate' (vague, leaves the model to invent a bar). The latest models (Opus 4.8) interpret prompts literally and do not silently generalize, which makes specificity even more important: state exactly which categories to report (bugs that cause incorrect behavior, security issues) and which to skip (minor style, naming, local conventions).

A common but ineffective fix is confidence-based filtering. Instructions like 'be conservative' or 'only report high-confidence findings' do not reliably raise precision because the model has no shared definition of the threshold. The official guidance is to 'be concrete about where the bar is rather than using qualitative terms like "important"' — for example, 'report any bug that could cause incorrect behavior, a test failure, or a misleading result; omit pure style or naming preferences.' Categorical, example-anchored criteria outperform a numeric confidence dial.

False positives are not evenly damaging. A single high-false-positive category undermines developer trust in every other category, including the accurate ones — once reviewers learn to ignore one noisy bucket, they start ignoring all output. The practical architectural move is to temporarily disable a high-false-positive category entirely (restoring trust in what remains) while you iterate on its prompt offline, rather than shipping it noisy.

For consistent severity classification, define each severity level with concrete code examples of what qualifies (e.g., a worked 'critical' example, a worked 'minor' example). Anchoring severities to examples produces far more consistent labeling than adjectives like 'serious' or 'important' alone. Pair this with measurement: track dismissal rates per category so you can see which criteria are actually working.

Few-shot (multishot) prompting is the most reliable technique for getting consistently formatted, actionable output when detailed instructions alone still produce variance. Anthropic's guidance: 'Examples are one of the most reliable ways to steer Claude's output format, tone, and structure,' and they 'can dramatically improve accuracy and consistency.' The recommended range is roughly 2-5 examples (the docs suggest 3-5 for best results), wrapped in <example> tags (multiple in an <examples> block) so the model distinguishes demonstrations from instructions, and made diverse enough to cover edge cases so the model doesn't latch onto an unintended pattern.

The highest-leverage use of few-shot is ambiguous-case handling. For a tool-selection decision, a branch-level test-coverage gap, or an extraction from an oddly structured document, a worked example that shows the reasoning — why one action was chosen over a plausible alternative — teaches judgment rather than a lookup table. This is why examples enable generalization to novel patterns instead of merely matching pre-specified ones. You can place a <thinking> block inside an example to demonstrate the reasoning pattern you want; the model will generalize that style.

For output consistency, show the exact target shape in each example (e.g., location, issue, severity, suggested fix). Demonstrating the format is more effective than describing it. For false-positive reduction, include contrastive examples: an acceptable code pattern labeled 'do not flag' next to a genuine issue labeled 'flag' — this draws the boundary while still letting the model generalize.

Few-shot is also the primary tool against extraction hallucination. Examples that show correct handling of informal measurements, inline citations versus bibliographies, methodology sections versus embedded details, and varied document layouts teach the model how to behave on structures it hasn't seen — and crucially, examples that show returning null/empty for a genuinely-absent field prevent the model from fabricating values to satisfy a field. Keep examples relevant (mirror the real use case) and ask Claude to evaluate your set for relevance and diversity if unsure.

The most reliable way to get schema-compliant structured output from Claude is to constrain generation rather than to ask politely for JSON. Two mechanisms do this in 2026: (1) tool use, where you define a tool with an input_schema (JSON Schema) and read structured data from the returned tool_use block; and (2) native Structured Outputs / strict tool use, which add constrained (grammar-based) decoding so the model literally cannot emit tokens that violate the schema. Adding strict: true to a tool definition guarantees the tool input conforms to its input_schema (e.g., passengers becomes 2, never "two"); the JSON Outputs mode (output_config.format with type: json_schema, gated by the beta header structured-outputs-2025-11-13) guarantees the message text is schema-valid JSON. Both eliminate JSON syntax errors — but neither prevents semantic errors: line items can still fail to sum to the stated total, or a value can land in the wrong field. Schema validity is not correctness.

tool_choice controls invocation. auto (the default when tools are present) lets the model return text instead of calling a tool. any forces it to call exactly one of the provided tools but lets it choose which — ideal when you have several extraction schemas and don't yet know the document type. {"type": "tool", "name": "extract_metadata"} forces a specific named tool — use this to guarantee a particular extraction runs before any enrichment step. none disables tools. Important: with any or tool the API prefills the assistant turn, so the model emits no natural-language preamble before the tool_use block. (Forced tool use is also incompatible with extended thinking.)

Schema design choices materially affect quality. Mark fields optional / nullable ("type": ["string", "null"]) when the source may legitimately lack them — required fields pressure the model to fabricate a value to satisfy the schema. Use enums for closed categories, and add an "unclear" enum value for ambiguous cases plus an "other" value paired with a free-text detail field for extensible categorization. Because constrained decoding only enforces structure, put format-normalization rules (date formats, units, currency) in the prompt alongside the schema so inconsistent source formatting is cleaned up before it fills your fields.

Even with schema-guaranteed output, extraction quality requires validation, retry, and feedback loops because constrained decoding fixes syntax, not meaning. The core retry pattern is retry-with-error-feedback: when a result fails a semantic validation check, send a follow-up request that includes the original document, the failed extraction, and the specific validation errors, so the model can self-correct toward a valid answer. Generic 'try again' is far weaker than naming the exact discrepancy.

Retries have a hard limit you must recognize: they help with format mismatches and structural output errors, but they cannot recover information that is simply absent from the source. If a required value lives only in an external document you didn't provide, no number of retries will surface it — the correct response is to mark that field nullable and return null, or to fetch the missing document, not to loop. Knowing when a retry will succeed (format/structure) versus when it will waste calls (missing data) is an explicitly tested skill.

Design self-correcting validation directly into the schema. To catch arithmetic inconsistency, have the model emit both a stated_total (copied from the document) and a calculated_total (summed from line items); your code flags any discrepancy and triggers a targeted retry. For inconsistent source data, add a conflict_detected boolean so the model can surface that two parts of the document disagree rather than silently picking one. This shifts validation from a post-hoc guess to a structured, checkable signal.

Feedback loops also drive long-term precision. Add a detected_pattern field to each finding that records which code construct triggered it (e.g., 'null check after non-null return'). When developers dismiss findings, you can aggregate by detected_pattern to see exactly which constructs generate false positives, then fix the prompt or disable that pattern. The throughline: separate the two error classes — schema syntax errors are eliminated by tool use; semantic errors (values don't sum, wrong field placement, source conflicts) need explicit validation, targeted retries, and instrumentation.

The Message Batches API processes large volumes of Messages requests asynchronously at 50% of standard API prices. Most batches finish in under an hour, but the only guarantee is that results are available when all requests complete or after 24 hours, whichever comes first — there is no latency SLA, and requests not completed within 24 hours expire (and are not billed). This makes batch processing the right choice for non-blocking, latency-tolerant workloads (overnight reports, weekly audits, nightly test generation) and the wrong choice for blocking workflows like pre-merge checks, where you must use the synchronous Messages API.

Each batch request carries a custom_id (1-64 chars, ^[a-zA-Z0-9_-]+$) that you use to correlate responses with requests — essential because results may return in any order, not the submission order. A batch is capped at 100,000 requests or 256 MB. While batches CAN include tool use (and the request params are standard Messages params), the practical constraint for the exam is that batch requests are not interactive: you cannot pause mid-request to execute a client tool and feed the result back within the same request, so multi-turn client-tool loops belong in the synchronous API.

Matching frequency to an SLA is a tested calculation. Because a batch can take up to 24 hours, to honor a 30-hour end-to-end SLA you submit on a window narrower than the slack — e.g., a 4-6 hour submission cadence leaves margin (24h processing + buffer) under 30 hours. Pick a window such that submission interval + 24h worst-case processing stays within the promised SLA.

Failure handling is per-request. Results carry a type of succeeded, errored, canceled, or expired. Resubmit only the failed items, keyed by custom_id, applying appropriate fixes — for instance, chunking a document that exceeded the context window before resubmitting, or simply retrying transient server errors (invalid_request_error requires fixing the request body first). Finally, refine and test your prompt on a small sample synchronously before committing a large batch; maximizing first-pass success rate avoids expensive iterative resubmission across thousands of requests.

Self-review is structurally weak. When the same Claude instance that generated an answer is asked to critique it in the same session, it retains the reasoning context that led to its decisions and is therefore less likely to question them — it tends to rationalize rather than re-examine. Neither a 'now review your own work' instruction nor extended thinking fully overcomes this. The architecturally stronger pattern is an independent review instance: a second Claude call that sees the artifact (the generated code or extraction) but not the generator's reasoning trace. Without the prior context, the reviewer approaches the work fresh and catches subtle issues the author missed. This mirrors Anthropic's broader guidance that for tasks with multiple considerations, separate LLM calls focused on each consideration outperform one call doing everything (the evaluator-optimizer and sectioning patterns).

Large multi-file reviews degrade for a different reason: attention dilution. Asking one pass to review many files at once spreads the model thin, producing shallow and sometimes contradictory findings. The fix is multi-pass review. Run focused per-file passes that look only for local issues within each file (where the model can attend deeply), then run separate integration passes that analyze cross-file concerns — data flow between modules, interface contracts, and consistency across files. Splitting local from cross-cutting analysis keeps each pass tractable and avoids the contradictions that arise when one pass juggles both scopes.

A complementary technique is calibrated verification: run a pass in which the model self-reports a confidence level alongside each finding. You then route reviews by confidence — auto-accept high-confidence findings, send low-confidence ones to human review or a second instance — turning confidence into a routing signal rather than a filter applied inside a single prompt. Combined, these patterns (independent reviewer, per-file plus integration passes, confidence-routed verification) form the multi-instance / multi-pass architecture the exam expects for high-quality review at scale.