Coverage controls

import os
from openai import OpenAI
from solwyn import Solwyn, UntrackedSpendSurfaceError

client = Solwyn(
    OpenAI(),
    api_key=os.environ["SOLWYN_API_KEY"],
    on_unmetered="raise",                      # refuse anything Solwyn does not meter
    acknowledge_untracked={"models.list"},     # ...except this exact leaf
)

client.models.list()                           # acknowledged: passes through silently

try:
    client.moderations.create(model="omni-moderation-latest", input="...")
except UntrackedSpendSurfaceError as e:
    print(e.surface)   # "moderations.create"
    print(e.token)     # "moderations.create" -- the exact token that would acknowledge it

SDK v0.6.0+. Solwyn classifies the entire reachable public surface of every provider client it wraps — not just the handful of billable surfaces it meters. A metered surface gets a budget check and a cost event. Everything else is either a structural attribute that is always safe, a namespace, a surface refused outright (blocked or unsupported), a capability Solwyn has reviewed and does not meter yet, or something new the SDK has never seen. on_unmetered decides what happens for the last two, and coverage(client) lets you pin the whole classification so a provider-SDK upgrade cannot quietly open a hole in your budget.

Before 0.6.0 a small allow-list of billable surfaces warned once and everything else passed through in silence. Now every untracked or unknown leaf warns once by default. If your code touched files, batches, fine_tuning, or models.list without a warning yesterday, that is the change you are seeing — see Silencing a known surface.

The three postures

Set it on the client (on_unmetered=) or with SOLWYN_ON_UNMETERED. An acknowledged token is checked first and short-circuits all three postures.

The warning reads:

Provider 'openai' client shape 'openai_sdk' exposes untracked surface 'models.list' (scope: operation); no budget check and no cost event will be emitted. Tracking for this surface is coming.

It is emitted by the solwyn._base logger. The latch is process-global and shared by every client, keyed by provider, client shape, sync/async mode, and the dotted path, so a sync and an async client warn separately for the same leaf. After 512 distinct keys the SDK logs one further notice and stops counting new surfaces individually.

What Solwyn knows about each surface

Every public attribute path on a wrapped client resolves to one of these kinds:

Two refinements matter in practice:

• Guarded namespaces. Accessing client.audio returns a guard, not the raw resource, and client.audio.speech.create is resolved as its own leaf when you reach it. Access to a parent never grants its descendants, and the with / async with protocol is not forwarded on a guard in any posture.
• Capability scope. Each untracked rule carries a scope: operation for an ordinary call, and client, resource, raw_response, or arbitrary_endpoint for escapes that hand back something broader — with_options, copy, get/post, audio.translations, the whole with_raw_response / with_streaming_response family. The scope appears in the warning and on the exception.

Private attributes (any name starting with _ that is not _solwyn_) belong to the provider client and bypass the guard entirely.

UntrackedSpendSurfaceError

Raised under on_unmetered="raise", before provider I/O — at attribute-access time for a guarded read, or as the first statement of an intercepted method. It subclasses both SolwynError and AttributeError, so hasattr(client, "post") and getattr(client, "post", None) keep their feature-probe semantics instead of exploding.

The message names the surface, the provider, the scope, and the fix: Solwyn refused untracked surface 'moderations.create' for provider openai (scope: operation); acknowledge exact token 'moderations.create' or choose on_unmetered='warn'/'allow'. Every attribute is a structural label; none carries request, prompt, response, or credential content.

Acknowledging surfaces

acknowledge_untracked is a collection of exact terminal capability tokens — or SOLWYN_ACKNOWLEDGE_UNTRACKED, comma-delimited. An acknowledged token suppresses the warning, the strict refusal, and the advisory report for that one leaf:

client = Solwyn(
    OpenAI(),
    api_key=os.environ["SOLWYN_API_KEY"],
    on_unmetered="raise",
    acknowledge_untracked={"models.list", "responses.retrieve", "audio.speech.create:gpt-4o-mini-tts"},
)

Tokens are validated at construction against the actual client, so a typo or a token that names the wrong kind of thing fails before any call. Each rejection is ConfigurationError with field="acknowledge_untracked":

An acknowledgment grants exactly its leaf. Acknowledging responses.retrieve leaves responses.delete refused, and the responses namespace stays guarded. A token at raw_response scope, acknowledged at exactly its own path, returns the provider's raw object.

One token is conditional: audio.speech.create:gpt-4o-mini-tts acknowledges the token-billed TTS model that Solwyn deliberately does not price. Acknowledging plain audio.speech.create does not cover it, and the token is matched as a whole string — nothing is parsed at runtime.

Pinning coverage in CI

solwyn.coverage(client) builds a CoverageReport from structural client metadata only — no network I/O, no provider operations, and it never reads request, prompt, response, or credential content. Pin it in a test so a provider-SDK release that adds a spend surface, or a Solwyn release that reclassifies one, fails your suite instead of your budget:

from openai import OpenAI
from solwyn import CoverageFingerprint, CoverageMismatchError, Solwyn, coverage

client = Solwyn(OpenAI(api_key="test"), api_key="sk_proj_" + "0" * 64)

# Run once to capture the digests, then paste them into the test.
print(coverage(client).fingerprint())

EXPECTED = CoverageFingerprint(
    guarded_namespaces="sha256:...",
    tracked="sha256:...",
    untracked="sha256:...",
    unknown="sha256:...",
    scoped_escapes="sha256:...",
    blocked="sha256:...",
    unsupported="sha256:...",
    conditional="sha256:...",
    safe="sha256:...",
)

def test_openai_coverage_is_pinned():
    coverage(client).expect(EXPECTED)   # raises CoverageMismatchError on drift

CoverageReport carries provider, dialect, client_shape, posture, the configured provider_chain (one CoverageRuntime per failover hop), the sorted acknowledgments, and entries — one CoverageEntry per effective rule with its rule_id, surface, token, kind, policy_action, dispatch_action, usage_basis, source, capability_scope, condition, reason, and the expected versus observed shape. For metered chat surfaces, usage_basis is aggregated across every runtime in the chain and reports the weakest guarantee — the one that still holds after a failover hop.

report.fingerprint() returns a CoverageFingerprint: nine sha256: digests, one per audit category, each covering every field of every entry in that category. report.expect(...) accepts either a fingerprint (category-level diff) or a full CoverageExpectation (rule-level diff, listing each added, removed, or changed rule_id). On any difference it raises CoverageMismatchError, whose differences tuple names each category that moved. An empty category always hashes to the same digest, so a fingerprint captured on one machine is valid on another.

Fingerprints are structural and stable at the tested breakpoint versions below; an intermediate release that adds an unclassified surface resolves it as unknown, moves that digest, and fails the pin by design. Pin one per provider client shape you use — Azure OpenAI and native OpenAI, for instance, are separate pins. If inspection itself cannot observe part of the client graph, SurfaceInspectionError (a RuntimeError) reports the path and stage that failed.

Advisory reports

Solwyn Cloud cannot meter what it does not know about. By default the SDK sends a small structural report for every unacknowledged surface it observed under warn or allow, so the Providers tab can show "Surfaces Solwyn does not meter yet" for your project and Solwyn can prioritize tracking.

A report carries exactly twelve fields — provider, client_shape, mode (sync/async), surface (the dotted path), rule_kind, capability_scope, posture, occurrences, first_seen_at, last_seen_at, sdk_instance_id, and a fresh report_id. No model names, request arguments, prompts, or responses. Nothing about a raise refusal or an acknowledged surface is ever sent.

Delivery never touches the provider call: the calling thread does in-memory bookkeeping and a wake-up, and a dedicated background worker posts to /api/v1/untracked-surfaces at most once every fifteen minutes per surface, starting immediately on first sight. Reporting adds no budget check and no cost event. Failures are silent — no retry queue, no log — and the cadence advances regardless, so a broken path costs one attempt per surface per fifteen minutes. The path requires a writable project key; a read-only key's 403 is swallowed on this channel. Paths the wire cannot carry (over 128 characters, deeper than eight segments, or containing a non-ASCII segment) are counted and warned about locally and never sent.

Counts are bounded overcounts: a report whose acknowledgment is lost is resent with the same delta under a new id and summed again server-side. They are never billing truth and never undercount.

Turn it off with report_untracked_surfaces=False or SOLWYN_REPORT_UNTRACKED_SURFACES=false. Opting out changes nothing locally — on_unmetered and the warn-once latch behave exactly the same.

Strict mode is a cooperative guard, not a sandbox

on_unmetered="raise" catches the paths that flow through the wrapper. It does not, and cannot, intercept:

• the raw provider client you keep a reference to;
• private wrapper state (_solwyn_*) or private provider attributes;
• a scoped raw escape you explicitly acknowledged (with_options, copy, with_raw_response, get/post);
• native behavior on objects a call returns — pages, jobs, streams, and operations are not re-guarded.

Keep the wrapped client as the only handle your application code holds, and strict mode covers everything that matters.

Tested provider-SDK versions

Classification is exercised against a floor, named structural breakpoints, and the latest release of each provider SDK. Versions between breakpoints may expose surfaces Solwyn has not classified; those resolve as unknown and follow on_unmetered.

Related

• Surface coverage — which surfaces are priced, which are recorded unpriced, and the vocabulary above applied per provider
• Testing budget enforcement — plane.untracked_reports records the advisory reports the SDK sent
• Exceptions — UntrackedSpendSurfaceError, CoverageMismatchError, and SurfaceInspectionError
• SolwynConfig — on_unmetered, acknowledge_untracked, report_untracked_surfaces
• Privacy — the advisory report as an outbound payload