Telemetry

Applies to: TDSE Runtime 1.0.0-rc1 with TDSE_ENABLE_TELEMETRY=ON.

Use this chapter when you need runtime observability: enable telemetry, attach it to models, export events, and decide when continuous event collection is a better fit than a short measurement pass.

For RC evaluation, decide three things before you wire telemetry into a host:

• whether the delivered evaluation package actually includes telemetry support
• whether you need continuous observability or a short profiler-driven sizing pass
• whether the exported data is engineering evidence for debugging, or part of a formal qualification record you manage in your own host program

Profiler coverage is intentionally split into the next chapter, Profiler. Use this chapter for always-on or long-running observability. Use the Profiler chapter when the question is measurement methodology, backend comparison, or runtime-plan generation.

Telemetry Versus Profiler

What Telemetry Provides

TDSE telemetry adds observability to runtime step execution without changing numerical behavior. When enabled, the runtime automatically records step-begin events plus completion events for tdse_step_op and tdse_step_commit; it does not instrument every step API.

Use telemetry for long-running or production-like runs where you want a steady record of step activity. Use the Profiler when you want a focused performance investigation, backend comparison, or runtime-plan session.

Typical uses:

• measure per-step latency and throughput in production simulations
• detect performance regressions across SDK upgrades
• feed step-level metrics into existing monitoring infrastructure
• record resource usage alongside simulation progress

Telemetry is compiled out by default and has zero runtime cost when disabled.

Package Availability

Telemetry is package-variant-specific in the RC line. Before wiring the API, confirm that your delivered evaluation package or delivery notes explicitly say telemetry support is enabled (TDSE_ENABLE_TELEMETRY=ON).

If telemetry is not enabled in the delivered package, treat that as a package variant choice, not as a runtime misconfiguration in your integration.

Two-Step Lifecycle

Telemetry follows an explicit telemetry context + per-model attachment pattern. Create one context for each independent telemetry sink you want to own, then pass that handle when attaching models.

Step 1: Create a Telemetry Context

#include <tdse/tdse_telemetry.h>

tdse_telemetry_service_config_t service_cfg;
tdse_telemetry_service_config_init(&service_cfg);
service_cfg.json_output_path = "tdse_telemetry.json";
service_cfg.worker_sleep_ms = 10;

tdse_telemetry_context_t* telemetry = NULL;
tdse_telemetry_context_create(&service_cfg, &telemetry);

tdse_telemetry_service_config_t fields:

Call tdse_telemetry_context_is_initialized(telemetry) to check status before attaching models.

Step 2: Attach Each Model

tdse_telemetry_model_config_t model_cfg;
tdse_telemetry_model_config_init(&model_cfg);
model_cfg.sampling_interval_steps = 1;
model_cfg.ring_capacity = 1024;
tdse_model_telemetry_attach(model, telemetry, &model_cfg);

tdse_telemetry_model_config_t fields:

Set sampling_interval_steps to a larger value (e.g., 10 or 100) to reduce overhead in long-running simulations where per-step detail is not required.

Telemetry is designed for observability, not for zero-overhead measurement. When you are trying to prove peak throughput, strict WCET, or backend selection, use the Profiler first and add telemetry only if you also need a persistent operational trace.

Detach and Destroy

/* per model */
tdse_model_telemetry_detach(model);

/* when no attached model uses the context */
tdse_telemetry_context_destroy(&telemetry);

Detach before destroying the model. Destroy the telemetry context after all models attached to it are detached.

Event Reference

Telemetry Levels

Set per-model via tdse_model_telemetry_set_level():

Event Kinds

Events are typed via tdse_telemetry_event_kind_t. Key event kinds include:

Extended Statistics

Query accumulated statistics at any time:

uint64_t events_queued = 0;
uint64_t events_dropped = 0;
uint32_t ring_capacity = 0;
uint32_t ring_used = 0;
tdse_model_telemetry_get_stats(
    model, &events_queued, &events_dropped, &ring_capacity, &ring_used);

/* events_queued: cumulative successful event enqueues
 * events_dropped: cumulative events rejected because the model ring was full
 * ring_used: current ring occupancy
 */

tdse_telemetry_model_counters_t counters =
    tdse_telemetry_model_counters_init();
tdse_model_telemetry_get_counters(model, &counters);

/* steps_seen counts step-begin notifications before sampling.
 * events_produced counts sampled events offered to the model ring.
 * events_enqueued and events_dropped classify every produced event exactly
 * once: produced == enqueued + dropped (before uint64_t wraparound).
 */

tdse_model_telemetry_get_stats() is the compatibility query for enqueue and ring occupancy data. New integrations should use tdse_model_telemetry_get_counters() when they need an unambiguous production funnel. Export success and failure are context-scoped rather than model-scoped; read the versioned exporter snapshot with tdse_telemetry_get_health_status().

Custom Tags

Attach key-value tags for correlation in multi-model deployments:

const char* keys[] = {"circuit", "run"};
const char* values[] = {"my_netlist", "42"};
tdse_model_telemetry_set_custom_tags(model, 2, keys, values);

Instance IDs

Each attached model receives a unique instance ID for log correlation:

uint64_t id = tdse_model_telemetry_get_instance_id(model);

Exporters

JSON Lines (built-in)

Enabled by the initialized default configuration. When json_output_path is non-null, events are written as JSON lines to that path; each line is a self-contained JSON object. A host that supplies a custom configuration can set the path to NULL to omit this sink.

OpenTelemetry

Export to an OTLP-compatible endpoint:

tdse_telemetry_export_opentelemetry(
    telemetry, "http://127.0.0.1:4318", "tdse-host", "1.0.0");

Security note: the OpenTelemetry exporter only accepts loopback or private targets by default (e.g., http://127.0.0.1:4318). This is intentional to prevent accidental data exposure.

HTTPS export on non-Windows builds requires OpenSSL to be found at configure time.

Prometheus

Export to a Prometheus push gateway:

tdse_telemetry_export_prometheus(telemetry, "http://127.0.0.1:9091", "tdse_host");

Same loopback/private restriction applies.

What Telemetry Does Not Prove

Telemetry can help you preserve evidence, correlate incidents, and compare runs under the same host policy. It does not by itself prove:

• release qualification on a new host platform
• WCET or target-machine timing acceptance for RT/HIL
• correctness of a runtime-plan or backend recommendation
• support for optional accelerators that were not included in the delivered package

Health Status

Check service health with the versioned snapshot, which is serialized with the worker's drain/export path:

tdse_telemetry_health_status_t health =
    tdse_telemetry_health_status_init();
tdse_status_t status =
    tdse_telemetry_get_health_status(telemetry, &health);

The snapshot adds the oldest queued event age, exporter health, the last export and flush statuses, an export error class, and an explicit consistency version. tdse_telemetry_flush_events() drains the events that were pending when the call acquired the drain coordinator and returns the first export failure observed while draining them.

System Metrics

Record or query system-level metrics:

tdse_telemetry_system_metrics_v2_t sys;
tdse_telemetry_system_metrics_init(&sys);
tdse_telemetry_get_system_metrics(telemetry, &sys);

if (sys.total_memory_bytes.valid) {
    use_memory_sample(sys.total_memory_bytes.value);
}
tdse_telemetry_record_memory_usage(model, 512ULL * 1024ULL * 1024ULL, 1024);
tdse_telemetry_record_gpu_usage(model, 256ULL * 1024ULL * 1024ULL, 25, "cuda0");

Each v2 sample includes valid, source, error_class, sample_age_ns, and is_proxy. Numeric values are meaningful only when valid is set. A valid zero is therefore distinct from a failed or unsupported probe.

The same quality contract is available for model-level RSS, CPU, and GPU samples:

tdse_telemetry_model_stats_v2_t model_stats;
tdse_telemetry_model_stats_init(&model_stats);
tdse_model_telemetry_get_extended_stats(model, &model_stats);

if (model_stats.gpu_utilization_percent.valid) {
    use_gpu_sample(model_stats.gpu_utilization_percent.value);
}

tdse_model_telemetry_get_extended_stats() returns that versioned model snapshot; it does not provide a legacy numeric projection.

Performance Alerts

Record custom performance alerts:

tdse_telemetry_record_performance_alert(
    model,
    TDSE_TELEMETRY_ALERT_WARNING,
    "step_latency_exceeded",
    "duration_us=150");

Alert severities:

Backend Switch Events

Hosts that need a backend-switch record can emit one explicitly:

tdse_telemetry_record_backend_switch(
    model, old_backend_id, new_backend_id, "runtime plan update");

Flush

Force-flush pending events:

tdse_telemetry_flush_events(telemetry);

Complete Example

#include <tdse/tdse.h>
#include <tdse/tdse_telemetry.h>

void run_with_telemetry(tdse_model_t* model,
                        tdse_telemetry_context_t* telemetry,
                        size_t nsteps) {
    /* A telemetry context is assumed to be created before this function. */

    tdse_telemetry_model_config_t model_cfg;
    tdse_telemetry_model_config_init(&model_cfg);
    model_cfg.sampling_interval_steps = 1;
    model_cfg.ring_capacity = 2048;
    tdse_model_telemetry_attach(model, telemetry, &model_cfg);

    const char* keys[] = {"scenario"};
    const char* values[] = {"baseline"};
    tdse_model_telemetry_set_custom_tags(model, 1, keys, values);
    tdse_model_telemetry_set_level(model, TDSE_TELEMETRY_LEVEL_STANDARD);

    /* Normal step loop */
    for (size_t n = 0; n < nsteps; ++n) {
        tdse_step_begin(model, n * 0.001, 0.001);
        /* op, hr, ir, solve, commit */
        tdse_step_commit(model, primary, np);
    }

    tdse_model_telemetry_detach(model);
}

Production Deployment Checklist

• Confirm the delivered RC package enables telemetry (TDSE_ENABLE_TELEMETRY=ON)
• Create a tdse_telemetry_context_t for each telemetry sink
• Attach each model before stepping
• Set sampling_interval_steps intentionally (1 for dev, higher for prod)
• Configure exporter endpoints (JSON always active; OTLP/Prom optional)
• Ensure exporter targets are loopback/private or explicitly authorized
• On Linux, OpenSSL is available if HTTPS export is needed
• Detach before model destroy, then destroy the telemetry context
• Verify tdse_telemetry_context_is_initialized() returns true before attaching
• Archive telemetry JSON alongside simulation results for post-hoc analysis