Telemetry lifecycle, event reference, exporters, deployment, and runtime observability.

Applies to: TDSE Runtime 1.0.0-rc1 with TDSE_ENABLE_TELEMETRY=ON.

Use this chapter when you need runtime observability: enable telemetry, attach it to models, export events, and decide when continuous event collection is a better fit than a short measurement pass.

For RC evaluation, decide three things before you wire telemetry into a host:

whether the delivered evaluation package actually includes telemetry support
whether you need continuous observability or a short profiler-driven sizing pass
whether the exported data is engineering evidence for debugging, or part of a formal qualification record you manage in your own host program

Profiler coverage is intentionally split into the next chapter, Profiler. Use this chapter for always-on or long-running observability. Use the Profiler chapter when the question is measurement methodology, backend comparison, or runtime-plan generation.

Telemetry Versus Profiler

Need	Start Here	Why
capture a steady operational trace	this chapter	telemetry emits ongoing runtime events
benchmark one model or compare backend choices	Profiler	profiler is the measurement tool
generate a runtime plan to apply in code	Profiler	runtime plans are profiler outputs
keep debugging evidence from long runs	this chapter	telemetry is built for persistent observability

What Telemetry Provides

TDSE telemetry adds observability to runtime step execution without changing numerical behavior. When enabled, the runtime automatically emits step-level events from tdse_step_begin, tdse_step_op, tdse_step_hr, tdse_step_ir, and tdse_step_commit.

Use telemetry for long-running or production-like runs where you want a steady record of step activity. Use the Profiler when you want a focused performance investigation, backend comparison, or runtime-plan session.

Typical uses:

measure per-step latency and throughput in production simulations
detect performance regressions across SDK upgrades
feed step-level metrics into existing monitoring infrastructure
record resource usage alongside simulation progress

Telemetry is compiled out by default and has zero runtime cost when disabled.

Package Availability

Telemetry is package-variant-specific in the RC line. Before wiring the API, confirm that your delivered evaluation package or delivery notes explicitly say telemetry support is enabled (TDSE_ENABLE_TELEMETRY=ON).

If telemetry is not enabled in the delivered package, treat that as a package variant choice, not as a runtime misconfiguration in your integration.

Two-Step Lifecycle

Telemetry follows an explicit telemetry context + per-model attachment pattern. Create one context for each independent telemetry sink you want to own, then pass that handle when attaching models.

Step 1: Create a Telemetry Context

#include <tdse/tdse_telemetry.h>

tdse_telemetry_service_config_t service_cfg;
tdse_telemetry_service_config_init(&service_cfg);
service_cfg.json_output_path = "tdse_telemetry.json";
service_cfg.worker_sleep_ms = 10;

tdse_telemetry_context_t* telemetry = NULL;
tdse_telemetry_context_create(&service_cfg, &telemetry);

tdse_telemetry_service_config_t fields:

Field	Meaning	Recommended Default
`json_output_path`	file path for JSON lines export	`"tdse_telemetry.json"`
`worker_sleep_ms`	background worker poll interval in milliseconds	`10`

Call tdse_telemetry_context_is_initialized(telemetry) to check status before attaching models.

Step 2: Attach Each Model

tdse_telemetry_model_config_t model_cfg;
tdse_telemetry_model_config_init(&model_cfg);
model_cfg.sampling_interval_steps = 1;
model_cfg.ring_capacity = 1024;
tdse_model_telemetry_attach(model, telemetry, &model_cfg);

tdse_telemetry_model_config_t fields:

Field	Meaning	Recommended Default
`sampling_interval_steps`	record every N-th step	`1` (every step)
`ring_capacity`	number of events retained in the ring buffer	`1024`

Set sampling_interval_steps to a larger value (e.g., 10 or 100) to reduce overhead in long-running simulations where per-step detail is not required.

Telemetry is designed for observability, not for zero-overhead measurement. When you are trying to prove peak throughput, strict WCET, or backend selection, use the Profiler first and add telemetry only if you also need a persistent operational trace.

Detach and Destroy

/* per model */
tdse_model_telemetry_detach(model);

/* when no attached model uses the context */
tdse_telemetry_context_destroy_v2(&telemetry);

Detach before destroying the model. Destroy the telemetry context after all models attached to it are detached.

Event Reference

Telemetry Levels

Set per-model via tdse_model_telemetry_set_level():

Level	Behavior
disabled	no telemetry collection
light	lifecycle and light step-level monitoring
standard	step-level timing and standard runtime events

Event Kinds

Events are typed via tdse_telemetry_event_kind_t. Key kinds emitted automatically by the runtime:

Kind	When Emitted	Payload
step begin	`tdse_step_begin()`	`t`, `dt`
step commit	`tdse_step_commit_v2()`	`committed_steps`
model lifecycle	create / destroy	handle metadata

Extended Statistics

Query accumulated statistics at any time:

uint64_t events_queued = 0;
uint64_t events_dropped = 0;
uint32_t ring_capacity = 0;
uint32_t ring_used = 0;
tdse_model_telemetry_get_stats(
    model, &events_queued, &events_dropped, &ring_capacity, &ring_used);

/* events_queued: cumulative successful event enqueues
 * events_dropped: cumulative events rejected because the model ring was full
 * ring_used: current ring occupancy
 */

tdse_telemetry_model_counters_t counters =
    tdse_telemetry_model_counters_init();
tdse_model_telemetry_get_counters(model, &counters);

/* steps_seen counts step-begin notifications before sampling.
 * events_produced counts sampled events offered to the model ring.
 * events_enqueued and events_dropped classify every produced event exactly
 * once: produced == enqueued + dropped (before uint64_t wraparound).
 */

uint64_t memory_usage_bytes = 0;
uint32_t cpu_utilization_percent = 0;
uint32_t gpu_utilization_percent = 0;
uint64_t extended_events_dropped = 0;
uint32_t alerts_triggered = 0;
tdse_model_telemetry_get_extended_stats(model,
    &memory_usage_bytes,
    &cpu_utilization_percent,
    &gpu_utilization_percent,
    &extended_events_dropped,
    &alerts_triggered);

tdse_model_telemetry_get_stats() is the compatibility query for enqueue and ring occupancy data. New integrations should use tdse_model_telemetry_get_counters() when they need an unambiguous production funnel. Export success and failure are context-scoped rather than model-scoped; read the versioned exporter snapshot with tdse_telemetry_get_health_status_v2().

Custom Tags

Attach key-value tags for correlation in multi-model deployments:

const char* keys[] = {"circuit", "run"};
const char* values[] = {"my_netlist", "42"};
tdse_model_telemetry_set_custom_tags(model, 2, keys, values);

Instance IDs

Each attached model receives a unique instance ID for log correlation:

uint64_t id = tdse_model_telemetry_get_instance_id(model);

Exporters

JSON Lines (built-in)

Always active when the service is initialized. Events are written as JSON lines to json_output_path. Each line is a self-contained JSON object.

OpenTelemetry

Export to an OTLP-compatible endpoint:

tdse_telemetry_export_opentelemetry(
    telemetry, "http://127.0.0.1:4318", "tdse-host", "1.0.0");

Security note: the OpenTelemetry exporter only accepts loopback or private targets by default (e.g., http://127.0.0.1:4318). This is intentional to prevent accidental data exposure.

HTTPS export on non-Windows builds requires OpenSSL to be found at configure time.

Prometheus

Export to a Prometheus push gateway:

tdse_telemetry_export_prometheus(telemetry, "http://127.0.0.1:9091", "tdse_host");

Same loopback/private restriction applies.

What Telemetry Does Not Prove

Telemetry can help you preserve evidence, correlate incidents, and compare runs under the same host policy. It does not by itself prove:

release qualification on a new host platform
WCET or target-machine timing acceptance for RT/HIL
correctness of a runtime-plan or backend recommendation
support for optional accelerators that were not included in the delivered package

Health Status

Check service health at any time:

uint32_t attached_models = 0;
uint64_t total_events = 0;
uint64_t total_dropped = 0;
uint32_t worker_queue_size = 0;
tdse_telemetry_get_health_status(
    telemetry, &attached_models, &total_events, &total_dropped, &worker_queue_size);

total_events is the number successfully exported. total_dropped combines model-ring rejections and events lost to exporter failure, while worker_queue_size is the current total occupancy of attached model rings.

New integrations should use the versioned snapshot, which is serialized with the worker's drain/export path:

tdse_telemetry_health_status_t health =
    tdse_telemetry_health_status_init();
tdse_status_t status =
    tdse_telemetry_get_health_status_v2(telemetry, &health);

The snapshot adds the oldest queued event age, exporter health, the last export and flush statuses, an export error class, and an explicit consistency version. tdse_telemetry_flush_events() drains the events that were pending when the call acquired the drain coordinator and returns the first export failure observed while draining them.

System Metrics

Record or query system-level metrics:

tdse_telemetry_system_metrics_v2_t sys;
tdse_telemetry_system_metrics_v2_init(&sys);
tdse_telemetry_get_system_metrics_v2(telemetry, &sys);

if (sys.total_memory_bytes.valid) {
    use_memory_sample(sys.total_memory_bytes.value);
}
tdse_telemetry_record_memory_usage(model, 512ULL * 1024ULL * 1024ULL, 1024);
tdse_telemetry_record_gpu_usage(model, 256ULL * 1024ULL * 1024ULL, 25, "cuda0");

Each v2 sample includes valid, source, error_class, sample_age_ns, and is_proxy. Numeric values are meaningful only when valid is set. A valid zero is therefore distinct from a failed or unsupported probe. The legacy tdse_telemetry_get_system_metrics() API remains available, but projects an invalid sample to zero and cannot preserve that distinction.

The same quality contract is available for model-level RSS, CPU, and GPU samples:

tdse_telemetry_model_stats_v2_t model_stats;
tdse_telemetry_model_stats_v2_init(&model_stats);
tdse_model_telemetry_get_extended_stats_v2(model, &model_stats);

if (model_stats.gpu_utilization_percent.valid) {
    use_gpu_sample(model_stats.gpu_utilization_percent.value);
}

tdse_model_telemetry_get_extended_stats() remains the legacy numeric projection and maps invalid or unavailable samples to zero.

Performance Alerts

Record custom performance alerts:

tdse_telemetry_record_performance_alert(
    model,
    TDSE_TELEMETRY_ALERT_WARNING,
    "step_latency_exceeded",
    "duration_us=150");

Alert severities:

Severity	Use When
info	informational note
warning	performance degraded but tolerable
error	performance issue requiring attention
critical	simulation may be invalid

Backend Switch Events

The runtime automatically records backend switch events. You can also record custom events:

tdse_telemetry_record_backend_switch(
    model, old_backend_id, new_backend_id, "runtime plan update");

Flush

Force-flush pending events:

tdse_telemetry_flush_events(telemetry);

Complete Example

#include <tdse/tdse.h>
#include <tdse/tdse_telemetry.h>

void run_with_telemetry(tdse_model_t* model,
                        tdse_telemetry_context_t* telemetry,
                        size_t nsteps) {
    /* A telemetry context is assumed to be created before this function. */

    tdse_telemetry_model_config_t model_cfg;
    tdse_telemetry_model_config_init(&model_cfg);
    model_cfg.sampling_interval_steps = 1;
    model_cfg.ring_capacity = 2048;
    tdse_model_telemetry_attach(model, telemetry, &model_cfg);

    const char* keys[] = {"scenario"};
    const char* values[] = {"baseline"};
    tdse_model_telemetry_set_custom_tags(model, 1, keys, values);
    tdse_model_telemetry_set_level(model, TDSE_TELEMETRY_LEVEL_STANDARD);

    /* Normal step loop */
    for (size_t n = 0; n < nsteps; ++n) {
        tdse_step_begin(model, n * 0.001, 0.001);
        /* op, hr, ir, solve, commit */
        tdse_step_commit_v2(model, primary, np);
    }

    tdse_model_telemetry_detach(model);
}

Telemetry