Backend selection, runtime plans, CUDA, precision, thread control, and benchmark guidance.

Audience: Integration engineers choosing execution backends for production deployments, and anyone tuning GPU or CPU performance for large models.

Use this chapter when you are choosing an execution backend for deployment or trying to improve throughput on existing hardware. It shows how to discover available backends, pick the right one for a model, and tune the settings that matter most.

For best results, read this chapter after Profiler. The Profiler tells you how to measure your real workload; this chapter tells you how to interpret the measurements, pick a backend policy, and separate sizing guidance from stronger deployment evidence.

This chapter owns backend concepts, discovery, and selection semantics. The Runtime Backend Support Matrix is the only authority for lifecycle and qualification status. Detailed target measurement procedures and knobs belong to Runtime Performance Tuning.

Backend Overview

TDSE runtime backends control how convolution and linear algebra operations are executed. The backend is selected per model before the first step.

Discovering Available Backends

uint32_t count = tdse_backend_registry_count();
for (uint32_t i = 0; i < count; ++i) {
    tdse_backend_capability_t cap;
    tdse_backend_registry_get(i, &cap);
    printf("backend %u: %s (np_max=%zu, cuda=%d)\n",
           i, tdse_backend_id_name(cap.id), (size_t)cap.max_np, cap.has_cuda);
}

Product Backend Identifier Reference

Backend ID	Runtime token	Description	Best For
`TDSE_BACKEND_AUTO`	`auto`	Product default. Uses the signed CPU route policy and fails closed to conservative CPU routing. Unsigned preview CUDA routes are not selected.	Production default
`TDSE_BACKEND_CPU_PACK_GEMV`	`cpu_pack_gemv`	CPU meta backend. The evidence-free AUTO fallback uses the generic fused-ring route; qualified manifests may enable narrower sub-routes.	Most CPU deployments
`TDSE_BACKEND_CPU_BLAS`	`cpu_blas`	Functionally stable BLAS GEMV path. Direct AUTO thresholds require exact host/compiler/provider evidence.	Explicit measured deployments until qualification evidence is attached
`TDSE_BACKEND_GPU_DIRECT_HISTORY`	`gpu_direct_history`	CUDA direct dense history kernel as a first-class backend.	TDSE GPU Add-on provider route for `Np <= 32`
`TDSE_BACKEND_GPU_BLAS_GRAPH`	`gpu_blas_graph`	CUDA graph replay over device-resident pack/mirror + cuBLAS GEMV.	TDSE GPU Add-on provider route for larger `Np`
`TDSE_BACKEND_GPU_PACK_BLAS_GRAPH`	`gpu_pack_blas_graph`	Legacy CUDA graph token retained for compatibility.	Explicit diagnostics only
`TDSE_BACKEND_GPU_FUSED_KERNEL_STREAM`	`gpu_fused_kernel_stream`	CUDA fused history kernel for shape-sensitive small-state cases.	Explicit diagnostics only
`TDSE_BACKEND_FPGA_XRT`	`fpga_xrt`	Fail-closed FPGA XRT route.	Certified FPGA deployments
`TDSE_BACKEND_FPGA_AWS_HDK`	`fpga_aws_hdk`	Fail-closed AWS F2 HDK route.	Certified AWS FPGA deployments
`TDSE_BACKEND_PARTITIONED_HETEROGENEOUS`	`partitioned_heterogeneous`	Diagnostic result identity when one partitioned operation spans multiple concrete backend families; it cannot be selected directly.	Observability and diagnostics

Not all backends are available in every build. Use the registry API to check what is compiled in.

Backend token parsing is case-insensitive for ASCII letters, ignores whitespace, and treats - or . as _. The product aliases cpu, cpu_gemv, pack_gemv, and cpu_pack resolve to cpu_pack_gemv; blas and cblas resolve to cpu_blas; batch and threaded_cpu resolve to cpu_batch. Use tdse_backend_id_from_name_ex() when accepting user input: it returns the canonical backend id plus matched, used_alias, deprecated, canonical_name, matched_name, and a stable parse reason. Unknown names are not fatal and map to TDSE_BACKEND_AUTO with matched=0, preserving the legacy parser behavior while making typos auditable.

The public registry is intentionally small. cpu_ref, cpu_blocked, cpu_batch, standalone cpu_smallkernel, standalone cpu_simd_avx2 / cpu_simd_avx512, gpu_pack_kernel_stream, and stub backends are diagnostic-only. They remain in the ABI so old tools and explicit debugging workflows do not break, but they are hidden from product discovery, excluded from default benchmark/gate matrices, and must not be reintroduced into AUTO without a new diagnostic-to-product promotion record.

Setting the Backend

tdse_backend_id_t id = tdse_backend_id_from_name("cpu_blas");
tdse_status_t st = tdse_backend_set(model, id);

Important: tdse_backend_set() must be called before the first successful tdse_step_begin(). After the first step begins, configuration is frozen and tdse_backend_set() returns TDSE_STATUS_UNSUPPORTED.

Query the active backend at any time:

tdse_backend_id_t active = tdse_backend_get_active(model);

Delayed-History Representations

For models with delayed history (nh > 1), tdse_model_create() allocates only the canonical FP64 history ring. Derived representations are allocated when the selected backend and precision first require them:

Representation	Allocated For
FP64 ring	Canonical committed history for every delayed-history model
FP32 ring	FP32 CPU history routes and FPGA/XRT prepare
FP64 linear	FP64 BLAS / packed GEMV history routes
FP32 linear	FP32 BLAS / packed GEMV history routes

This keeps the default FP64 CPU path at one history copy and avoids extra per-step writes. The runtime backfills any newly allocated representation from the committed FP64 ring before the backend reads it. CUDA backends seed their device history from the FP64 ring and do not require the host FP32 ring.

tdse_model_state_info_t.history_representation_flags is a legacy delayed-history subset (bit0 FP64 ring, bit1 FP32 ring, bit2 FP64 linear, bit3 FP32 linear). New integrations use the compact tdse_model_diagnostics_summary_t plus versioned detail payloads:

the summary reports numeric_storage_policy, history_compute_type, ir_compute_type, op_compute_type, and available_payload_mask;
TDSE_MODEL_DIAGNOSTICS_PAYLOAD_NUMERIC_V1 reports allocated representations, conversion counts, and retained accepted-history state;
TDSE_MODEL_DIAGNOSTICS_PAYLOAD_MEMORY_V1 reports the runtime budget, owner/residency accounting, and CPU/CUDA/FPGA resident bytes;
TDSE_MODEL_DIAGNOSTICS_PAYLOAD_CPU_EXECUTION_V1 reports CPU cache churn, BLAS ring-pack versus linear-mirror decisions, batch scheduling, ISA, and tracked environment overrides;
TDSE_MODEL_DIAGNOSTICS_PAYLOAD_CPU_ROUTE_V1 identifies the policy, feature, environment, and semantic signatures behind the latest CPU route;
TDSE_MODEL_DIAGNOSTICS_PAYLOAD_ACCELERATOR_V1 reports H2D/D2H totals, device residency, zero-copy state, and device-resident history packing;
CPU precision qualification and partition-runtime detail have their own CPU_PRECISION_V1 and PARTITION_V1 payloads.

Use tdse_model_diagnostics_payload(...) for one initialized payload. Use tdse_model_diagnostics_query_many(...) when a summary and several payloads must be projected from the same internal snapshot. Backend selection remains controlled by the backend APIs above.

Runtime Plans

For repeatable deployments, prefer tdse_backend_apply_plan() over manual backend selection. A runtime plan is a JSON document that captures backend selection and optional per-scenario overrides in one place.

Applying a Plan

tdse_backend_apply_plan(model, plan_json);

The plan JSON structure:

{
  "default": {
    "backend": "cpu_pack_gemv",
    "route": "generic"
  },
  "scenarios": [
    {
      "name": "np32_nh2048_fp64",
      "signature": {
        "np": 32,
        "nh": 2048
      },
      "backend": "cpu_pack_gemv",
      "route": "blas"
    }
  ]
}

Profiler-style reports can wrap the same runtime plan:

{
  "schema_version": "tdse_profile_report_v1",
  "runtime_cpu_route_plan": {
    "schema": 1,
    "cpu_feature": "x64-avx2",
    "blas_vendor": "unknown",
    "dtype_filter": 64,
    "entries": [
      {
        "name": "np32_nh2048_fp64",
        "signature": {
          "np": 32,
          "nh": 2048
        },
        "backend": "cpu_pack_gemv",
        "route": "blas",
        "dtype_bits": 64,
        "hist_op_kind": "cpu_blas_gemv_fp64"
      }
    ]
  },
  "runtime_plan": {
    "default": {
      "backend": "cpu_pack_gemv",
      "route": "generic"
    },
    "scenarios": [
      {
        "name": "np32_nh2048_fp64",
        "signature": {
          "np": 32,
          "nh": 2048
        },
        "backend": "cpu_pack_gemv",
        "route": "blas"
      }
    ]
  }
}

The default section applies when no scenario matches. Runtime scenario matching currently uses signature.np and signature.nh. For CPU cpu_pack_gemv plans, route may be smallnp, generic, blas, or pack_ref.

Plans are typically generated by the TDSE Profiler or by the CPU route matrix workflow. See Profiler for details.

CUDA Configuration

When the CUDA backend is available, configure per-model options:

tdse_cuda_backend_config_t cuda_cfg;
tdse_cuda_backend_get_config(model, &cuda_cfg);
cuda_cfg.pipeline_mode = TDSE_CUDA_PIPELINE_DEVICE_ONLY;
tdse_cuda_backend_set_config(model, &cuda_cfg);

Pipeline Modes

Mode	Description	When to Use
sync	synchronous host-device transfer	debugging, small models
async	overlapping compute and transfer	production, large models

GPU Memory Management

If GPU allocation fails, tdse_model_create() or step APIs return TDSE_STATUS_OUT_OF_MEMORY. Monitor GPU memory with nvidia-smi.

For multi-model GPU deployment, resource sharing, and memory planning, see the multi-model deployment patterns in Multi-Model Deployment.

GPU Recommendations

Prefer GPU for models with np > 10 and nh > 100
Small models may not amortize the CPU-GPU transfer overhead
Limit concurrent GPU models if memory is constrained
GPU runtime evidence should be collected with FP32 as the primary path: tdse_runtime_backend_bench and the local GPU decision matrix default to FP32 compute plus FP32 canonical storage. Use the matrix's small FP64 confirmation slice, or an explicit --dtype fp64 run, as the accuracy and regression companion rather than the main throughput signal.
For reviewable local GPU evidence, use the decision matrix engineering preset rather than smoke settings. It covers Np=[1,4,8,16,32,64,128,256] and Nh=[256,512,1024,2048] with 128 measured steps per sample, 5 warmup samples, and 9 timing samples. The intermediate Np=4/16 and Nh=512 cases are intentionally included because they often determine the CPU/GPU crossover boundary.

Compute Precision

Control the floating-point precision used for delayed-history convolution:

tdse_compute_precision_set(model, TDSE_COMPUTE_PRECISION_FP32);

Precision	Description	Impact
`FP64`	double precision (default)	highest accuracy, moderate performance
`FP32`	single precision	faster, reduced accuracy in history term

This setting is per model. It is not a process-global precision switch and it does not change canonical storage. The compute profile applies to history, independent-response (step_ir), and instantaneous-operator (step_op) operations while the public API boundary remains FP64. Use tdse_model_diagnostics_summary() to confirm the active history_compute_type; request TDSE_MODEL_DIAGNOSTICS_PAYLOAD_NUMERIC_V1 for the currently allocated typed representations.

When to use FP32: large nh where the history term dominates step cost and the application tolerates reduced mantissa precision in the history accumulation.

Historical OpenBLAS measurements for the primary CPU application range (np=1..32, nh=512/1024/2048) can guide an explicit FP32 experiment, but they are not a portable AUTO qualification:

np=1/2: FP32 normally stays on the tiny-state CPU path and is not materially faster than FP64.
np=4/8/16/32 with nh>=512: FP32 can be faster when application-level error checks pass. On AWS Intel OpenBLAS validation, FP32 AUTO was about 1.46x to 3.52x faster than FP64 AUTO for these shapes, with maximum history-output relative L2 error below 1e-6 in the benchmark matrix.
np=64+ remains a separate qualification region. Do not extrapolate the np=1..32 application matrix without measuring the target host and pack.

Large FP32 CPU histories use the conservative generic PACK route unless a signed evidence manifest exactly matches the current commit, CPU stepping, compiler, BLAS provider/version/link mode/effective threads, and OpenMP runtime. An explicit or qualified BLAS path reports the diagnostic hist_op_kind value cpu_blas_sgemv_fp32 means the history path is using single-precision CBLAS GEMV and then converting the result back to the runtime output type. Treat FP32 as a performance/accuracy policy and validate it against your application tolerances before deployment.

TDSE_CPU_FP32_REL_L2_BUDGET=<positive float> sets the relative L2 budget used when an FP32 decision is created. The default is 5e-3, which covers the conservative any-order FP32 reduction bound for qualified production shapes. Values below the validated floor or malformed values make tdse_compute_precision_set(...FP32) fail with TDSE_STATUS_INVALID_ARG without changing the model. FP32 canonical storage creation follows the same fail-closed rule. A valid decision is bound to the validated Runtime Pack content hash only after two checks pass: the coefficient-only H/IR FP64-to-FP32 quantization error and a deterministic response suite covering impulse, step, alternating-sign, seeded-random, multiple ring-head phases, long-run accumulation, and IR time steps. The suite uses a conservative IEEE-754 bound that covers scalar, SIMD, and BLAS reduction orders, while matching the runtime's double-precision IR weights. It records relative L2 and maximum absolute response error bounds. Nonfinite values and packs over budget are rejected. cpu_fp32_coefficient_quantization_rel_l2 names the coefficient-only measurement explicitly; cpu_fp32_pack_quantization_rel_l2 remains its ABI compatibility alias. Response results are exposed as cpu_fp32_response_rel_l2, cpu_fp32_response_max_abs, and cpu_fp32_response_validation_valid. The stable cpu_fp32_validation_evidence_id begins with cpu-fp32-response-v3:. The full binding includes the pack and route-policy SHA-256 values, suite and reduction-contract identifiers, CPU/ISA/BLAS context fingerprint, and budget. Diagnostics report the decision captured by the model, so changing the environment later does not rewrite the execution evidence. Capture cpu_fp32_policy_valid, cpu_fp32_validation_required, cpu_fp32_validation_evidence_id, and cpu_fp32_validation_report in release evidence.

When to stay with FP64: stiff systems, high-frequency dynamics, or when bit-exact reproducibility is required across hardware.

Query current setting:

tdse_compute_precision_t prec = tdse_compute_precision_get(model);

Thread Control

Override the number of CPU threads used internally by the runtime:

tdse_local_threads_set(model, 4);

By default, AUTO uses the product CPU route policy. tdse_local_threads_set() is an execution override; it does not force AUTO to select cpu_batch. CPU_BATCH is currently preview and explicit-only; AUTO cannot select it until commit-bound qualification evidence is attached. Explicit cpu_batch requests bind the segment/tap/row-tile local-thread route. Its stable algorithm token is cpu_batch_tap_segment_row_tile. If only one worker is available, the same route runs with scheduler policy serial_row_tiles instead of changing active backend; threaded execution reports threaded_row_tiles. Inspect cpu_batch_backend_algorithm, cpu_batch_scheduler_policy, and cpu_batch_kernel_isa to confirm the path that actually ran.

If neither tdse_local_threads_set() nor TDSE_LOCAL_THREADS is set, the runtime uses local_threads=1. This keeps route decisions and benchmark results reproducible across machines. Increase the value only after measuring the target np/nh/dtype shape on the target CPU.

Guidelines:

Keep AUTO on the default CPU route unless measurement shows a pinned backend is better for your workload.
Use explicit cpu_batch when you want to evaluate the preview local-thread route and are prepared to qualify the target shape with the CPU_BATCH evidence gates.
For N-model parallel workflows, divide CPU cores across models before using explicit cpu_batch.
More threads are not automatically faster. Small and medium shapes often lose time to synchronization, cache pressure, or memory bandwidth limits.
Setting local_threads higher than available cores provides no benefit.

Backend Selection Guide

Start with AUTO
|-- tiny np
|   `-- CPU AUTO uses cpu_pack_gemv with the generic fused-ring sub-route
|-- medium CPU shape
|   `-- CPU AUTO uses conservative cpu_pack_gemv unless signed evidence matches
|-- large fp64 history
|   `-- pin cpu_blas explicitly for experiments; qualify before promoting it to AUTO
|-- large fp32 history
|   `-- validate response error first; host-specific BLAS promotion requires evidence
`-- large GPU-friendly shape with CUDA available
    `-- request a preview CUDA backend explicitly; ordinary AUTO remains CPU fail-closed

TDSE_ENABLE_PREVIEW_CUDA_AUTO=ON is a release-unsafe build-time escape hatch for controlled benchmark lanes. Production builds leave it OFF until CUDA AUTO has a signed execution-context qualification descriptor.

Additional factors:

Factor	Backend Impact
Sparse matrix structure	CPU_BLAS_SPARSE may outperform dense even at moderate np
Multiple models in parallel	Each gets its own handle; divide CPU threads or GPU memory
Variable dt	Fast-path backends (CPU_BLAS, CUDA) optimize for uniform stepping
Pack size	Large nh increases convolution cost; GPU benefits more

When a route decision matters, inspect profiler or benchmark fields active_backend, route_reason, and hist_op_kind. These fields explain both the selected backend and the internal CPU history operation.

Build Features

Check what features the current build was compiled with:

/* Query required buffer size */
size_t json_len = 0;
tdse_perf_get_build_features_json(NULL, &json_len);

/* Allocate and query */
char* json = malloc(json_len);
tdse_perf_get_build_features_json(json, &json_len);
printf("Features: %s\n", json);
free(json);

This returns a JSON object listing compiled-in features such as CUDA support, BLAS backend, telemetry, and other optional components.

For CPU route triage, preserve the CPU feature portion of this JSON with any benchmark or customer support bundle. It records both compiled ISA lanes and runtime CPU support:

compiled_avx2, compiled_fma, compiled_avx512, compiled_neon, and compiled_sve describe code paths present in this binary.
runtime_cpuid_avx2, runtime_cpuid_fma, runtime_cpuid_avx512, runtime_cpuid_neon, and runtime_cpuid_sve describe hardware and OS support detected on this host.
supported_simd_width_bits and compiled_simd_width_bits separate host capability from code present in the binary. Build-feature JSON deliberately reports selected_simd_width_bits=0/selected_simd_isa=none; only per-model cpu_selected_simd_width_bits/cpu_selected_simd_isa are evidence of the latest executed kernel.
runtime_prefer_avx512 is the product-dispatch signal for AVX512. AVX512 support alone is not enough to prove it should be preferred on a specific CPU, because downclocking can erase the wider-vector benefit.
compiler_target_flags, blas_provider, blas_version, blas_thread_count, blas_link_mode, and openmp_runtime explain why CPU route evidence may differ across machines.
cpu_env_overrides captures route-affecting environment overrides such as TDSE_CPU_FORCE_SMALLNP, TDSE_CPU_FORCE_PACK_GEMV, TDSE_CPU_FORCE_FUSED_RING, TDSE_CPU_PREFER_AVX512, TDSE_LOCAL_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, and OMP_NUM_THREADS.

The versioned CPU_EXECUTION_V1 payload exposes the CPU compiled/runtime fields, cache policy, and environment overrides; CPU_ROUTE_V1 exposes the route policy/feature/environment/semantic signatures. Use tdse_model_diagnostics_query_many(...) when both must describe the same live model snapshot. Use the build-features JSON when collecting process or release evidence before model creation.

Performance Benchmarks

The numbers below are sizing guidance, not guarantees. They help you estimate whether TDSE is in the right range for your deployment before you run your own measurements. Unless noted otherwise, all data uses the default CPU backend (CPU_GENERIC) on a representative x86_64 workstation.

Treat this section as early sizing guidance, not release evidence. Procurement, PoC, and real-time sign-off decisions should be based on measurements from your target machine, target pack shape, and target host-loop policy.

Use the numbers below for triage questions such as "is CPU enough?", "is CUDA worth testing?", or "is this pack likely to fit in memory?". Do not use them as substitutes for customer acceptance criteria, target-machine qualification, or published product guarantees.

Representative Step Latency

Measurements with nh = 256, dt = 1e-6, and 10,000 warm-up steps followed by 10,000 measured steps:

np	nq	Backend	Step Latency (us)	Throughput (steps/s)
1	1	CPU_GENERIC	0.3-0.8	1.2M-3.3M
3	3	CPU_GENERIC	0.5-1.5	670K-2.0M
3	3	CPU_BLAS	0.4-1.2	830K-2.5M
10	10	CPU_GENERIC	2-8	125K-500K
10	10	CPU_BLAS	1-4	250K-1.0M
10	10	CUDA	5-15*	67K-200K
50	50	CPU_BLAS	20-80	12K-50K
50	50	CUDA	8-25	40K-125K
100	100	CPU_BLAS	100-400	2.5K-10K
100	100	CUDA	15-50	20K-67K

*CUDA numbers include host-device transfer overhead. Small models may not benefit from GPU due to transfer latency.

Memory Footprint

Approximate per-model memory usage:

np	nh	Dense Memory	Notes
3	256	~50 KB	Small model, typical transmission line
10	1024	~2 MB	Medium model, multi-port subsystem
50	2048	~80 MB	Large model, distribution network
100	4096	~600 MB	Very large, dense subsystem

Memory scales approximately as nh * nq * np * 8 bytes for the H tensor plus nq * np * 8 bytes for workspace.

Scaling Behavior

Linear in nh: doubling history depth approximately doubles per-step time
Quadratic in np: port count has the strongest impact; minimize ports where possible
Linear in model count: N independent models consume approximately N times the memory and can run in parallel on separate threads

Benchmarking Your Workload

To measure performance for your specific model:

tdse profiler calibrate --np <your_np> --nh <your_nh> --dtype 64 \
  --out-json ./profile.json --out-md ./profile.md

The profiler report includes:

per-step latency for each available backend
optimal backend recommendation
generated runtime plan for tdse_backend_apply_plan()

For local GPU runtime backend evidence, run:

python scripts/performance/run_local_gpu_runtime_decision_matrix.py \
  --preset engineering \
  --dtype both \
  --profile-scope history-output-optimizations \
  --skip-fp64-confirmation \
  --out-dir artifacts/local-gpu-runtime-engineering

Use --preset smoke only for quick route checks. Use --preset release when the run will be archived as formal evidence; it increases repetitions and adds a large stress shape.

The history-output-optimizations scope uses the maintained product CUDA policy (gpu_direct_history for Np <= 32, gpu_blas_graph for Np > 32) and tests the graph-D2H boundary only where graph replay is stable. Zero-copy host output is excluded from this scope unless --include-zero-copy-experiments is passed. See GPU Runtime History-Output Boundary for the timing definitions and current product decision.

For real-time deployments, also measure WCET with deterministic mode enabled (see Platform Notes).

For procurement, PoC, or customer-facing reporting, keep three evidence classes separate:

sizing guidance from this chapter
profiler output for your exact pack and hardware
target-machine timing or field qualification records from your deployment program

When sharing performance numbers outside the immediate engineering team, record at least:

CPU and GPU model
operating system and compiler/toolchain
build type and enabled backend/features
model shape (np, nq, nh, dt)
whether the host used fixed-step, variable-step, single-model, or multi-model execution
whether telemetry, deterministic mode, or additional tracing was enabled during measurement

Performance Monitoring Checklist

Verify backend selection with tdse_backend_get_active() after set
Compare step latency between backends for your model
Set local_threads intentionally rather than relying on default
Monitor guard metrics when tuning precision or dt strategy
Use Profiler to derive an optimal plan rather than manual tuning
Archive the runtime plan alongside pack artifacts for release evidence

Before Deployment Sign-Off

Use the next chapter based on the kind of risk you are trying to close:

If the open question is...	Go next
target platform support or qualification boundary	Platform Notes
plugin deployment, manifest, or ABI compatibility	Plugin System
circuit-input subset or netlist support risk	Element Reference
long-running observability rather than benchmark measurement	Telemetry

Backend Selection and Performance