How it works

Measured, then kept

A Turbokern run looks like a careful performance engineer's week, compressed: baseline, profile, try many changes, keep the ones that survive measurement, ship the result.

A typical runPlanning estimate
Input
Model, GPU, objective
Duration
1-3 days
Candidates
100s generated
Benchmarked
Top 5-10%
Output
Engine, recipe, report
Planning estimates for one LLM endpoint.

The run

Six steps, in order

Each step writes to the run report, so you can see what was tried and why.

  1. 01Hour 1

    Baseline

    Run the model as you serve it today and record latency, throughput, memory and accuracy.

  2. 02Hours 1-4

    Profile

    Profyle traces the workload and produces a ranked opportunity map.

  3. 03Minutes per batch

    Predict

    Simyx scores every candidate the optimizers propose and passes on the most promising.

  4. 04Days 1-3

    Optimize

    Kernl, Quantra and Servex try kernels, precisions and serving configs.

  5. 05Per candidate

    Verify

    Each candidate runs on the real GPU against correctness and accuracy guards.

  6. 06Hours

    Package

    Winning changes are combined, re-measured together and packaged.

Who does what

Clear lines between you and the agent

TaskTurbokernYour team
Define the objective and guardsSuggests defaultsDecides
Provide the evaluation setValidates it runsProvides
Profile and find bottlenecksDoes itReviews the map
Generate and test candidatesDoes itNothing required
Approve deploymentPackages the engineApproves and ships
Re-run on model or GPU changeTriggers automaticallySets the policy

Configuration

Targets and guards in one file

The objective says what better means. Guards say what must never get worse. The budget says how much GPU time the search may use.

  • If the target cannot be met within the guards, the run says so and reports the best result it found.
  • You can lock any stage off, for example no INT4 for a safety-reviewed model.
turbokern.yaml
objective:
  throughput_tok_s: maximize
  latency_p99_ms: 120   # hard limit
guards:
  max_accuracy_drop: 0.01
  stages: [profyle, kernl, quantra, servex]
budget:
  gpu_hours: 96

Kept or rejected

Every candidate gets a verdict

This is what the run report's candidate ledger looks like. Most candidates are rejected; that is the point of measuring.

  1. C-014Fused attention, FlashAttention-style128×64 tiles, 2-stage pipelineAttention was memory-bound; fusion cuts HBM traffic.−14.2 msKept
  2. C-021Split-K GEMM on o_projsplit 4Predicted gain on small batch; measured slower.+2.1 msRejected
  3. C-033CUTLASS GEMM, FP8 inputs256×128 tile, persistentTensor-core utilization rose from 58% to 81%.−6.4 msKept
  4. C-038Fused RMSNorm and residual addone kernel instead of twoRemoves a launch and a round trip to memory.−1.9 msKept
  5. C-041Attention, 64×64 tiles3-stage pipelineFailed numerical check: max error above tolerance.fails checkRejected
  6. C-047Rotary embedding fused into QKVfused epilogueFaster on prefill, slower on decode; net worse at p95.+0.6 msRejected

Example run on one H100. Candidates, speedups and reasons are illustrative, not measured results.

What you get

Three things at the end of every run

Artifact

A deployable engine

A TensorRT-LLM engine, Triton model repository or NIM container, built for the exact GPU you named.

Recipe

A reproducible recipe

Every kept change with versions and parameters, so the engine can be rebuilt and diffed.

Report

A run report

Baseline, final numbers, every candidate tried, and why each was kept or rejected.

FAQ

Questions engineers ask

How long does a run take?

For one LLM endpoint we plan for one to three days, mostly GPU time. Smaller vision or speech models are faster. The compute budget you set caps it.

What if the target cannot be met?

The run ends with the best result inside your guards and a report explaining which bottleneck remains, for example memory bandwidth at that batch size.

Can I keep part of the result?

Yes. Each kept change is listed separately in the recipe, and you can rebuild with any subset.

What triggers a re-run?

A new model version, a new GPU type, a TensorRT-LLM upgrade or a traffic shift beyond a threshold you set.

Next step

Start with a profile

The pilot begins with Profyle on one workload, so you see the opportunity map before committing GPU time to optimization.