01
Per-layer mixed precision
FP8 and INT4 chosen layer by layer, not one setting for the whole model.
Quantize · Quantra
Quantra searches per-layer FP8 and INT4 precision and sparsity, calibrates on representative data, and measures speed and accuracy on real GPUs. It keeps only configurations inside your accuracy budget.
Capabilities
01
FP8 and INT4 chosen layer by layer, not one setting for the whole model.
02
Structured and unstructured sparsity tried and validated on hardware.
03
Representative calibration samples picked automatically from your data.
04
Every configuration scored on your evaluation set before it can be kept.
05
Accuracy against throughput for every point tried, so you choose with evidence.
06
Compressed models for cloud, on-prem and edge through Edgyn.
Accuracy budget
Every configuration Quantra tries is measured for speed and accuracy. Move the budget and see which ones would be kept. Example data.
Best kept configuration
FP8 + KV cache FP8
Inside budget: 7 of 10
Example configurations for one 70B chat model on H100. Values are illustrative.
How it works
Select calibration data and record baseline accuracy.
Propose per-layer precision and sparsity policies.
Run each on the GPU for throughput and on the eval set for accuracy.
Keep the fastest point inside the budget.
Specification
| Item | Detail | Status |
|---|---|---|
| Input | Model, eval set, accuracy budget | Pilot |
| Precisions | FP8, INT8, INT4 weights | Pilot |
| Sparsity | 2:4 structured, unstructured | Next |
| Engine | TensorRT-LLM quantized engines | Pilot |
| Targets | Hopper, Blackwell, Jetson Orin | Pilot · Planned |
| Output | Compressed model, accuracy report | Pilot |
Place in the loop
It takes Profyle's targets and hands compressed models to Servex for the cloud and to Edgyn for devices.
FAQ
You do, on your own evaluation set. We suggest a starting point, but the number and the eval set are yours.
Quantra reports the confidence of each accuracy measurement and flags results where the eval set is too small to tell.
Yes. Quantra often keeps sensitive layers, such as some attention layers, in BF16, and you can pin layers yourself.
FP8 throughput gains need Hopper or Blackwell tensor cores. On older GPUs Quantra uses INT8 and INT4 where they help.
Next step
In the pilot Quantra produces a full accuracy-throughput report for one model on your evaluation set.