TensorRT Optimization

GPU latency কমানো।

🎬 গল্প দিয়ে শুরু

একই YOLOv8n PyTorch-এ 30 ms, TensorRT FP16-এ 5 ms। সমান GPU, কিন্তু ৬× দ্রুত। NVIDIA-র TensorRT production inference-এর gold standard।

TensorRT কী করে?

Layer fusion (Conv+BN+ReLU = এক kernel)।
Precision calibration — FP32 → FP16 / INT8।
Kernel auto-tuning — GPU অনুযায়ী সেরা CUDA kernel।
Memory reuse — workspace optimization।
Dynamic shape সমর্থন।

Workflow

text

PyTorch (.pt)
    ↓ export
ONNX (.onnx)
    ↓ trtexec / TensorRT API
Engine (.engine / .plan)   ← GPU-specific, ঐ device-এ build করতে হয়
    ↓
TensorRT Runtime inference

Portability

Engine file একই GPU + একই TRT version-এ run করে। Target device-এ (Jetson, A100, 4090) আলাদা build দরকার।

ONNX → Engine (trtexec CLI)

bash

trtexec --onnx=yolov8n.onnx \
        --saveEngine=yolov8n_fp16.engine \
        --fp16 \
        --minShapes=images:1x3x640x640 \
        --optShapes=images:1x3x640x640 \
        --maxShapes=images:4x3x640x640

Ultralytics shortcut

python

from ultralytics import YOLO
m = YOLO("yolov8n.pt")
m.export(format="engine", half=True, dynamic=True, device=0)
# inference
YOLO("yolov8n.engine")("test.jpg")

Python API — manual inference

python

import tensorrt as trt, pycuda.autoinit, pycuda.driver as cuda
import numpy as np

logger = trt.Logger(trt.Logger.WARNING)
with open("model.engine", "rb") as f, trt.Runtime(logger) as rt:
    engine = rt.deserialize_cuda_engine(f.read())
ctx = engine.create_execution_context()

# allocate
inp  = np.random.randn(1, 3, 640, 640).astype(np.float32)
d_in = cuda.mem_alloc(inp.nbytes)
out_shape = (1, 84, 8400)
out  = np.empty(out_shape, np.float32)
d_out= cuda.mem_alloc(out.nbytes)

stream = cuda.Stream()
cuda.memcpy_htod_async(d_in, inp, stream)
ctx.execute_async_v2([int(d_in), int(d_out)], stream.handle)
cuda.memcpy_dtoh_async(out, d_out, stream)
stream.synchronize()

Precision strategy

FP32 — baseline, accuracy 100%।
FP16 — 2× দ্রুত, accuracy drop নগণ্য (best default)।
INT8 — 3–4× দ্রুত, calibration dataset দরকার, কিছু accuracy hit।
INT4 / sparsity — Hopper/Ada GPU-তে আরও আক্রমণাত্মক।

Production tip — Triton Inference Server

Triton

NVIDIA Triton multi-model, dynamic batching, A/B test, metrics — সব built-in। TRT engine drop করলেই সব features পান। Production-এ single-process FastAPI-র চেয়ে অনেক superior।

প্র্যাকটিস টাস্ক

YOLOv8n FP32 / FP16 / INT8 engine বানিয়ে FPS compare করুন।
Dynamic batch shape দিয়ে 1/2/4/8 batch-এ throughput মাপুন।
Triton-এ YOLO + classifier দুটি model একসাথে serve করুন।