This page covers the engine specifically — the inferencing platform that owns Model lifecycle, GPU scheduling, and sleep/wake. For the layers above (Projects / Services UI, customer-facing API keys) see LiquidCompute and AI Console. For the whole picture see Platform Overview.

Components

The engine ships as four Go binaries from a single Helm chart:

Routing — getting an HTTP request from the cluster edge to the right Model pod — is not owned by the engine. Use a standard Ingress controller, or layer LiquidCompute on top for a unified /v1/* proxy.

High-level flow

1. You declare a Model. The webhook validates cross-field constraints (parallelism, sleep config, role layout).
2. The controller renders a pod with two containers: vLLM + the GreenThread sidecar. It also renders a per-Model ResourceClaim against the greenthread.ai DeviceClass.
3. When the pod is scheduled, the agent's DRA kubelet plugin receives NodePrepareResources, reserves a slot on the requested GPU, and writes a per-claim CDI spec.
4. containerd reads the CDI spec and injects libgreenthread.so plus GPU device nodes into the workload.
5. The sidecar takes over: it owns the HTTP front-door, the sleep/wake state machine, and coordinates with the per-GPU scheduler daemon for fair multi-tenant access.

Custom resources

$ kubectl get crds | grep greenthread
gpuquota.greenthread.ai
gpus.greenthread.ai
gpushareclaims.greenthread.ai
models.greenthread.ai
storagenodes.greenthread.ai

The Model CR

Most users only ever write Model. Example from a live cluster:

apiVersion: greenthread.ai/v1alpha1
kind: Model
metadata:
  name: whisper-turbo
  namespace: default
spec:
  modelName: openai/whisper-large-v3-turbo
  modelType: asr                 # chat | tts | asr | embedding | reranker
  dtype: bfloat16
  loadFormat: gthread            # gthread (fast DMA) | auto (vLLM default)
  endpoints: [audio_transcriptions, audio_translations]
  parallelism: { tensor: 1, pipeline: 1 }
  replicas: 1
  fairness: { popular: true }    # pin to GPU; never preempted
  sleep: {}                      # defaults: idleTimeout=10m, evacuateContext=false
  product: NVIDIA-RTX-PRO-4000-Blackwell
  uuids:
    - GPU-f78cce19-fbed-7f66-d72f-a24cae367fc8

Status reflects live state:

status:
  phase: Ready
  readyReplicas: 1
  replicaStatus:
    - podName: whisper-turbo-0
      phase: Serving                # Serving | Sleeping | Loading | Waking | Failed
      message: sidecar state=serving
  memory:
    servingHuman: "20.0 GiB"
    sleepingHuman: "0 B"
  node: blackwell-0

See Model CRD Reference for the full field list and Lifecycle & States for the state machine.

GPU sharing modes

The DRA driver gates each physical GPU into one of:

Once a GPU's mode is set, only matching-class claims are eligible — the slice's CEL selectors enforce this at the scheduler. See Fairness Policy for how the per-GPU scheduler arbitrates between siblings.

Supported APIs

Every Model exposes standard endpoints. By default the controller routes them at /<model-name>/v1/* on the cluster Ingress; layer LiquidCompute for a unified /v1/* with model selection by request body.

OpenAI-compatible

Anthropic-compatible

Scoring / reranking / tokenization

See API Reference for full details, request/response examples, and SDK usage.

Key features

True zero VRAM sleep

When a Model sleeps, the engine releases all GPU memory — not 90%, not "most of it" — truly zero bytes. The GPU is fully available for other Models with no wasted memory.

Sub-second wake

Wake from CPU RAM in ~600 ms. Clients see slightly higher latency on the first request, but the response format is identical. Subsequent requests are at full vLLM speed.

Fair GPU sharing

The fairness policy ensures no Model starves. Models idle longest are slept first; fairness.popular: true pins a Model to its GPU and exempts it from preemption.

Per-Model routing

Each Model gets its own URL path (/<model-name>/v1/...) by default. Or use LiquidCompute for a single /v1/* with selection by request model field.

Monitoring built in

Prometheus metrics, Grafana dashboards, and alerting are available out of the box. See Monitoring for setup.

Deployment options

GreenThread deploys on any environment with NVIDIA GPUs. Pick your platform:

Next steps

• Prerequisites — GPU Operator + DRA + cert-manager
• Install GreenThread — Helm install and verify
• LiquidCompute — add the UI + unified /v1/* on top
• AI Console — customer-facing portal
• Deploying Models — deploy your first models