Models are deployed as Model custom resources in the greenthread-system namespace. The Model CRD controls every aspect of how your model is served — from GPU memory allocation and tensor parallelism to sleep behavior, fairness policy, and LoRA adapter support.

Full spec

apiVersion: greenthread.ai/v1alpha1
kind: Model
metadata:
  name: llama-3-1-8b
  namespace: greenthread-system
spec:
  modelName: meta-llama/Llama-3.1-8B-Instruct
  dtype: bfloat16
  gpuMemoryUtilization: "0.90"
  tensorParallelSize: 1
  replicas: 1
  quantization: ""
  extraArgs:
    - "--enforce-eager"
  sleep:
    idleTimeout: 5m
    drainTimeout: 30s
  fairness:
    minRuntime: 10s
    maxWaitTime: 5s
    popular: false
  staging:
    pinned: false
  lora:
    enabled: false
    maxRank: 64
    cacheDir: ""
    downloadTimeout: 5m
  kvCache:
    enabled: false
    maxSize: ""
  resources: {}

Core fields

modelName

modelName: meta-llama/Llama-3.1-8B-Instruct

The HuggingFace model identifier. GreenThread downloads the model, converts it to the gthread load format, and stores it on NVMe. This field is required.

dtype

dtype: bfloat16

Data type for model weights. Affects VRAM usage and inference precision.

gpuMemoryUtilization

gpuMemoryUtilization: "0.90"

Fraction of GPU memory vLLM is allowed to use. Range: (0, 1]. Stored as a string to preserve decimal precision.

Lower values leave room for other models to share the same GPU via sleep/wake scheduling.

tensorParallelSize

tensorParallelSize: 2

Number of GPUs to shard the model across using tensor parallelism. Must be a power of 2.

Each replica uses tensorParallelSize GPUs. The scheduler assigns contiguous GPU indices on the same node.

replicas

replicas: 1

Number of serving pods to create. Each replica independently claims GPUs and handles requests.

quantization

quantization: mxfp4

Quantization method applied during model conversion. Must be compatible with the model weights.

extraArgs

extraArgs:
  - "--enforce-eager"
  - "--max-model-len"
  - "8192"
  - "--max-num-seqs"
  - "128"

Raw CLI arguments passed directly to the vLLM inference engine. Each argument is a separate list item.

Common extraArgs:

Sleep spec

Controls when and how models go to sleep. See Lifecycle & States for details on sleep/wake transitions.

sleep:
  idleTimeout: 5m
  drainTimeout: 30s

When a model sleeps, GreenThread releases all VRAM, making the GPU available for other models.

Fairness spec

Controls preemption behavior when multiple models compete for GPU memory. See GPU Scheduling for the full algorithm.

fairness:
  minRuntime: 10s
  maxWaitTime: 5s
  popular: false

Staging spec

Controls how model weights are staged in CPU memory for fast wake. See Storage & Pinning for details.

staging:
  pinned: false

LoRA spec

Configures LoRA adapter support. See LoRA Adapters for the integration pattern.

lora:
  enabled: true
  maxRank: 64
  cacheDir: /tmp/lora-cache
  downloadTimeout: 5m

KV cache spec

Configures KV cache persistence via LMCache. When enabled, vLLM stores computed KV cache entries in Redis so that repeated prompt prefixes skip recomputation.

kvCache:
  enabled: true
  maxSize: "10Gi"

Pod placement

Node selector

nodeSelector:
  nvidia.com/gpu.present: "true"
  topology.kubernetes.io/zone: "us-west-2a"

Standard Kubernetes node selector to constrain which nodes the model pod can run on.

Tolerations

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

Standard Kubernetes tolerations for GPU node taints.

Service account

serviceAccountName: my-model-sa

Kubernetes ServiceAccount for the model pod. Used for IRSA/workload identity when LoRA adapters need to download from object storage (e.g. S3).

Image overrides

sidecarImage: registry.example.com/greenthread-sidecar:v1.2.0
vllmImage: registry.example.com/vllm:v0.8.0

Override the default sidecar or vLLM container images. If empty, the controller uses its built-in defaults.

Resource overrides

resources:
  requests:
    cpu: "4"
    memory: "32Gi"
  limits:
    cpu: "8"
    memory: "64Gi"

Standard Kubernetes resource requirements for the vLLM container.

Status fields

The controller and sidecar populate these status fields. They are read-only.

kubectl get model llama-3-1-8b -n greenthread-system -o yaml

kubectl output

kubectl get model -A

NAMESPACE            NAME                       PHASE   REPLICAS   SERVING   SLEEPING   NODE        GPUS   VRAM/GPU     URL
greenthread-system   llama-3-1-8b               Ready   1          0         1          gpu-node1   0      17.2 GiB     http://<gateway>/llama-3-1-8b/v1
greenthread-system   qwen-3-5-35b-a3b           Ready   1          1         0          gpu-node1   1      88.2 GiB     http://<gateway>/qwen-3-5-35b-a3b/v1