Running DeepSeek V4.1 Flash in production: serving, KV caching and autoscaling on AWS

We've used frontier models for quite some time, and we still rely on them heavily for every customer. They're the backbone of how we serve customers today. But they're heavily guardrailed, and access for cyber use sits behind gated verification programs. For the offensive side of security work, that's a real constraint: a model can refuse an authorized exploit step halfway through an engagement, and access can hinge on a verification program you're still waiting to clear. No company can rely on frontier models alone for that work. Meanwhile, attackers are already equipped with open-source models and much more. We've been building and testing agent swarms on self-hosted models, and DeepSeek V4.1 Flash is what we run them on. In one black-box run, a swarm found an exploitable flaw in a password reset flow at negligible cost. This is our setup.

Why open-source models, and why fine-tune them

Gated access is worth waiting for, but waiting doesn't stop the work. Open-source models fill the gap in three ways:

• No gate. Open weights run on hardware we control, under our own authorization and scoping rules, so there's nothing to wait on and no mid-engagement refusals.
• Customer data stays on our GPUs. Target responses, credentials found in scope and tool output never leave infrastructure we own.
• We can fine-tune them. A frontier model is a fixed artifact. An open model can be trained on our own validated findings, tool-call traces and exploit chains, so it learns our harness, our tool schemas and what a real finding looks like. On narrow, high-volume steps like triage, payload variation and tool calling, a tuned open model can close much of the gap with a frontier model, on capacity we already pay for.

It isn't either-or. When gated access opens up, frontier models take the hardest reasoning and open models take the volume.

The rest of this guide covers how it's built: the serving configuration, the sizing model, KV cache tiering including a cache shared across machines, routing, autoscaling and cost. All numbers are from our production fleet.

Workload profile

1. Hardware

At these prompt lengths, capacity is set by KV cache bytes, not by weights. Blackwell's native FP4 support matters twice: for the MoE weights and for a 4-bit KV cache format. With nvfp4_ds_mla, one instance holds 7,732,336 tokens of KV in GPU memory at --gpu-memory-utilization 0.92, about 68 conversations of 113K tokens resident at once.

2. Serving configuration

We use vLLM with the vLLM-Moet r23 patches, which add SM120 kernels for the MoE, sparse MLA attention and the DSpark speculative head. Here's the full container launch, with secrets and paths shown as variables:

docker run -d --name vllm --restart unless-stopped --gpus all --ipc host --network host --shm-size 48g \
  --ulimit memlock=-1 --ulimit stack=67108864 --cap-add IPC_LOCK \
  -v $DATA/models:/models:ro -v $DATA/cache:/root/.cache -v $DATA/deep_gemm:/root/.deep_gemm -v $DATA/kvcache:/srv/kvcache \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_HTTP_TIMEOUT_KEEP_ALIVE=75 \
  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  -e NCCL_P2P_DISABLE=0 -e NCCL_P2P_LEVEL=SYS -e VLLM_MOET_P2P_AR=0 \
  -e VLLM_MOET_B12X_MOE_FUSED=1 -e VLLM_MOET_SPARSE_MLA_LDG=1 -e VLLM_MOET_TARGET_LM_HEAD_MXFP8=1 \
  -e VLLM_MOET_PREFILL_SP=1 -e VLLM_MOET_B12X_MOE_PREFILL_FAST=1 \
  -e NIXL_TELEMETRY_ENABLE=y \
  $IMAGE \
  --model /models --served-model-name deepseek-v4.1-flash --api-key $API_KEY --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 --disable-custom-all-reduce \
  --tokenizer-mode deepseek_v41 --reasoning-parser deepseek_v41 \
  --tool-call-parser deepseek_v41 --enable-auto-tool-choice \
  --kv-cache-dtype nvfp4_ds_mla --enable-prefix-caching \
  --gpu-memory-utilization 0.92 --max-model-len 524288 \
  --max-num-seqs 32 --max-num-batched-tokens 4096 \
  --max-cudagraph-capture-size 128 --moe-backend b12x \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3,"draft_sample_method":"probabilistic","rejection_sample_method":"block","moe_backend":"b12x"}' \
  --attention-config '{"indexer_kv_dtype":"mxfp4","indexer_tp_shard_min_rows":2048}' \
  --kv-offloading-size 256 \
  --kv-transfer-config "$KV_TRANSFER_CONFIG"   # section 5

What the important flags do:

How many draft tokens to use. We ran k=5 and k=3 side by side on live traffic, with the same mix at the same time. k=5 raised tokens per step from 3.09–3.11 to 3.26–3.39 (+8%), cut KV capacity from 7.73M to 7.17M tokens because of larger draft buffers, and verifies more tokens per step, which costs more at high batch sizes. k=3 is the better operating point for prose-heavy agent traffic. Code-heavy traffic accepts far more draft tokens, and the published vLLM-Moet runs at k=7 reach ~6 tokens per step on code.

3. Sizing: a per-step latency model

Decode is memory-bandwidth bound. Every step, every running sequence reads its whole KV history. At our context lengths, step time is close to linear in batch size:

t_step(N) ≈ 10 ms + 2.7 ms × N        (N = running sequences, ~113K tokens each)
tok/s per request ≈ a / t_step(N)      (a ≈ 3.0 accepted tokens per step)

That predicts ~95 tok/s at N=8 and ~57 tok/s at N=16. Measured on one instance, with ~84K-token prompts and 512 output tokens:

Aggregate throughput rises with N, but per-request speed falls roughly as 1/(c + N). For agents, wall-clock time per investigation is driven by per-request speed, so we fix a target and size from it: N ≈ 8 per instance → ~90 tok/s per request. That target sets the autoscaling threshold in section 7.

Published single-stream numbers for this hardware, around 750 tok/s, use code prompts with ~6 accepted tokens per step and contexts of 65K tokens or less. Benchmark at your own prompt length and content mix.

4. KV cache tiering on one instance

Prefill for a 165K-token prompt takes ~9 s on one instance. Agent turns re-send nearly the same prefix, so the target is to compute each prefix once. vLLM's OffloadingConnector with TieringOffloadingSpec adds a primary CPU tier and ordered secondary tiers below GPU memory:

Single-instance configuration:

{
  "kv_connector": "OffloadingConnector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {
    "spec_name": "TieringOffloadingSpec",
    "block_size": 1024,
    "secondary_tiers": [
      {"type": "fs", "root_dir": "/srv/kvcache/r23-b1024",
       "n_read_threads": 16, "n_write_threads": 16, "locality": "LOCAL"}
    ],
    "offload_prompt_only": false
  }
}

• block_size is the offload chunk size in tokens. It must be a multiple of the GPU block size, and it sets the granularity of every secondary-tier object.
• offload_prompt_only: false also offloads decode-phase KV, so content generated in one turn can be reused in the next.
• The RAM tier is a named region in /dev/shm. A cron job that removes regions no process maps (checked against /proc/*/maps) keeps restarts deterministic.

On our always-on instance, the prefix hit rate is ~99% of prompt tokens. Steady-state prefill is about 2,000 new tokens/s.

5. A shared KV tier across instances (S3)

With more than one instance, a conversation can land on an instance that has never seen it: router spill-over, a new replica, a Spot reclaim. Without a shared tier, that costs a full prefill. vLLM's obj secondary tier stores KV chunks in S3 through NIXL's OBJ backend, so any instance can load what another computed.

Configuration

Append to secondary_tiers:

{"type": "obj", "prefix": "r23-tp4-nvfp4-b1024", "io_threads": 64, "locality": "REMOTE",
 "store_config": {
   "bucket": "<kv-bucket>", "region": "us-east-2", "scheme": "https",
   "endpoint_override": "https://s3.us-east-2.amazonaws.com",
   "ca_bundle": "/etc/ssl/certs/ca-certificates.crt"}}

• Objects are keyed {prefix}/{hash_shard}/{chunk_hash}.bin. At startup the tier writes a connectivity probe at __nixl_probe__/connectivity_test.
• With no static keys in store_config, credentials come from the AWS SDK default chain. On EC2 that's the instance role through IMDSv2. With --network host, the container reaches IMDS at hop limit 1.
• io_threads sets the parallelism of the NIXL OBJ backend's S3 client.

Image requirements

The OBJ plugin, libplugin_OBJ.so in the nixl_cu13 wheel, links against NVIDIA cuObject and cuFile. Neither is in the stock image, and cuObject is packaged for CUDA 13.1 while the image runs CUDA 13.0. A minimal layer:

FROM vllm-moet-sm120:r23
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
      libcuobjclient-13-1 libcufile-13-1 librdmacm1t64 && rm -rf /var/lib/apt/lists/* \
 # keep the toolkit on 13.0; only cuFile/cuObject come from 13.1
 && update-alternatives --set cuda /usr/local/cuda-13.0 && update-alternatives --set cuda-13 /usr/local/cuda-13.0 \
 && L=/usr/local/cuda-13.1/targets/x86_64-linux/lib P=/usr/local/lib/python3.12/dist-packages/nvidia/cu13/lib \
 # one libcufile.so.0 per process: point the pip copy at cuFile 1.16, which cuObject 13.1 requires
 && mv $P/libcufile.so.0 $P/libcufile.so.0.cu130 && ln -s $L/libcufile.so.0 $P/libcufile.so.0 \
 && mv $P/libcufile_rdma.so.1 $P/libcufile_rdma.so.1.cu130 && ln -s $L/libcufile_rdma.so.1 $P/libcufile_rdma.so.1

Requirements, and why:

You can check the backend independently of vLLM with a round trip through NIXL:

from nixl._api import nixl_agent, nixl_agent_config
a = nixl_agent("t", nixl_agent_config(backends=[]))
a.create_backend("OBJ", {"bucket": "<kv-bucket>", "endpoint_override": "https://s3.us-east-2.amazonaws.com",
                         "scheme": "https", "region": "us-east-2",
                         "ca_bundle": "/etc/ssl/certs/ca-certificates.crt"})
# register a DRAM buffer and an OBJ descriptor (key in metaInfo), then WRITE and READ:
# 8 MiB object: write ~160 ms, read ~95 ms from an instance in the same region

Chunk size

block_size determines object size, and that's what matters most for S3:

Larger chunks mean fewer requests per byte and better per-request throughput. The cost is granularity: a prefix is reusable only up to the last full chunk, at most 1,023 tokens, under 1% of a 113K prompt. The chunk layout is part of the key space, so put block_size in both the S3 prefix and the fs root_dir.

Cache key compatibility

Instances can share entries only if they produce identical block hashes and identical KV bytes:

• same weights, TP degree, kv_cache_dtype, indexer dtype and block_size;
• same prefix-hash function. vLLM uses sha256 with a fixed seed unless PYTHONHASHSEED is set, so set it everywhere or nowhere.

Version the prefix whenever any of these change.

Measured results

• A freshly booted replica served 33–42% of its non-GPU prompt tokens from the shared tier in its first few hundred requests.
• Average S3 promotion: 0.6–0.7 s per load. A local miss triggers an asynchronous S3 lookup of ~0.6–0.75 s.
• Storage stays bounded with an S3 lifecycle rule: expire after 1 day, and abort incomplete multipart uploads after 1 day.

Observability

The tier exports per-tier Prometheus counters. The useful ones:

vllm:kv_offload_tiering_chunk_queries_total{tier="2:obj"}
vllm:kv_offload_tiering_chunk_hits_total{tier="2:obj"}
vllm:kv_offload_tiering_read_bytes_total{tier="2:obj"}      # S3 → CPU
vllm:kv_offload_tiering_write_bytes_total{tier="2:obj"}     # CPU → S3
vllm:kv_offload_tiering_promotion_latency_seconds{tier="2:obj",result="success"}
vllm:external_prefix_cache_hits_total / vllm:external_prefix_cache_queries_total

6. Routing

Request path: ALB (HTTPS) → router → vLLM replicas. The router is vllm-router in prefixaware mode, which keeps a prefix tree mapping prompt prefixes to backends, so each turn goes to the replica holding its KV.

For production we use three adjustments:

• Keep the prefix tree across config reloads. Rebuilding the routing policy when the backend list changes resets placement for every conversation.
• Bound the prefix tree. It grows with distinct prefixes. We reset it every 6 hours.
• Spill over on load imbalance. If the matched backend has ≥ 8 more in-flight requests than the least-loaded one, route to the least-loaded backend. The conversation's next turns follow it there, and the shared S3 tier turns the move into a ~2 s load instead of a full prefill.

A controller next to the router runs every 15 s. It discovers replicas (the always-on instance plus every InService ASG instance whose /health returns 200), writes the router's dynamic backend config, drains replicas held by an ASG termination lifecycle hook until in-flight reaches 0, and publishes CloudWatch metrics. A replica is marked unhealthy only after 15 minutes of failed health checks.

Spot replicas run in a second region (us-east-2), connected by VPC peering with ~11 ms RTT. They reach S3 through a VPC gateway endpoint. The always-on instance in us-east-1 reaches the us-east-2 bucket over the public endpoint with an instance role. Inter-region transfer is about $0.02/GB.

7. Autoscaling

Metric, published by the controller every 15 s:

Utilization = (Σ num_requests_running + Σ num_requests_waiting) / (32 × serving_replicas) × 100

Step scaling on the Spot ASG (min 0, max 8):

• Hysteresis. Removing one of N replicas at ≤ 10% leaves the rest at ≤ 10% × N/(N−1) ≤ 20%, below the 25% scale-out threshold, so the fleet doesn't oscillate.
• Scale in on the average. Agent traffic is bursty within 5-minute windows, so a Maximum statistic rarely meets a low threshold.
• Warm-up. EstimatedInstanceWarmup is 1,800 s, so repeated alarms during boot add only the difference.
• Spot settings. Capacity Rebalancing on, AZ rebalancing suspended so warm replicas aren't replaced to even out zones, and a single instance type.
• Drain on removal. The termination lifecycle hook holds the instance until in-flight requests reach 0.

Replica boot timeline, with everything served from an in-region S3 bucket:

Because boot takes about 18 minutes, size for sustained load. Over a typical week, the always-on instance alone served 78 of 168 hours, one or two Spot replicas covered 56 more, and the peak hour needed seven.

8. Unit costs

At ~8 requests per replica on our traffic, the fleet runs about $10,000–17,000 a month, depending on the per-request speed target at peak. Cost depends on provisioned capacity, not on tokens, so long contexts, retries and multi-step reasoning don't change the bill.

Checklist

1. Benchmark per-request tok/s at your real prompt length, fit t_step ≈ c + k·N, and choose a target N per replica.
2. Use nvfp4_ds_mla KV, an MXFP4 indexer and prefix caching. With long contexts, KV capacity is the constraint.
3. Keep --gpu-memory-utilization at 0.92 or below, and validate memory changes on production-shaped traffic.
4. Add RAM and NVMe offload tiers through TieringOffloadingSpec, and clean up orphaned /dev/shm regions.
5. For more than one replica, add the obj tier: cuObject + cuFile 1.16 in the image, ca_bundle, NIXL_TELEMETRY_ENABLE=y, block_size ≈ 1024, a versioned prefix, and a 1-day lifecycle rule.
6. Route with a prefix-aware router, with a bounded tree and load spill-over.
7. Autoscale on (running + waiting) / slots, with hysteresis, average-based scale-in and drain hooks.
8. Keep an owned baseline instance, and put burst capacity on Spot.