In this briefing
- 01Real input distributions change how training capacity is calculated
- 02Real service traffic drives continual adjustment of inference engines
- 03Long prompts make the key-value cache shared infrastructure
- 04Policy routing connects performance objectives to tenant governance
- →What to watch next
- ↗Sources and verification
- Anyscale and Lila Sciences report results from training GLM-5.3-Flash on 64 NVIDIA B200 graphics processing units. Real data made the load on the busiest expert nearly 4 times the average, and a configuration that could run with a random-token benchmark could exhaust graphics memory with real data.
- Together AI connects continual compilation to real production traffic, and Together AI reports a throughput improvement of up to 2.6 times. The result is supplier research, and the result depends on the specific model, hardware, baseline and service objective.
- Supermicro, IBM and NVIDIA extend the key-value cache into shared storage, while Buoyant places cache-aware routing, queue-aware routing and tenant policy in Kubernetes. Both sets of material require an enterprise to place performance, isolation, failure and version maturity in the same acceptance conditions.
Real input distributions change how training capacity is calculated
On 5 November, Anyscale published an engineering account of the work by the Lila Sciences team to scale training for GLM-5.3-Flash. The team used 64 NVIDIA B200 graphics processing units to process sequences as long as 1 million tokens, and the team reported throughput of 13,700 tokens per second for sequences of 1 million tokens and 24,900 tokens per second for 256,000-token sequences. The team also reported a 4.8% improvement in held-out loss on genomic data after 80 training steps. These figures come from team testing and not from independent testing.
Real training data changed the distribution of capacity pressure. The team observed that expert load on the busiest graphics processing unit was nearly 4 times the average expert load, and that a 512,000-token configuration accommodated by a random-token benchmark exhausted graphics memory with real data. The team estimated that a sequence of 1 million tokens would bring the input for one layer on each graphics processing unit close to 180 GB before context parallelism, and that the total for an 8-graphics-processing-unit node would exceed 1 TB. The team used full recomputation, central-processing-unit offloading and context parallelism to control peak graphics memory, but these methods respectively increased computing, transfer or communication costs.
For enterprise adoption, model parameter count and a synthetic benchmark cannot by themselves determine the usable capacity of a training cluster. An enterprise needs to replay capacity tests with its own sequence lengths, expert routing, sample distribution and batch conditions, and the enterprise needs to record peak graphics memory, communication, step time, failure rate and convergence changes together. The results from Anyscale and Lila Sciences show that real data can change the location of a bottleneck, but the Anyscale and Lila Sciences results cannot prove that another model, dataset or cluster will produce the same figures.
Real service traffic drives continual adjustment of inference engines
On 4 November, Together AI published the Together Compiler paper. Together AI divides compilation into continual experimentation, candidate validation and controlled deployment, and Together AI uses real production traffic to preserve representative input shapes, request mixes and service objectives. Together AI states that a one-off offline optimisation becomes obsolete as models, hardware, frameworks and traffic change, so continual compilation needs to compare candidate configurations continuously while an engine operates.
Together AI reports an average throughput improvement of 1.7 times and a maximum improvement of 2.6 times, and Together AI reports an improvement in experiment throughput of up to 252 times. The paper also lists the specific models, accelerators, inference frameworks and comparison baselines, so the figures describe only the test scope in the paper. Together AI runs correctness and performance checks before a candidate engine enters production, but the public paper does not replace independent acceptance of accuracy, tail latency, failure recovery and cost by the target enterprise.
For enterprise adoption, inference optimisation is changing from a one-off deployment setting into a continually changing operating process. An enterprise needs to retain the model version, compiler version, traffic sample, accuracy result, latency distribution, rollback condition and approval record for every candidate configuration, and the enterprise needs to prevent experimental traffic from affecting production tenants. The results from Together AI support the direction that real traffic can improve the representativeness of optimisation, but the Together AI supplier research cannot prove that every workload will achieve the same benefit.
Long prompts make the key-value cache shared infrastructure
On 5 November, Supermicro announced a key-value cache offering built jointly with IBM and NVIDIA. The offering connects Supermicro Petascale all-flash servers, IBM Storage Scale, NVIDIA Dynamo and NVIDIA Spectrum-X, and the offering extends the key-value cache for long prompts from one inference node into shared storage. The joint team tested openai/gpt-oss-120b with 4 NVIDIA RTX PRO 6000 graphics processing units, 8 storage nodes, vLLM v0.14.1 and NVIDIA Dynamo v0.9.0 or later.
The IBM Redbooks document defines the scope of the joint test. The joint team sent 200 requests, used 28 concurrent connections and used 100 unique prompts, and the joint team processed approximately 24 million tokens and an 825 GB cache. The joint team included a cold start in the test, and the joint team reported an improvement of up to 56 times in time to first token with a 130,000-token prompt and a change from 0.19 requests per second to 4.26 requests per second. The joint team also reported an improvement of up to 18 times under a noisy-neighbour condition. Every result comes from joint supplier testing, and the results do not prove that another network, storage system, model or request mix will produce the same performance.
For enterprise adoption, long-context inference places storage throughput, network paths, cache hits, tenant isolation and failure recovery in the critical path of model serving. An enterprise needs to test a cold cache, a warm cache, storage failure, network congestion and a noisy neighbour separately under real prompt-reuse and concurrency conditions, and the enterprise needs to check how sensitive context in the cache is encrypted, isolated, expired and audited. The joint offering provides a reproducible test boundary, but the joint offering provides no independent testing, common price, complete supply list or support conditions for every region.
Policy routing connects performance objectives to tenant governance
On 4 November, Buoyant announced that Kinference was generally available, and Buoyant described Kinference as an open-source inference-serving platform for Kubernetes. Kinference builds on vLLM, and Kinference uses cache awareness, queue awareness and request characteristics to select an endpoint. Buoyant states that Kinference also supports recording usage by tenant, enforcing policy and expanding requests to cloud resources.
The product announcement and the public documentation provide different maturity signals. The Buoyant announcement uses generally available language, but the Kinference public documentation identifies the current version as v0.1.0-alpha.7. The public material provides no common price, complete compatibility matrix, long-term support period or independent production benchmark, so an enterprise still needs to validate version stability, upgrade methods, cloud-expansion boundaries and failure conditions separately.
For enterprise adoption, a router cannot pursue only a higher cache-hit rate or a shorter queue. An enterprise needs to make every routing decision follow data-residency, tenant-isolation, model-licensing, cost-budget, service-level and human-approval rules together, and the enterprise needs to retain every routing choice and its result. The open-source form of Kinference can provide an inspectable implementation entry point, but the generally available status in the announcement and the pre-release version marker in the documentation require an enterprise to include software maturity and support responsibility in the acceptance scope.
What to watch next
- Can Anyscale and Lila Sciences publish more results for convergence, communication, energy use and training failures under real data distributions?
- Can Together AI provide more material for independent reproduction, and can Together AI explain how continual experimentation limits production-traffic risk and configuration rollback time?
- Can Supermicro, IBM, NVIDIA and Buoyant publish common pricing, supply regions, long-term support periods, tenant-isolation validation and independent production benchmarks?
Sources and verification
Golden Data has edited this briefing from the public materials listed above. The original sources govern facts and figures. The enterprise relevance sections are Golden Data editorial analysis and do not constitute an endorsement of any third-party product.
← Back to AI Daily Briefing