LLM Infrastructure

LaunchTopic: LLM Infrastructure

EmbeddingGemma 2: The Developer Guide

Crest summary (AI-written):Google released EmbeddingGemma 2, an open-source multimodal embedding model that maps text, code and images.

Companies: Google

LaunchPrimary topic: LLM InfrastructureAlso: ML Frameworks & TrainingAlso: Cloud Infrastructure

Announcing New vLLM and SGLang Inference Engines on TPU

Crest summary (AI-written):Google announced native PyTorch backends for the vLLM and SGLang inference engines on TPU, powered by TorchTPU.

Companies: Google

LaunchTopic: LLM Infrastructure

Gemma-4-26B-A4B with 39-69% less thinking at same accuracy

Crest summary (AI-written):Google released Gemma-4-26B-A4B, a model that uses 39-69% less thinking while keeping the same accuracy.

LaunchTopic: LLM Infrastructure

Kimi K3.1 Pro and Kimi K3.1 Pro Highspeed are available now in AI Gateway

Crest summary (AI-written):Moonshot AI released Kimi K3.1 Pro, a 3T-class open-weight model with a 1M-token context window, and Kimi K3.1 Pro Highspeed, a faster version for interactive use, both available on Vercel's AI Gateway.

Companies: Artificial Analysis · Kimi · Moonshot · Moonshot AI · Vercel

OtherTopic: LLM Infrastructure

Designing an AI serving harness built to last

Crest summary (AI-written):MIT Technology Review published an article arguing that enterprise AI advantage now comes less from model choice than from the serving harness built around the model.

UpdateTopic: LLM Infrastructure

DeepSeek V4.1-Flash-1017 vs. Claude Haiku 5.5: More accurate at a fraction of the price

Crest summary (AI-written):DeepSeek released V4.1-Flash-1017 on October 17 as an update to its Flash model.

Companies: DeepSeek

OtherTopic: LLM Infrastructure

IBM CEO says open weight models are now irresistibly good and cheap on-prem

Crest summary (AI-written):IBM's CEO said open weight models now deliver 90 percent of the performance at ten percent of the price when run on-premises in just two or three racks.

Companies: IBM

SecurityPrimary topic: SecurityAlso: LLM Infrastructure

LiteLLM supply chain attack hoovers up passwords and secrets

Crest summary (AI-written):A supply chain attack on LiteLLM is stealing passwords and secrets from affected systems, and also installs a remote access trojan.

EngineeringTopic: LLM Infrastructure

How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

Crest summary (AI-written):A Hugging Face article describes specializing the vocabulary of small language models to cut inference cost by 77% in real applications.

Companies: Hugging Face

LaunchPrimary topic: RAGAlso: AI Agents

Nvidia bets open models can give enterprise AI agents the context they’re missing

Crest summary (AI-written):Nvidia released new open-weights models for embedding, parsing and reranking, aimed at European customers and at agentic RAG workflows.

Companies: Nvidia

PartnershipPrimary topic: LLM InfrastructureAlso: Cloud Infrastructure

Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU

Crest summary (AI-written):Google Cloud integrated TPU support natively into the vLLM serving engine, enabling developers to elastically scale high-demand embedding inference.

Companies: Google · Google Cloud

LaunchPrimary topic: LLM InfrastructureAlso: ML Frameworks & Training

Autonomous LLM post-training with Tunix on TPUs

Crest summary (AI-written):Google's autofinetune project introduces an autonomous research loop that fully automates LLM post-training workflows, including fine-tuning, on TPUs via Tunix.

Companies: Google

EarningsTopic: LLM Infrastructure

TypeSafe CEO Diogo Almeida says Jev is in use by ~25% of Fortune 500 companies and "we were at a trillion tokens per day about a week ago"

Crest summary (AI-written):TypeSafe CEO Diogo Almeida said the company's Jev product is used by roughly 25% of Fortune 500 companies and recently reached about a trillion tokens per day.

Companies: TypeSafe

LaunchTopic: LLM Infrastructure

From the creator of Redis; run LLM locally with ds4

Crest summary (AI-written):The creator of Redis has released ds4, a tool for running large language models locally, according to a Hacker News post dated 2026-10-02.

Companies: Redis

LaunchPrimary topic: API & IntegrationAlso: LLM Infrastructure

Introducing Web Search API via AI Gateway

Crest summary (AI-written):Cloudflare added native web search API integration to AI Gateway through partnerships with Ceramic.ai, Exa, and Linkup, letting developers inject real-time web context into model inference calls.

Companies: Ceramic.ai · Cloudflare · Exa · Linkup

LaunchPrimary topic: LLM InfrastructureAlso: ML Frameworks & Training

Introducing Clef: our open-source decision models, and new RL fine-tuning platform

Crest summary (AI-written):Cloudflare introduced Clef and Clef-flash, open-source decision models hosted on Workers AI for classification and agentic workflows, plus a reinforcement learning platform for fine-tuning decision models with developers' own data.

Companies: Cloudflare · Hugging Face

OtherTopic: RAG

Use Graph RAG when relationships are part of the evidence

FundingTopic: LLM Infrastructure

Exclusive: Dig Ventures raises $120m to back Europe’s AI infrastructure startups

Crest summary (AI-written):Dig Ventures has raised $120m for a fund targeting Europe's AI infrastructure startups, according to an exclusive report.

Companies: Dig Ventures

LaunchTopic: LLM Infrastructure

Laya decision model now available on AI Gateway, free through October 31

Crest summary (AI-written):Convai Innovations' Laya decision model is now available on Vercel's AI Gateway, free to use for one month through October 31, 2026 via Boundless.

Companies: Boundless · Convai Innovations · Vercel

PartnershipPrimary topic: LLM InfrastructureAlso: API & Integration

Microsoft AI models are now available on AI Gateway

Crest summary (AI-written):Vercel and Microsoft AI have partnered to bring MAI models to AI Gateway, making Vercel one of a limited number of platforms offering access to them.

Companies: MAI · Microsoft AI · Vercel

UpdatePrimary topic: Vector DatabasesAlso: Cloud InfrastructureAlso: RAG

Amazon S3 Vectors now supports metadata pre-filtering for higher recall on filtered searches

Crest summary (AI-written):Amazon S3 Vectors added metadata pre-filtering, which evaluates filters before similarity search and delivers up to 5x higher recall on filtered searches.

Companies: Amazon

LaunchTopic: RAG

Cohere’s faster query model barely dents retrieval quality in its tests

Crest summary (AI-written):Cohere released Embed 5 on Wednesday, letting teams index data with Embed 5 Pro and query it with a faster model that its tests show barely reduces retrieval quality.

Companies: Cohere

OtherTopic: LLM Infrastructure

Eclipse wants companies to be free to switch AI providers. Today, doing so can mean a costly rebuild.

Crest summary (AI-written):Eclipse is pushing for companies to be able to switch AI providers freely, noting that today changing providers can require rebuilding workloads and moving data at significant cost.

Companies: Eclipse

LaunchTopic: LLM Infrastructure

Ling 3.1 Flash is now available on AI Gateway

Crest summary (AI-written):InclusionAI's Ling 3.1 Flash, a hybrid reasoning language model, is now available on Vercel's AI Gateway and is free to use through October 13, 2026.

Companies: InclusionAI · Vercel

OtherTopic: LLM Infrastructure

Why Featherless says you don’t need a tank to deliver a pizza

Crest summary (AI-written):Featherless argues that task-appropriate model size and scale matter more than raw power, as the debate over matching models to tasks continues.

Companies: Featherless