Data Engineering

ResearchPrimary topic: Data EngineeringAlso: AI & Machine Learning

Article: AI Changes the Data Interface, Not the Query Engine

Crest summary (AI-written):InfoQ published an article arguing that AI changes the data interface rather than the query engine, illustrated with a disaster-recovery analytics workflow.

EngineeringPrimary topic: SecurityAlso: Cloud InfrastructureAlso: Data Engineering

Netflix Bridges AWS IAM and Internal Identity with Workload Attestation

Crest summary (AI-written):Netflix built a workload attestation mechanism for Apache Spark that connects AWS IAM with its internal identity system.

Companies: AWS · Netflix

PartnershipPrimary topic: ObservabilityAlso: Data Engineering

Monitor Databricks with Datadog

Crest summary (AI-written):Datadog published guidance on monitoring Databricks, dated 2026-10-02.

Companies: Databricks · Datadog

OtherPrimary topic: ObservabilityAlso: Data Engineering

Databricks’ native monitoring resources

Crest summary (AI-written):Datadog published guidance on collecting metrics from Databricks' native monitoring resources.

Companies: Databricks · Datadog

LaunchPrimary topic: Cloud InfrastructureAlso: Data Engineering

Announcing Cloudflare K2: serverless event streams

Crest summary (AI-written):Cloudflare announced K2, a serverless event streaming service built on R2 object storage, on 2026-10-01. It targets high-scale data movement and long-term retention without traditional broker operations.

Companies: Cloudflare

LaunchPrimary topic: Data EngineeringAlso: Cloud Infrastructure

Introducing Cloudflare Basin: an open, serverless data platform, now generally available

Crest summary (AI-written):Cloudflare released Basin, a serverless data platform built on Apache Iceberg and R2 Object Storage, now generally available. Developers can ingest, manage, and query large datasets at scale without data egress fees.

Companies: Cloudflare

ResearchPrimary topic: AI AgentsAlso: Data Engineering

AI agents speed up development — data access slows them down

Crest summary (AI-written):AI agents are accelerating application and internal tool development, but data access is now the bottleneck slowing teams down, according to The New Stack.

EngineeringPrimary topic: Data EngineeringAlso: Databases

How we extended Apache DataFusion to execute one query across many machines

Crest summary (AI-written):Datadog described extending Apache DataFusion so a single query can execute across many machines, according to a post dated 2026-10-01.

Companies: Datadog

UpdatePrimary topic: Data EngineeringAlso: Cloud InfrastructureAlso: Databases

Amazon S3 Tables now support all Apache Iceberg V3 data types

Crest summary (AI-written):Amazon S3 Tables now support all Apache Iceberg V3 data types, plus column default values, deletion vectors, and row lineage. Existing V2 tables can be upgraded or new V3 tables created.

Companies: Amazon S3 Tables

UpdatePrimary topic: DatabasesAlso: Data EngineeringAlso: Cloud Infrastructure

Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data in your data lake

Crest summary (AI-written):Amazon Aurora PostgreSQL can now query Apache Iceberg and Parquet data in data lakes directly, using embedded DuckDB, without ETL pipelines. The capability allows single queries joining transactional and historical data.

Companies: Amazon

UpdatePrimary topic: AI AgentsAlso: Data Engineering

Microsoft Fabric is where AI agents learn how the business works

Crest summary (AI-written):Microsoft is positioning its Fabric data platform as the place where enterprise AI agents learn how a business operates.

Companies: Microsoft

LaunchPrimary topic: ObservabilityAlso: Data Engineering

Troubleshoot Kafka issues across every layer of your stack with Kafka Console

Crest summary (AI-written):Datadog announced Kafka Console, a tool for troubleshooting Kafka issues across every layer of the stack, in a post dated 2026-09-10.

Companies: Datadog