Framework
Open-source distributed computing engine for large-scale data processing and analytics, widely used for batch, streaming, and SQL workloads across clusters.
A dev.to author measured Spark 4.2.0's Python payload at roughly 480 MB for full PySpark and about 20 MB for pyspark-client. The post argues that modules unreachable from that client, such as GraphX, SparkR and DStreams, are the ones fading.
Reality
A Spark benchmark's "Arrow" UDF row ran the pickle path, so its 48.9x against 48.5x tie says nothing about Arrow transport. The same benchmark's useArrow=True UDFs ran about 2x faster than the plain ones.
Reality
Spark 4.2 makes the Arrow paths PySpark's defaults, according to a dev.to history of how the format reached each JVM-to-Python boundary. Teams upgrading get that serialization without opting in, so jobs with UDFs and pandas conversions need a test run first.
Reality
Iceberg 1.11.0 ships a File Format API that sends Spark, Flink, Hive and plain-Java reads and writes through one registry of format models. A dev.to walkthrough says the file format list stays fixed, so Vortex and Lance still need a change inside Iceberg.
Reality
RustFS embeds an Apache-2.0 Iceberg REST catalog in its object store and serves it on the S3 port, according to a dev.to walkthrough. Teams can drop a separate metastore, but catalog metadata then shares a failure domain with the data.
Reality
Databricks has released Funke, a PySpark library and streaming pipeline that parses HL7v2 messages directly into a native Spark column. Teams that use it to replace a FHIR converter or a flattening vendor take on the gold-layer field mapping themselves, sender quirks included.
Reality
Amazon Web Services added Iceberg materialized views to Redshift, so a result Redshift computes once can be read by Spark and Athena, a dev.to write-up says. Teams that copy shared metrics between engines with ETL jobs can retire those jobs only if the views stay fresh enough for their readers.
Reality
Salesforce Data 360 instruments about eight million Spark runs a day at one submission gateway, without changing application code. The plugin fails open, so a broken collector leaves the platform team a data gap to chase while customer jobs keep running.
Reality
Dell is adding a semantic layer and knowledge graph to its AI Data Platform for AI agents, with both due in the first half of 2027. Dell says agents now burn tokens on every query piecing together answers that should already exist.
Perspective Coverage
3 publishersReality
Cloudflare made its Data Platform generally available as Basin, three serverless services that ingest, catalog and query Apache Iceberg tables on R2. Outside Iceberg engines read the same tables, so Workers teams can price Basin SQL against a warehouse on identical data.
Perspective Coverage
3 publishersReality
Apache Iceberg's REST Catalog spec now lets a catalog return per-user row filters and nine column actions for a trusted engine to enforce, Databricks says. Policies that need subqueries or lookup tables do not fit that vocabulary, so the portability covers simple rules running on engines a deployment already trusts.
Reality
Dropping three driver-side rewrite passes and one Step Functions wave produced most measured savings in a Glue pipeline's 56% runtime cut. An unindexed Oracle partition and a driver memory ceiling show where more concurrency stops helping.
Reality
Keeping the Iceberg table format governance-unaware was a deliberate call. That leaves the catalog as the only thing standing on the path of every read. A dev.to guide lists six pieces an operator has to source above it.
Reality
Astronomer's State of Apache Airflow 2026 puts 5,818 practitioners behind two migration numbers measured on different denominators. Recomputed on one base, about 62 percent are planning a move they have not started.
Reality
A GCP walkthrough put Spark, Trino and BigQuery on the same orders table through Google's Iceberg REST catalog. The 409 ConflictException it logged is the part of the demo a production pipeline has to plan for.
Reality
The partition count was frozen at checkpoint creation because each partition has its own RocksDB instance on disk. In Databricks Runtime 18, a separate config resizes the store and moves the state to match.
Reality
A new Iceberg REST Catalog addition lets the catalog evaluate policy for one principal and return nine column actions plus row filters for a trusted engine to apply, while the spec leaves establishing that trust to the deployment.
Reality
Microsoft documents no configurable mirroring window, and support confirmed that restarting a mirror reloads the whole table instead of resuming from the last CDC position. The estate here has roughly 2,500 tables.
Reality
Uber reported 169:1 on 200TB of Spark logs a day by separating static templates from variable fields before anything was written to HDFS. Whether that ratio transfers depends on how few call sites your lines come from.
Reality
Adyen traced sandbox setup failures on its big data clusters to a garbage collector that walks every tracked connection in order, which turns the table-size knob you raised for headroom into part of the fault.
Reality
Leadership · August 27, 2026 · One report1 publisher