Spark Performance Optimization Series: #1. Skew

In Spark cluster data is typically read in as 128 MB partitions which ensures even distribution of data. However, as the data is transformed (e.g. aggregated), it is possible to have significantly…

apache spark Archives - Sync

List: Spark Optimization, Curated by Ashwin Krishnan

Optimizing the Skew in Spark

i.ytimg.com/vi/d41_X78ojCg/sddefault.jpg

Handling Data Skew in Apache Spark, by Dima Statz

Databricks Notebook Promotion using Azure DevOps, by Himansu Sekhar, road to data engineering

Spark Performance Optimization Series: #1. Skew, by Himansu Sekhar, road to data engineering

Optimizing Apache Spark Performance: Tackling Data Skew for Faster Big Data Processing, by VivekR

How to Optimize Your Apache Spark Application with Partitions - Salesforce Engineering Blog

Cranking the Voltage on Spark: Achieve Peak Performance with Optimization, by BlackRockEngineering

Monitoring Apache Spark – We're building a better Spark UI - KDnuggets

Himansu Sekhar – Medium

List: Apache Spark, Curated by Luan Moreno M. Maciel