AWS says Amazon EMR Serverless now supports shuffle operations of up to 1 TB, compared with what it describes as a previous 200 GB per-job limit.
The change may matter to Spark teams constrained by shuffle storage, but the announcement does not establish how any particular job will perform. Here is what changed, where AWS says it is available, and what remains to be checked.
Data Center News & Trends
What changed—and are the two figures directly comparable?
In an announcement published October 1, 2026, AWS said Serverless Storage on Amazon EMR Serverless supports shuffle operations of up to 1 TB. It described the previous limit as 200 GB per job. The announcement does not explicitly say that the new 1 TB figure is also a per-job limit, so the figures should not be presented as a like-for-like per-job comparison.
AWS — Serverless Storage on Amazon EMR Serverless now supports terabyte-scale shuffle
Which Spark workloads might care?
Spark can produce substantial shuffle data when it rearranges data for joins, aggregations and sorting. AWS highlights large-table joins and high-cardinality aggregations as examples. A team whose jobs encountered the previous shuffle-storage limit has a reason to examine the new support. AWS also says it added spill support, which offloads data to disk when needed during memory-intensive operations.
Dataset size is not the shuffle allowance. AWS mentions joins across multi-terabyte datasets, but that example does not mean multi-terabyte shuffle operations are supported; the announced shuffle figure is up to 1 TB.
Where is it available, and what has not been demonstrated?
AWS lists Amazon emr-7.14, emr-spark-8.1 and later, across 18 AWS Regions where EMR Serverless is available. It directs readers to its documentation for the supported Regions and their applicable limits; the announcement does not list each regional limit.
AWS describes improved reliability and job success rates as benefits of spill support. The announcement, however, provides no measured results for a particular job’s success rate, runtime or cost. Availability of the feature is not evidence that a given job will finish faster or cost less.
What should a team take away?
Treat the announcement as a reason to reassess jobs that may have been constrained by shuffle storage, not as a performance result. Check the job’s EMR version, Region and applicable limit against its shuffle demand. Then assess completion, runtime and cost on the job itself. That separates a newly stated storage capability from operational improvements that still need to be established.
No comments:
Post a Comment