Amazon's Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2

I'm one of the creators of Ray. A few thoughts :)

1. This is truly impressive work from AWS. Patrick Ames began speaking about this a couple years ago, though at this point the blog post is probably the best reference. https://www.youtube.com/watch?v=h7svj_oAY14

2. This is not a "typical" Ray use case. I'm not aware of any other exabyte scale data processing workloads. Our bread and butter is ML workloads: training, inference, and unstructured data processing.

3. We have a data processing library called Ray Data for ingesting and processing data, often done in conjunction with training and inference. However, I believe in this particular use case, the heavy lifting is largely done with Ray's core APIs (tasks & actors), which are lower level and more flexible, which makes sense for highly custom use cases. Most Ray users use the Ray libraries (train, data, serve), but power users often use the Ray core APIs.

4. Since people often ask about data processing with Ray and Spark, Spark use cases tend to be more geared toward structured data and CPU processing. If you are joining a bunch of tables together or running SQL queries, Spark is going to be way better. If you're working with unstructured data (images, text, video, audio, etc), need mixed CPU & GPU compute, are doing deep learning and running inference, etc, then Ray is going to be much better.

I was in a workshop that taught me Ray. It was interesting to know that the people who started Spark were also involved in making Ray.

This is not badmouthing either project just an observation and if you architected one task you would be good at attacking the same problem better .

Im curious, how do data scientists use these massive datasets, especially the old stuff. Is it more of a compliance and need/should-save type thing or is the data actually useful? Im baffled by these numbers having never used a large BI tool, and am genuinely curious how the data is actually used operationally.

As a layman, I imagine lots of it loses relevancy very quickly, e.g Amazon sales data from 5 years ago is marginally useful to determining future trends and analyzing new consumer behavior regimes?

Anyone know enough about ray to comment on what the exact performance unlock was ?. They mention that it gave them enough control over the distribution of work so that they could avoid unnecessary reads/write. That seems like a good win but I would assume that doing compaction in python would be quite slow.

Video from the author deep diving this. https://www.youtube.com/watch?v=h7svj_oAY14

Crazy that the project took almost 4 years end-to-end, and it's still ongoing.

I had no idea anything at AWS had that long of an attention span.

It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!

I remember when everyone shifted from Apache Hadoop to Apache Spark. This seems like a possibly similar sea change. I am not sure if many other users will embrace Ray over Spark, but this is a sign that people are looking to either improve Spark on some fundamental levels, or are going to reach out to new technologies to resolve their problems. Cool stuff.

I work on Daft and we’ve been collaborating with the team at Amazon to make this happen for about a year now!

We love Ray, and are excited about the awesome ecosystem of useful + scalable tools that run on it for model training and serving. We hope that Daft can complement the rest of the Ray ecosystem to enable large scale ETL/analytics to also run on your existing Ray clusters. If you have an existing Ray cluster setup, you absolutely should have access to best-in-class ETL/analytics without having to run a separate Spark cluster.

Also, on the nerdier side of things - the primitives that Ray provides gives us a real opportunity to build a solid non-JVM based, vectorized distributed query engine. We’re already seeing extremely good performance improvements here vs Spark, and are really excited about some of the upcoming work to get even better performance and memory stability.

This collaboration with Amazon really battle-tested our framework :) happy to answer any questions if folks have them.

We chose Ray over Spark in my previous company mostly because we were a Python shop and Ray is Python-native (though it's implemented in C++ I believe). It worked very well for us even for real-time queries - though we were obviously nowhere near the scale that AWS is at.

Has anyone found a good ELI5 site for all the different AI toolsets.

Eg Ray, Databricks, Notebooks etc

Absolutely insane work. So much data you’d think they would come up with a custom solution instead of using the “newest available toolkit” but I understand how much of a mess dealing with that much data is.

Can you share any data on how big of a cluster is running Ray jobs?

As someone who used Ray for ML, this is impressive as hell.

I did not even realize it was that viable a substitute for straight data processing tasks.

Why did it take multiple years to do that I wonder? After a new compaction framework is ironed out it should not be that difficult to onboard/spread it across to all the tables in especially considering they had a parallel setup in Spark still, so they can afford hiccups in a new setup.

Can you help us understand how others can use and derive value from Ray DeltaCAT? What would be the specific use cases?

I wonder if similar performance can be achieved with Spark accelerator like https://github.com/apache/datafusion-comet. Of course it didn’t exist 4 years ago, but would it cheaper to build?

I wonder if there are any good primers to these technologies. Maybe a DDIA-like book or some lectures?

> From the typical Amazon EC2 customer’s perspective, this translates to saving over $120MM/year on Amazon EC2 on-demand R5 instance charges.

Does the sales team know about this? /jk

Rather nuts. New challenge: build datacenters quickly enough to support the new platform.

Ray user here, what language actors are they using? Ray support Python Java and cpp actors…

Are we talking about big data ETL here? I did not know Ray was suited for it.

Slightly flip, but it's interesting that no one believes in or brags about cost savings via statistical sampling techniques these days.

I wonder what they’re doing to combat the growth rate of their data. A 13x speed up, or 82% cost reduction is great, but doesn’t seem significant enough compared to the growth of the business and (my assumption) demand for adding new data sources and added data to existing sources.

Like, if the current latency is ~60 minutes for 90% of updates, will it ever be better than that? Won’t it just slowly degrade until the next multi-year migration?

PS: this article was infuriating to read on iPad - it kept jumping back to the top of the page and couldn’t figure out why

Are they that bored over there at amazon.