Massive Data - Spotify Wrapped 2021

I, alongside two other amazing engineers, presented a talk at SpEC 2022- Spotify's annual Engineering Conference! We talked about how we built and shipped personalized data pipelines for Spotify Wrapped 2021 for millions of users.

Embedding

At Spotify, engineers have the opportunity to do an "embed" in another team, which means switching teams for a period of time (anywhere from weeks to months) and working on projects different than their usual team. I was very lucky in discovering an embed opportunity that resulted in an amazing learning experience.

My day-to-day at Spotify is primarily working as a backend engineer on the artist-facing side, Spotify for Artists. In 2021, I embedded with a team working on the consumer-facing side project, Spotify Wrapped, and instead, as a data engineer.

So much data

Joining the Wrapped team was a brand new experience. It felt like I had just started a new job again. I learned so much in the weeks that followed- optimizing my Scala code, writing new data pipelines, debugging bottlenecks with hours-long running data jobs, and so much more. And what made this such an incredible and cool project was working with some of the largest data pipelines within Spotify, and enabling workflows to contribute to a personalized data experience that millions of users enjoy each year.

There's a lot more I could write about in working on the data side of Wrapped, but my teammates and I did a talk that goes over some highlights! Watch it below or you can watch it directly on YouTube!