Spotify is one of the largest digital systems in the world—its codebase spans over 20 million lines. How does the company handle the challenges of scaling, security, and performance? Discover the technologies, architectures, and teams behind this engineering behemoth.
In a digital world where applications must serve millions of users, code scalability is no longer a luxury—it is a necessity. As a streaming giant, Spotify has spent years developing systems that allow it to manage a massive codebase without losing control over quality, security, and performance. In 2026, with the number of lines of code in the company's core projects exceeding 20 million, the question is no longer "does it work?", but "how does it work at this scale?".
The answer lies in a combination of modern tools, decentralized architecture, and automation—primarily driven by agent systems that act as "digital assistants" for engineers. But what does this look like in practice? Which technologies underpin such a system, and what challenges must be overcome to maintain operational fluidity?
Technologies keeping 20+ million lines of code in check
Spotify does not rely on a single technology; it uses a mosaic of tools, languages, and frameworks that complement each other. The key components of this puzzle are:
- Programming languages: The company still maintains some legacy systems in
Javie, but newer components are built primarily inScali,Kotlinie(for mobile apps), andGo(for high-performance microservices). Python is used mainly for automation scripts and data analysis. - Version control system: Git is the foundation, but Spotify takes it a step further—some projects use a monorepo, a single, massive repository containing the entire application code and its dependencies. This approach simplifies dependency management but requires robust build and test tooling.
- Build and test: To manage the monorepo, the company adopted Bazel—a build system that allows for incremental compilation of only the changed parts of the code. This is crucial when dealing with millions of lines that must be compiled in a reasonable timeframe.
- Orchestration and deployment: Kubernetes acts as the "engine" that runs microservices and manages their lifecycle. To simplify deployment processes, Spotify uses GitOps—an approach where the cluster state is synchronized directly from a Git repository, for example, using Argo CD.
All these tools are integrated into a cohesive ecosystem where a code change can be deployed to production within minutes—provided, of course, it passes all tests and reviews.
Agent architecture: from microservices to "digital assistants"
Agent systems at Spotify are not individual applications, but a distributed network of microservices, each serving a specific role. This architecture is based on several key principles:
1. Autonomous microservices
Each agent acts as a separate microservice, communicating with others via gRPC or REST. Examples of such agents include:
- Validation agents: Automatically review pull requests, checking for code quality, standard compliance, and potential security issues.
- Deployment agents: Manage the rollout of changes to production environments using GitOps and Kubernetes.
- Monitoring agents: Collect performance metrics, alerts, and logs, helping SRE teams maintain system stability.
- Automation agents: Handle simple tasks, such as automatically merging PRs after all tests pass.
This model allows for flexibility and resilience—the failure of one agent does not block the entire system, and new features can be added without disrupting the existing structure.
2. Central "control plane" — Backstage
Although agent systems are distributed, their management is facilitated by Backstage—an open-source platform created by Spotify that acts as a "command center." Backstage integrates with:
- Git (for code review and dependency management),
- Kubernetes (for monitoring cluster state),
- CI/CD (e.g., GitHub Actions or Jenkins),
- Bazel (for building and testing).
Thanks to Backstage, engineers have one place where they can find documentation, deployment status, and even trigger automated workflows. This tool is critical for maintaining consistency in such a complex ecosystem.
You could say that Backstage is the "operating system" for agent systems—without it, managing such a large number of components would be significantly more difficult.
Scaling challenges: how to handle millions of lines of code?
A scale of 20+ million lines of code is not just a technical challenge, but an organizational one as well. Spotify faces several key issues:
1. CI/CD performance in a monorepo
Building the entire codebase on every change would be unthinkable. The solution is incremental building via Bazel, which compiles only the changed parts of the project. Additionally, Spotify uses self-hosted GitHub Actions runners to offload public servers and accelerate processes.
2. Data consistency and code quality
To ensure that changes do not introduce bugs, Spotify employs automated code reviews using validation agents. These tools check for:
- Compliance with coding standards,
- Common errors (e.g., null pointers in Java),
- Security issues (e.g., vulnerabilities in dependencies).
Additionally, the company uses automated dependency updates (e.g., using Renovate) to avoid outdated libraries.
3. Security at scale
As the codebase grows, so does the attack surface. Spotify employs multi-stage security scanning:
- Automated dependency scanning (e.g., using Snyk),
- Real-time vulnerability monitoring,
- Restricting microservice permissions (principle of least privilege).
This approach minimizes the risk of a single dependency vulnerability going undetected.
4. Teams responsible for system maintenance
To keep everything running like clockwork, Spotify relies on specialized teams:
- Platform Engineering: Responsible for Backstage, Bazel, and Kubernetes—the tools that enable agent systems to function.
- Site Reliability Engineering (SRE): Ensures stability, monitoring, and incident response. Their work is critical when dealing with millions of users streaming music globally.
- Developer Productivity: Focuses on improving the developer experience, for instance, by automating repetitive tasks.
All these teams collaborate to create a seamless engineering ecosystem where code changes are deployed quickly without losing control over quality.
Public resources and the future of agent systems
Spotify does not keep its solutions secret—some of its knowledge is shared publicly. You can find:
- Spotify Engineering Blog, where case studies on automation, CI/CD, and microservice architecture are described. Example post on deployment automation.
- Backstage.IO, documentation and source code for the platform that helps manage agent systems. Official documentation.
- Conference talks, e.g., at KubeCon, where Spotify engineers share their experiences with Kubernetes automation. KubeCon 2023 presentation.
As for the future, Spotify plans to further implement AI-driven development. They are already experimenting with AI-powered automated code review tools that can suggest fixes or even repair simple bugs. Such an approach could further accelerate development processes.
Summary: what can we learn from Spotify's approach?
Agent systems, monorepo, GitOps, and distributed architecture—these are all pieces of the puzzle that allow Spotify to keep over 20 million lines of code in check. Key takeaways for other companies developing software at scale include:
- Choose the right tools for the scale: Bazel, Kubernetes, and GitOps are essential, but every project requires an individual approach.
- Automate everything possible: From code reviews to deployments—the more repetitive tasks you offload to agents, the fewer bugs will reach production.
- Maintain consistency via central platforms like Backstage, which simplify the management of complex systems.
- Invest in teams and culture: Without the right SRE, DevOps, and Platform Engineering teams, even the best tools will not be enough.
Spotify proves that scale does not have to mean chaos. With the right architecture, tools, and approach to automation, even the largest systems can run efficiently without losing control over quality and security.
One might ask: "Will this approach work in my company?" The answer depends on many factors—code size, number of teams, and development speed expectations. But one thing is certain: if you want to scale software, you must start by scaling your processes and tools. And that is exactly where agent systems excel.
Sources
- https://youtu.be/9DHZLw5653E?si=-7sb3YV2KXLZmewY
- https://engineering.atspotify.com/2021/04/14/scaling-our-backend-with-backstage/
- https://www.cncf.io/webinars/managing-monorepos-at-scale/
- https://blog.bazel.build/2022/10/20/spotify-bazel.html
- https://engineering.atspotify.com/2020/02/25/ci-cd-at-spotify/
- https://kubernetes.io/blog/2021/04/spotify-kubernetes/
- https://engineering.atspotify.com/2018/02/14/our-microservice-architecture/
- https://backstage.io/case-studies/spotify
- https://github.blog/2021/03/18/spotify-github-actions/
- https://engineering.atspotify.com/2020/05/14/code-reviews-at-scale/
- https://snyk.io/blog/spotify-security/
- https://docs.renovatebot.com/case-studies/spotify/
- https://static.sched.com/hosted_files/kccnceu23/75/Automating%20Kubernetes%20with%20Agents.pdf
Comments