Perfect Scalability - by Michael Nash

50:31 1216 views 100% Published 6 years ago

This talk was recorded at Scala Days New York, 2016. Follow along on Twitter @scaladays and on the website for more information

One of the great benefits of the Lightbend ecosystem is its ability to empower great scalability, and it is often considered for this very reason.Indeed, if employed correctly, Scala, Akka and the rest can make it possible to much more easily write systems that scale up (and out) to extremes. However, not every design is able to scale, and the approach and architecture can sharply limit this option, no matter how good the tools. What does ""perfect scalability"" look like? Is there even such a thing? Can systems be designed to scale virtually without limit? Where does the pattern break down, and what can you do about it?

In this session, we will examine the practical architectural constraints on systems intended to scale up in near-linear fashion. It turns out that having the ability to scale to extreme load is more about what not to do than it is about what to do. Why do some system scale up and others don't? How can you take a design that has problems and refactor to a design that has fewer limits? Our teams encounter these problems every day, and we keep track of what works and what doesn't in the real world. What have we seen in the field? What proven real-world design and architecture approaches can we use to help ensure our systems can scale, and what should we avoid? How can we leverage the power of the Lightbend platform to architect such systems? Which parts for which kinds of problems? How does Akka and the actor model fit these patterns? How does Spark, Kafka, Akka streaming?

Highly scalable systems often put together tools such as Docker, Ansible, Salt, Mesos, ConductR with techniques such as microservices, monitoring, and continious delivery. Automation of the deploy pipeline and deep performance monitoring are essential parts of highly scalable systems - we will look at examples of such systems, and which of the tools in the stack they employ, and why. In this talk, we'll examine both the practices and the pitfalls, and what combinations we have used to scale not only within single clusters, but across data centers for continent-spanning applications where required.We will use use-cases all the way from massive IoT scale wearable devices all the way to high finance, and find the commonalities in the solutions that don't limit their own expansion. If your organization needs to take your scalability to significantly higher levels, this talk will be where you need to start.

Watch on YouTube