Phỏng vấn ở Sillicon Valley phần 2

Tiếp theo phần 1, mình xin kể tiếp câu chuyện phỏng vấn ở Silicon Valley. Sau lần thất bại đó, đơn vị tiếp tục đưa ứng viên thứ 2 ra phỏng vấn. Thật bất ngờ, khi bác Director of Software engineer lại sử dụng y chang bộ câu hỏi lần trước. Đó là tình huống không được tính đến, vì ai cũng nghĩ rằng cụ sẽ hỏi câu hỏi khác. Kết quả thì ai cũng biết, lại tạch.

Tuy nhiên, với phương châm không bỏ cuộc, never give up, vì nếu fail phỏng vấn đồng nghĩa là lost dự án, ảnh hưởng đến business của công ty. Theo chỉ thị của lãnh đạo, các ban bệ vào cuộc, toàn bộ record của phỏng vấn được nghe lại, mổ xẻ như phân tích băng ghi hình trận đá bóng. Câu hỏi phỏng vấn được Google search, gửi tới các chuyên gia để được tư vấn. Thậm chí còn được post lên diễn đàn công nghệ để tham khảo (bà con cũng nhảy vô cãi nhau ỏm tỏi, mỗi người 1 đáp án, gạch đá ném đủ xây nhà). Bản thân mình cũng phải cố gắng tìm hiểu Google, tham khảo ý kiến của nhiều người để tìm câu trả lời đúng nhất.

Sau hàng loạt các phân tích bình luận của các chuyên gia thì đã tìm ra câu trả lời có thể nói là chính xác nhất.

Câu 1) Viết code ví dụ về Singleton class

Thật ra có đến mấy kiểu viết code về Singleton class trong Java trong đó có 2 kiểu thông dụng là:

Early Instantiation: Tạo instance tại thời điểm class loade time.

Kiểu Lazy Loading: Instance được tạo khi Singleton class được dùng lần đầu tiên, như code đã trình bày ở phần 1

Cần phải trình bày cả trường hợp đa luồng như phần 1 nữa. Túm váy phải phân tích từng trường hợp, đánh giá ưu nhược điểm

Và 1 Singleton có thể tạm gọi là chuẩn man là như sau:

Quan trọng nhất là câu hỏi này, quyết định việc fail hay pass

Có trường hợp nào Singleton có tạo nhiều instance không?

Câu trả lời đúng ở đây là YES.

Java có một khái niệm là Reflection, API cho phép kiểm tra và modify behavior của methods, class at runtime.

Reflection này có thể gây ra việc destroy singleton property của singleton class.

Dẫn tới việc singleton bị destroy và tạo 2 object khác nhau thuộc sample class
Output:-
instance1.hashCode():- 366712642
instance2.hashCode():- 1829164700
Ngoài ra còn có trường hợp serialize/desterilize object cũng xảy ra:

Trích dẫn nguyên văn câu trong Head First Design Patterns: Be careful if you are using multiple class loaders; this could defeat Singleton implementation and result in multiple instances.
Câu 2: Coding
Viết chuỗi đảo ngược kí tự
Ngoài việc dùng vòng lặp đảo vị trí phần tử như phần 1, thì có thể sử dụng thuật toán đệ quy.

Code đơn giản hơn không phải lặp, nhưng mình không nghĩ đệ quy là tối ưu nhất vì trong đệ quy hàm sẽ được gọi nhiều lần và bộ nhớ stack sẽ bị chồng thêm dữ liệu dễ gây issue về bộ nhớ.

Bài code số 2:

Cho số dương integer n, tìm số cách để để chia n thành 5 số dương nhỏ hơn sao cho cộng lại thì vẫn là n

Ví dụ n = 5 thì output 1 vì chỉ có 1 case (1, 1, 1, 1, 1)

N = 12 thì có 7 cách

Nhờ sự trợ giúp của anh Guc Gồ thì đã phát hiện ra đằng sau bài toán này là cả 1 bầu trời kiến thức về toán học

https://www.mathpages.com/home/kmath556/kmath556.htm

Đọc xong cảm thấy ngu người luôn.

Đại loại công thức để tìm số cách thế này. N là số dương cần chia, K là số lượng số dương < N

Tạm tóm lược thay kiểu nông dân như thế này:

Phát biểu tổng quát: cho n viên kẹo chia cho k em bé, hỏi có bao nhiêu cách? (n, k nguyên dương)
Tóm tắt cách giải:

• Trải đều n viên kẹo ra bàn, nhận thấy có n – 1 chỗ trống giữa các viên kẹo
• Ta có k em bé -> cần nhét k-1 vách ngăn để ra k khoảng trống -> đáp án là tổ hợp chập (k-1) của (n-1), hay (n-1)C(k-1)

N = 7, k = 5 theo cách tính tổ hợp chập thì có 15 cách cụ thể là:

{2, 2, 1, 1, 1}
{2, 1, 2, 1, 1}
{2, 1, 1, 2, 1}
{2, 1, 1, 1, 2}

{1, 2, 2, 1, 1}
{1, 2, 1, 2, 1}
{1, 2, 1, 1, 2}

{1, 1, 2, 2, 1}
{1, 1, 2, 1, 2}

{1, 1, 1, 2, 2}
(Thực ra chỉ có 1 cách là 2 số 2 và 3 số 1)

{3, 1, 1, 1, 1}
{1, 3, 1, 1, 1}
{1, 1, 3, 1, 1}
{1, 1, 1, 3, 1}
{1, 1, 1, 1, 3

(Thực ra chỉ có 1 cách là 1 số 3 và 4 số 1)

Tuy nhiên do bài toán ko care đến order nên cần thêm bước loại trừ những kết quả trùng nhau nữa.

Câu 3: Câu hỏi về vấn đề hệ thống read data 99%, write 1% thì nên chọn loại dữ liệu gì?

Câu trả lời là ArrayList như phần 1, nhưng trường hợp đa luồng thì vẫn có thể dùng ArrayList nhưng không share list này giữa các thread. Tóm lại là handle with care, hàng dễ vỡ xài cẩn thận.

Hoặc 1 câu trả lời khác như sau (Xin tư vấn của 1 cao thủ Java).

Nếu là môi trường single-thread thì dùng ArrayList tốt hơn LinkedList trong trường hợp ở đây (nhất là khi list có kích thước tương đối lớn) là vì ArrayList là random access trong khi LinkedList thì không.

Đổi lại thì thêm, remove phần tử trên ArrayList sẽ kém hơn về performance so với LinkedList (do phải xử lý cái backed array). Tất nhiên là performance của hệ thống thì ko thể dựa hoàn toàn trên tính toán định tính, mà cần dựa trên các công cụ monitor với con số cụ thể.

Nếu là môi trường multi-thread thì nên dùng CopyOnWriteArrayList, lý do là nó đảm bảo thread-safe trong khi implement của hàm read() trong mô hình CopyOnWrite nó ko đòi hỏi phải sử dụng lock.

Tuy nhiên, câu hỏi thế này thì nên hỏi ngược lại vì nó thiếu thông tin cần thiết để có đáp án chính xác.Dẫn chứng nguyên văn của một tiền bối: Read là read kiểu gì, write là write như thế nào, vặn lại luôn là đã có điểm cộng rồi. Đề bài cho thế ko thể đủ dữ kiện phán đoán được. Ví dụ random read thì array hơn, nhưng sequential read như nhau. Write cũng vậy, nếu write vào cuối thì array là nhanh nhất. Chém gió nâng cao thì nếu lượng phần tử cực lớn và sequential read thì array sẽ có ưu thế rõ ràng hơn do memory được allocate contiguous chứ không bị fragment như linked list. Do đó nó fit vào cùng cache line, vì thời buổi này bottle neck chủ yếu ở memory, cứ fragment xong miss cache một phát là bay vài trăm cycle ngay.Câu 4: Anh có service expose API cho client, nếu anh dùng GET api call service qua HTTPS chú thấy có secure không?Trong trường hợp sensitive data phải include trong GET request, nó sẽ vi phạm security best practices. Full URL query parameters (bao gồm query parameters) của GET requess sẽ bị logged trong server logs và có nguy cơ được truy cập trái phép. Trong trường hợp data on the query parameters không nên xuất hiện trong logs, bạn nên sử dụng POST thay cho GET.

OK, vậy là sau collect đủ các thông tin cần thiết, team đã build thành 1 bộ đề phỏng vấn bao gồm câu hỏi và đáp án. Y chang kiểu bộ đề tuyển sinh đại học.

Công việc tiếp theo là trao bộ đề lại cho 1 thanh niên thứ 3 đi phỏng vấn, vào cầu cho ông cụ hỏi đúng câu cũ.

Ứng viên thứ 3 có English khá tốt, bộ đề, đáp án đã được bàn giao và đọc trước. Khác với lần phỏng vấn trước, lần này có đầy đủ ban bệ lãnh đạo có mặt. Vì xác định lần này tạch thì coi như mất dự án, anh em treo niêu.

Anh e đã làm hết sức mình, còn lại trông chờ vào kĩ năng chém gió của candidate. Nghe nói em manager tối còn phát đốt hương khấn vái.

Đồng hồ dịch chuyển dần đến giờ interview 9:30, mọi người hồi hộp chờ đợi…

Lại bác Director of Software engineering xuất hiện, nụ cười hiền từ.

Câu hỏi thứ nhất

Câu 1) Viết code ví dụ về Singleton class

Câu 2) Viết code đảo chiều chuỗi…

Câu 3)….

Trời. Đúng là kiếp trước ăn ở tốt, trời phật phù hộ độ trì, các câu hỏi y chang. Kết quả thế nào đã rõ…

Cảm xúc vỡ òa, đúng là bạc già không bằng gà son. Lần này đã bắt bài được cách interview, cụ không lừa em được nữa nhá

Nhưng giờ vẫn nghĩ không thông được vì sao cụ lại dùng chung 1 bộ câu hỏi để interview?

Kết luận

Các cụ ngày xưa đã nói rằng “Người thất bại là người bỏ cuộc”, người chiến thắng là người không từ bỏ. Đôi khi thành công đến được còn cần 1 chút may mắn. Nhưng may mắn chỉ có được khi bạn đã cố gắng hết sức của mình. Thành công của người này có được có thể phải đánh đổi bằng sự đóng góp, hi sinh của tập thể, nhưng đều dành cho mục đích chung.

Đỗ Trọng Nguyên

Scaling Your Web App 101: Lessons in Architecture Under Load

It’s the classic champagne problem that most successful web apps will deal with – there are so many users on your site that things are starting to get bogged down.

Pages load slowly, network connections start timing out and your servers are starting to creak under heavy load. Congratulations – your web app has hit scale!

But now what? You need to keep everything online and want the user’s experience to be fast – speed is a feature after all.

Scaling Comes at a Price

But before we go any further, an important caveat – you shouldn’t attempt to “scale” your web app before you’ve actually run into real scaling problems.

While it may be fun to read about Facebook’s architecture on their engineering blog, it can be disastrous to think that their solutions apply to your fledgling project.

A lot of the solutions to common scaling bottleneck introduce complexity, abstraction and indirection which makes systems more difficult to reason about. This can create all sorts of problems:

• Adding new features takes longer
• Code can be harder to test
• Finding and fixing bugs is more frustrating
• Getting local and production environments to match is more difficult

You should only be willing to accept these tradeoffs if your app is actually at the limits of what it can handle. Don’t introduce complexity until it’s warranted.

As the famous quote goes:

Premature optimization is the root of all evil.
— Donald Knuth

Find the Actual Bottleneck using Metrics

The first step to alleviating any problem – in software or otherwise – is to clearly and accurately define what the problem actually is.

A problem well stated is a problem half-solved.
— Charles Kettering

For a web app that’s under too much load, that means finding out what resource your application is running out of on the server.

At a high level, the answer is usually going to be one of four things:

1. Memory
2. CPU
3. Network I/O
4. Disk I/O

Until you figure out what resource your application is bounded by, no one can help you scale your app and any solutions you come up with will be complete guesses.

Figuring out what you’re bounded by means checking your resource monitoring – or adding some if you’ve never done it before.

What gets measured, gets managed
— Peter Drucker

If you’re managing your own servers, installing Munin is a great first step. If you’re running on Amazon’s EC2, AWS offers some decent instance monitoring out of the box. If you’re on Heroku, New Relic seems to be the best approach.

Use the graphs to look for spikes or flat tops. These usually imply that some resource was overwhelmed or completely at capacity and couldn’t handle any new operations.

If you don’t see any resources that seem to be at capacity, but your app is just slow in general, sprinkle some logging throughout heavily-used operations and check the logs to see if there’s some resource that’s taking a long time to load over the network.

It could be that another server is introducing delays – potentially your database server or a third-party API.

If you host your database on a different machine than your web servers (which you should) it’s important to check your resource monitoring for that machine as well as for your web servers.

The database is usually the first place scaling issues start to show up.

Scaling a Web App from 10,000 Feet

Now that you’ve got a much better sense of what the problem is, you should start to tackle it by trying the simplest solution that directly addresses the issues – remember, we’re always trying to avoid adding unnecessary complexity.

At a high level, the goal of any scaling solutions should be to make your web stack do less work for the most common requests.

If you’ve already figured out the answer to a query, reuse it. Or if you can avoid computing it or looking up all together, even better.

In a tangible sense, this usually means one of the following:

• Store results of common operations so you're not repeating work
• Reuse data you've already looked up, even if it's a bit stale
• Avoid doing complex operations in the request-response cycle
• Don't make requests from the client for things it already has

These all basically boil down to some form of caching.

Memory is not only inexpensive to add to a server, it’s usually many orders of magnitude faster for accessing data when compared to disk or the network.

Hosting Topology

Whether your application is hosted in the cloud or on hardware, some part of your stack will inevitably fail. You should host and arrange your web servers to take this into account.

Your domain should point to some sort of load balancer, which should then route requests between two or more web servers.

Not only does this setup make it easy to survive failures, it also makes handling increased load easier as well.

With a load balancer in front of two web servers, you can horizontally scale your application by bring up new web servers and putting them behind the load balancer. Now the requests are spread across more machines, meaning each one is doing less work overall.

This allows you to grow your application gracefully over time, as well as handle temporary surges of traffic.

I should also add that setting up a load balancer and two web servers is a one-time setup that doesn’t add much on-going complexity, so it’s something you should consider doing up-front, even before you’ve run into scaling problems.

Cache Database Queries

This is one of the simplest improvements you can make. There’s usually a few common queries that make up the majority of load on your database.

Most databases support query logging, and there are many tools that will ingest those logs and run some analysis to tell you what queries are run most frequently, and what queries tend to take the longest to complete.

Simply cache the responses to frequent or slow queries so they live in memory on the web server and don’t require a round-trip over the network or any extra load on the database.

Obviously, data that’s cached can grow “stale” or out-of-date quickly if the underlying information in the database is updated frequently. Your business or product requirements will dictate what can or can’t be cached.

Database Indexes

Database indexes ensure that needle-in-a-haystack type lookups are O(1) instead of O(n).

In layman’s terms, this means the database can find the right row immediately, rather than having to compare the queried conditions against every single row in the table.

If your table has tens of thousands of rows, this could shave noticeable amount of time off of any queries that use that column.

As a very simple example, if your application has profile pages that look up a user by their handle or username, an un-indexed query would examine every single row in the users table, looking for the ones where the “handle” column matched the handle in the URL.

By simply adding an index to that table for the “handle” column, the database could pull out that row immediately without requiring a full table scan.

Session Storage

A lot of applications handle sessions by storing a session ID in a cookie, and then storing the actual key/value data for each and every session in a database table.

If you find your database is getting slammed and your application does a lot of reading and writing to session data, it might be smart to rethink how and where you store your session data.

One option is to move your session storage to a faster, in-memory caching tool like redis or memcached.

Since these use volatile memory rather than persistent disk storage (which most databases use) they’re usually much faster to access – but the tradeoff is that you run the risk of losing all of your session data if the caching system needs to reboot or go offline.

Another option is to move the session information into the cookie itself. This obviously leaves it open to being tampered with by the user, so it shouldn’t be used if you’re storing anything private or sensitive in the session.

By moving session data out of the database, you’ll likely eliminate several database queries per page load, which can help your database’s performance tremendously.

Run Computations Offline

If you have some long-running queries or complex business logic that takes several seconds to run, you probably shouldn’t be running it in the request-response cycle during a page load.

Instead, make it “offline” and have a pool of workers that can chug away at it and put the results in a database or in-memory cache.

Then when the page loads, your web server can simply and quickly pull the precomputed data out of the cache and show it to the user.

A drawback here is that the data you’re showing the user is no longer “real time,” but having data that’s a few minutes old is often good enough for many use-cases.

If the data really takes a long time to generate, see if it can be parallelized so that multiple workers can work on different parts of the computation at the same time.

You’ll probably want to setup another cluster of machine for the work queue and the workers, since those will likely have different scaling properties than your web servers.

To take this architectural style to its logical conclusion, you can generate the HTML for your entire web app offline and simply serve it to users as static files.

This is the inspiration behind static site generators that are used to power a growing number of blogs (including this one), and it’s what the New York Times did to serve election night results.

HTML Fragment Caching

If you’re rendering HTML templates on the server-side, you want to avoid having your template engine wasting CPU cycles on every request generating the same HTML over and over again for content that doesn’t change often.

If there are certain sections of your site’s markup that change very infrequently – say the navigation, footer or sidebar – then that HTML should be cached somewhere and reused between requests.

Pay special attention to high-traffic pages. Sometimes you’ll be able to cache most of the page except for a few more dynamic or “real time” sections.

Putting Work Into Queues

We talked about using queues and workers for caching output that takes a long time to generate. You can also use workers for processing large amount of input asynchronously.

This has the effect of taking large, slow chunks of work and breaking them out from the main request-response cycle and taking it completely off your web servers.

Say you have a way for someone to import a CSV of their contacts and several people upload 50MB files. Instead of sending all of that data to a web server and having it take up memory and block the CPU while it’s being processed – put it on a static file host like s3 and have a worker that periodically checks for new file uploads and processes them offline.

You do have to be careful when you start putting lots of business logic into workers. You have to make sure you’re keeping track of what still needs to be processed, what is currently being processed, and what failed and needs to be processed again.

You also need to make sure you have enough workers running, otherwise the work queue will grow longer and longer, leading to silent delays that are easy to miss.

Client Side Improvements

Of course, another great way to decrease the load on your web servers is to decrease the number of requests they have to deal with.

Even with the same number of users in your app, there are a number of client-side improvements that can lower the number of requests your web stack has to deal with.

HTTP Caching
Just like you want to cache database queries to avoid regenerating answers you already know, you should avoid having the browser ask for content that it has already downloaded.

You should use HTTP caching headers for all of your static files – CSS, javascript and images.

Google and Mobify provide great overviews on how to use the headers, and your web framework will likely have some helpers to make it even easier.

Content Delivery Network
Ideally, your web servers wouldn’t serve any static content at all. You don’t need the overhead of loading your entire web framework or language runtime in order to serve a static file off of disk.

You should host your static content on a static file host that’s purpose built for sending files over the network. You can setup a simple one with nginx or use a dedicated service like Amazon’s S3.

When you’re splitting out your static content, it’s good to give the static file host a different CNAME or subdomain.

Once you’ve gotten that setup, it’s usually pretty straightforward to add a Content Delivery Network in front of your static file host.

This distributes your static content even faster – often without the requests ever reaching your web stack.

The CDN is essentially a geographically-distributed file cache that serves up copies of your static files that are often geographically closer to the end-user than your server.

The CDN will often have options to minify and gzip your static content to make the files even smaller and faster to send to the client.

When All Else Fails

If you’re still having issues with high load and you’ve cached as much as you can and your server budget is maxed out, there are still some less-than-ideal options you have to handle excess load.

Back Pressure
Back pressure is just a way of telling people they have to wait because the system is slow. If you’ve ever gone tried going to a cafe, saw a line out the door and decided to go elsewhere – that’s a great example of back pressure.

Even without much work on your end, back pressure can be implicit. The site will be slow for users, which will discourage them from clicking around as much.

They also might see increased error rates when they try to load things – think Twitter’s Fail Whale.

You can also make back pressure explicit by adding messaging to the UI telling people that parts of your applications are temporarily disabled due to high demand.

Shed Load
The other option is to shed load. You’re acknowledging that you’re not able to respond to everyone’s requests, so you’re not even going to try.

This is the nuclear option. Set aggressive timeout in your web server and let requests hang and return blank pages. Some people’s data will get lost in cyberspace.

Architecture of the scalability evolution - Single Server to a Global Audience

As a young engineer I used to build web applications that were hosted on a single server, and this is probably how most of us get started. During my career I have worked for different companies and I have witnessed applications in different scalability evolution stages. Before we go deeper into scalability, I would like to present some of these evolution stages to better explain how you go from a single server sitting under your desk to thousands of servers spread all over the world. I will keep it at a very high level here, as I will go into more detail in later chapters. Discussing evolution stages will also allow me to introduce different concepts and gradually move toward more complex topics. Keep in mind that many of the scalability evolution stages presented here can only work if you plan for them from the beginning. In most cases, a real-world system would not evolve exactly in this way, as it would likely need to be rewritten a couple of times. Most of the time, a system is designed and born in a particular evolution stage and remains in it for its lifetime, or manages to move up one or two steps on the ladder before reaching its architectural limits.

Single-Server Configuration 

Let’s begin with a single-server setup, as it is the simplest configuration possible and this is how many small projects get started. In this scenario, I assume that your entire application runs on a single machine. Figure 1-1 shows how all the traffic for every user request is handled by the same server. Usually, the Domain Name System (DNS) server is used as a paid service provided by the hosting company and is not running on your own server. In this scenario, users connect to the DNS to obtain the Internet Protocol (IP) address of the server where your website is hosted. Once the IP address is obtained, they send Hypertext Transfer Protocol (HTTP) requests directly to your web server. Since your setup consists of only one machine, it needs to perform all the duties necessary to make your application run. It may have a database management system running (like MySQL or Postgres), as well as serving images and dynamic content from within your application.

Figure 1-1 shows the distribution of traffic in a single-server configuration. Clients would first connect to the DNS server to resolve the IP address of your domain, and then they would start requesting multiple resources from your web server. Any web pages, images, Cascading Style Sheet (CSS) files, and videos have to be generated or served by your server, and all of the traffic and processing will have to be handled by your single machine. I use different weights of arrows on the diagram to indicate the proportion of traffic coming to each component. An application like this would be typical of a simple company website with a product catalog, a blog, a forum, or a self-service web application. Small websites may not even need a dedicated server and can often be hosted on a virtual private server (VPS) or on shared hosting

For sites with low traffic, a single-server configuration may be enough to handle the requests made by clients. There are many reasons, though, why this configuration is not going to take you far scalability-wise:
 ▶ Your user base grows, thereby increasing traffic. Each user creates additional load on the servers, and serving each user consumes more resources, including memory, CPU time, and disk input/output (I/O).
▶ Your database grows as you continue to add more data. As this happens, your database queries begin to slow down due to the extra CPU, memory, and I/O requirements.
▶ You extend your system by adding new functionality, which makes user interactions require more system resources.
▶ You experience any combination of these factors.

Making the Server Stronger: Scaling Vertically
There are a number of ways to scale vertically:
▶ Adding more I/O capacity by adding more hard drives in Redundant Array of Independent Disks (RAID) arrays. I/O throughput and disk saturation are the main bottlenecks in database servers. Adding more drives and setting up a RAID array can help to distribute reads and writes across more devices. In recent years, RAID 10 has become especially popular, as it gives both redundancy and increased throughput. From an application perspective, a RAID array looks like a single volume, but underneath it is a collection of drives sharing the reads and writes.
▶ Improving I/O access times by switching to solid-state drives (SSDs). Solid-state drives are becoming more and more popular as the technology matures and prices continue to fall. Random reads and writes using SSDs are between 10 and 100 times faster, depending on benchmark methodology. By replacing disks you can decrease I/O wait times in your application. Unfortunately, sequential reads and writes are not much faster and you will not see such a massive performance increase in real-world applications. In fact, most open-source databases (like MySQL) optimize data structures and algorithms to allow more sequential disk operations rather than depending on random access I/O. Some data stores, such as Cassandra, go even further, using solely sequential I/O for all writes and most reads, making SSD even less attractive.
▶ Reducing I/O operations by increasing RAM. (Even 128GB RAM is affordable nowadays if you are hosting your application on your own dedicated hardware.) Adding more memory means more space for the file system cache and more working memory for the applications. Memory size is especially important for efficiency of database servers.
▶ Improving network throughput by upgrading network interfaces or installing additional ones. If your server is streaming a lot of video/media content, you may need to upgrade your network provider’s connection or even upgrade your network adapters to allow greater throughput.
▶ Switching to servers with more processors or more virtual cores. Servers with 12 and even 24 threads (virtual cores) are affordable enough to be a reasonable scaling option. The more CPUs and virtual cores, the more processes that can be executing at the same time. Your system becomes faster, not only because processes do not have to share the CPU, but also because the operating system will have to perform fewer context switches to execute multiple processes on the same core.

Figure 1-2 shows the approximate relationship of price per capacity unit and the total capacity needed. It shows that you can scale up relatively cheaply first, but beyond a certain point, adding more capacity becomes extremely expensive. For example, getting 128GB of RAM (as of this writing) could cost you $3,000, but doubling that to 256GB could cost you $18,000, which is much more than double the 128GB price.


Isolation of Services
Vertical scalability is not the only option at this early stage of evolution. Another simple solution is moving different parts of the system to separate physical servers by installing each type of service on a separate physical machine. In this context, a service is an application like a web server (for example, Apache) or a database engine (for example, MySQL). This gives your web server and your database a separate, dedicated machine. In the same manner, you can deploy other services like File Transfer Protocol (FTP), DNS, cache, and others, each on a dedicated physical machine.

Cache is a server/service focused on reducing the latency and resources needed to generate the result by serving previously generated content. Caching is a very important technique for scalability.

Content Delivery Network: Scalability for Static Content
As applications grow and get more customers, it becomes beneficial to offload some of the traffic to a third-party content delivery network (CDN) service.

A content delivery network is a hosted service that takes care of global distribution of static files like images, JavaScript, CSS, and videos. It works as an HTTP proxy. Clients that need to download images, JavaScript, CSS, or videos connect to one of the servers owned by the CDN provider instead of your servers. If the CDN server does not have the requested content yet, it asks your server for it and caches it from then on. Once the file is cached by the CDN, subsequent clients are served without contacting your servers at all.

Distributing the Traffic: Horizontal Scalability 

All of the evolution stages discussed so far were rather simple modifications to the single-server configuration. Horizontal scalability, on the other hand, is much harder to achieve and in most cases it has to be considered before the application is built. In some rare cases, it can be “added” later on by modifying the architecture of the application, but it usually requires significant development effort. I will describe different horizontal scalability techniques throughout this book, but for now, let’s think of it as running each component on multiple servers and being able to add more servers whenever necessary. Systems that are truly horizontally scalable do not need strong servers—quite the opposite; they usually run on lots and lots of cheap “commodity” servers rather than a few powerful machines.

Scalability for a Global Audience

The largest of websites reach the final evolution stage, which is scalability for a global audience. Once you serve millions of users spread across the globe, you will require more than a single data center. A single data center can host plenty of servers, but it causes clients located on other continents to receive a degraded user experience. 

GeoDNS is a DNS service that allows domain names to be resolved to IP addresses based on the location of the customer. Regular DNS servers receive a domain name, like yahoo.com, and resolve it to an IP address, like 206.190.36.45. GeoDNS behaves the same way from the client’s perspective.

Edge cache is a HTTP cache server located near the customer, allowing the customer to partially cache the HTTP traffic. Requests from the customer’s browser go to the edge-cache server. The server can then decide to serve the page from the cache, or it can decide to assemble the missing pieces of the page by sending background requests to your web servers. It can also decide that the page is uncacheable and delegate fully to your web servers. Edge-cache servers can serve entire pages or cache fragments of HTTP responses. 

A load balancer is a software or hardware component that distributes traffic coming to a single IP address over multiple servers, which are hidden behind the load balancer. Load balancers are used to share the load evenly among multiple servers and to allow dynamic addition and removal of machines. Since clients can only see the load balancer, web servers can be added at any time without service disruption.

The second layer of our stack is the web application layer. It consists of web application servers (4) responsible for generating the actual HTML of our web application and handling clients’ HTTP requests. These machines would often use a lightweight (PHP, Java, Ruby, Groovy, etc.) web framework with a minimal amount of business logic, since the main responsibility of these servers is to render the user interface. All the web application layer is supposed to do is handle the user interactions and translate them to internal web services calls. The simpler and “dumber” the web application layer, the better. By pushing most of your business logic to web services, you allow more reuse and reduce the number of changes needed, since the presentation layer is the one that changes most often.

The third layer of our stack consists of web services (7). It is a critical layer, as it contains most of our application logic. We keep front-end servers simple and free of business logic since we want to decouple the presentation layer from the business logic. By creating web services, we also make it easier to create functional partitions. We can create web services specializing in certain functionality and scale them independently. For example, in an e-commerce web application, you could have a product catalog service and a user profile service, each providing very different types of functionality and each having very different scalability needs.

Both front-end servers (4) and web services (7) should be stateless, web applications often deploy additional components, such as object caches (5) and message queues (6). Object cache servers are used by both front-end application servers and web services to reduce the load put on the data stores and speed up responses by storing partially precomputed results. 

Data persistence layer (8) and (9). This is usually the most difficult layer to scale horizontally, so we’ll spend a lot of time discussing different scaling strategies and horizontal scalability options in that layer. This is also an area of rapid development of new technologies labeled as big data and NoSQL, 

The data layer has become increasingly more exciting in the past ten years, and the days of a single monolithic SQL database are gone. As Martin Fowler says, it is an era of polyglot persistence, where multiple data stores are used by the same company to leverage their unique benefits and to allow better scalability. 

Overview of the Application Architecture

So far, we’ve looked at the infrastructure and scalability evolution stages. Let’s now take a high-level look at the application itself. The application architecture should not revolve around a framework or any particular technology. Architecture is not about Java, PHP, PostgreSQL, or even database schema. Architecture should evolve around the business model. There are some great books written on domain-driven design and software architecture1–3 that can help you get familiar with best practices of software design. To follow these best practices, we put business logic in the center of our architecture. It is the business requirements that drive every other decision. Without the right model and the right business logic, our databases, message queues, and web frameworks are useless. Moreover, it is irrelevant if the application is a social networking website, a pharmaceutical service, or a gambling app—it will always have some business needs and a domain model. By putting that model in the center of our architecture, we make sure that other components surrounding it serve the business, not the other way around. By placing technology first, we may get a great Rails application, but it may not be a great pharmaceutical application.t1

A domain model is created to represent the core functionality of the application in the words of business people, not technical people. The domain model explains key terms, actors, and operations, without caring about technical implementation. The domain model of an automated teller machine (ATM) would mention things like cash, account, debit, credit, authentication, security policies, etc. At the same time, the domain model would be oblivious to hardware and software implementation of the problem. The domain model is a tool to create our mental picture of the business problems that our application is supposed to solve

The front end should have a single responsibility of becoming the user interface. The user can be interacting with the application via web pages, mobile applications, or web service calls. No matter what the actual delivery mechanism is, the front-end application should be the layer translating between the public interface and internal service calls. The front end should be considered as “skin,” or a plugin of the application, and as something used to present the functionality of the system to customers. It should not be considered a heart or the center of the system

Front-end code will be closely coupled to templates and the web framework of our choice (for example, Spring, Rails, Symfony). It will be constrained by the user interface, user experience requirements, and the web technologies used. Frontend applications will have to be developed in a way that will allow communication over HTTP, including AJAX and web sessions. By hiding that within the front-end layer, we can keep our services layer simpler and focused solely on the business logic, not on the presentation and web-specific technologies. Templating, web flows, and AJAX are all specific problems. Keeping them separated from your main business logic allows for fast and independent changes. Having the front end developed as a separate application within our system gives us another advantage: we can use a different technology stack to develop it. It is not unreasonable to use one technology to develop web services and a different one to develop the front-end application. 

As an example, you could develop the front end using Groovy, PHP, or Ruby, and web services could be developed in pure Java.

Web services are where most of the processing has to happen, and also the place where most of the business logic should live. 

Amazon deployment example