Scaling

Scaling and Performance

• Real world example: cafeteria flow chart
• Optimization:the search for bottlenecks.
• What’s a bottleneck? Refer back to the cafeteria example.
• Moving target. * When you eliminate/improve one bottleneck, it just reveals the next one. * You make starting the dashboard activity faster….so that now you can notice that drawing the map overlay is slow.
• Important: Measurement
• Worse sin: Optimizing early. Why?

Server Push

• Review architecture and why
• Look at code

Performance

• Performance is what what a user experiences as “slow” or “fast”
• Response time to an operation initiated by the user
• Perception!
  • Can you ‘fool’ the user into thinking the app is faster than it is?
  • Feedback: spinners etc
  • Anticipation: start doing work before user requests it
• Different (but intertwined with) scaling

Scaling

• Image above is likely a scaling problem

• “How many X per minute can you do”
  • (e.g. user log ins, page refreshes, notifications,…
• How many (users, sessions, videos, pictures, etc) does the site need to support
• Different from response time: “How long does it take to accomplish Y?” Related but different
• Scaling has to do with the load on the servers
• Big challenge: how fast or slow will the site or app grow?
• Architectural techniques apply equally
  • scaling up vs. scaling out
  • caching
  • load balancing
  • database partitioning and sharding
  • asynchronous processing

Patterns of scaling problems and solutions

• “Clients” = web browsers accessing the site, mobile apps accessing the site, etc.
• Load on the servers. Some scenarios, one or more of:
  • Too many clients asking the server to do operation O * Individual clients asking the server to do operation P too often
  • Operation Q is time consuming for the server to satisfy
• Solutions can be
  • Add an identical server to handle operations O, P or Q
  • Send operation O to one server and operation P to another server
  • Why are so many clients asking for O? Can we reduce the number?
  • What’s the reason why a client would ask for operation P so often? Can we reduce that?
  • Is there a way to make operation Q faster to satisfy?

Techniques: Caching

• Save the result of a request with a given set of parameters.
• In a future request with the same parameter (maybe) return the same result
• Memoization:
  • caching applied to an individual method
  • A basic programming technique
• System level caching. Storage:
  • In ‘local’ memory
  • In ‘remote’ memory
  • In database
  • In Cloud
• Name-value databases
  • Very fast searches and lookups
  • Distributed searches and distributed databases
  • Robust across system and application failures
• References:
  • Redis
  • Memcached
  • Amazon Elastic Cache

Techniques: Asycnhronous processing

• Real examples: Discussion: What do all of these have in common?
  • Account registraton confirmation emails (actually all emails)
  • Daily (periodic or episodic) notification emails
  • Automatic backups or archiving
  • Image Resizing
  • Spam checking
• Synchronous: Caller waits for response

• Asynchronous: Request returns immediately, but result comes later

• NB: web server mainly responds to http requests!
• Background processing happens even if no requests!
• How do processes work in a modern Operating System?
  • Scheduler part of the OS
  • Processes can be fairly heavy weight
  • Let’s estimate how many background processs you would need
  • How would you handle processes that:
    • Were taking too long?
    • Had crashed?
    • Needed to be restarted?
    • Or what if the server crashes as a whole?
    • It’s a mess. Need abstractions!

Case 1: Schedule based

• CRON jobs. Most elementary. Require “privileges”
• Alternative: Native “scheduler” application (e.g. Heroku Simple Job Scheduler)
• References:
  • Heroku Simple Job Scheduler
  • Unix Cron
  • Creating Cron Jobs on Mac
  • Rails Runner

Case 2: Request based

• Request handled ‘asynchronously’ of course
• By local server, in a separate process
• By a remote server
• Request queueing
  • What happens when requests come faster than we can handle them
  • How do we add more performance in handling request
• References
  • Comparing Background processing Libraries
  • Rails Delayed Job Gem

Techniques: Databases

Discussion: What if all the user names are stored in one table on one server?

• Databases are a major bottleneck! Must move the data to more than one machine!
• Need to undertand the access pattern: read intensive/write intensive
• Database Replication
  • What does it do
  • What are the benefits
• Database Partitioning
  • What are the different kinds: Horizontal and Vertical
  • What does it do
  • What are the benefits
• Database Sharding
  • What does it do
  • What are the benefits
• References
  • The Basics Of Database Sharding