449 lines
11 KiB
Org Mode
449 lines
11 KiB
Org Mode
* 1. Concurrency
|
||
|
||
- David Wen Riccardi-Zhu
|
||
- Senior Software Engineer @ Good Uncle
|
||
- dwrz@dwrz.net
|
||
|
||
Slides and code: https://github.com/dwrz/talks/concurrency.
|
||
|
||
* 2. Why
|
||
|
||
You'll likely encounter it at work:
|
||
|
||
- Client requests to your backend
|
||
- Database and API requests made by your backend
|
||
- Frontend rendering
|
||
|
||
* 3. Why
|
||
|
||
Performance Benefits
|
||
|
||
- Typically outweigh gains from optimized data structures and algorithms.
|
||
- I/O versus CPU bound applications.
|
||
|
||
- Hardware limitations.
|
||
- Rate of single core performance improvement is slowing.
|
||
- More cores.
|
||
|
||
* 4. Why
|
||
|
||
Expressive
|
||
|
||
* 5. Why
|
||
|
||
Difficult to do correctly.
|
||
|
||
Easy to do incorrectly.
|
||
|
||
* 6. Goals
|
||
|
||
- Give you a rough sense of what concurrency is.
|
||
- Provide context to help you understand how it's made possible.
|
||
- Practical demonstration, to expose some of the trade-offs.
|
||
|
||
* 7. Three Parts
|
||
|
||
1. Fundamentals
|
||
2. Application
|
||
3. Infrastructure
|
||
|
||
* 8. Definition
|
||
|
||
/Concurrency/
|
||
1. A running together in place or time.
|
||
2. Simultaneous occurrence.
|
||
|
||
OK starting point, but:
|
||
|
||
Time, and simultaneity, is /complicated/.
|
||
|
||
A second is a duration, not an instant.
|
||
|
||
Two things can happen in the same second and not be simultaneous.
|
||
|
||
In computer science, this nuance matters.
|
||
|
||
* 9. Time
|
||
|
||
|----------------------+----------------+----------------|
|
||
| System Event | Actual Latency | Scaled Latency |
|
||
|----------------------+----------------+----------------|
|
||
| One CPU cycle | 0.4 ns | 1 s |
|
||
| Level 1 cache access | 0.9 ns | 2 s |
|
||
| Level 2 cache access | 2.8 ns | 7 s |
|
||
| Level 3 cache access | 28 ns | 1 min |
|
||
| Main memory access | ~100 ns | 4 min |
|
||
| NVMe SSD I/O | ~25 μs | 17 hrs |
|
||
| SSD I/O | 50–150 μs | 1.5–4 days |
|
||
| Rotational disk I/O | 1–10 ms | 1–9 months |
|
||
| Internet: SF to NYC | 65 ms | 5 years |
|
||
| Internet: SF to HK | 141 ms | 11 years |
|
||
|----------------------+----------------+----------------|
|
||
|
||
Source: [[https://www.prowesscorp.com/computer-latency-at-a-human-scale/]]
|
||
|
||
* 10. Example
|
||
|
||
- Get the current temperature of ten most populous cities in the US.
|
||
|
||
- First, /sequentially./
|
||
- Then, /concurrently/.
|
||
|
||
[[file:cmd/temperatures/run.sh]]
|
||
|
||
* 11. Example
|
||
|
||
[[file:images/concurrent-requests.jpg]]
|
||
|
||
* 12. Cooking Recipes
|
||
|
||
- Step-by-step
|
||
- Concurrent
|
||
- Pre-heat oven
|
||
- Boil water
|
||
- Timers
|
||
|
||
* 13. Multitasking
|
||
|
||
Computers can "simultaneously" work on:
|
||
|
||
- Different /parts/ of a recipe
|
||
- Different /recipes/
|
||
|
||
* 14. htop
|
||
|
||
* 15. How?
|
||
|
||
Combination of:
|
||
- Operating System *Scheduler*
|
||
- Hardware (/interrupts/)
|
||
|
||
* 17. Cooperative Scheduling
|
||
[[file:images/cooperative-scheduling.jpg]]
|
||
|
||
* 18. Preemptive Scheduling
|
||
[[file:images/preemptive-scheduling.jpg]]
|
||
|
||
* 19. Preemptive Scheduling
|
||
[[file:images/preemptive-scheduling-tree.jpg]]
|
||
|
||
* 20. Multi Core
|
||
[[file:images/preemptive-scheduling-multi-core.jpg]]
|
||
|
||
* 21. Concurrency versus Parallelism
|
||
|
||
- Parallelism :: performing computations at the same time.
|
||
- Multiple cores.
|
||
- Multiples computers.
|
||
- Kind of like having multiple cooks in a kitchen.
|
||
- Can do different things at the same time.
|
||
- Can do one job faster (peeling a lot of potatoes).
|
||
|
||
- Concurrency :: performing computations over inter-leaving, non-sequential time periods.
|
||
- Single or multi-core.
|
||
|
||
* 22. Processes and Threads
|
||
A little more nuance.
|
||
|
||
- Different /recipes/
|
||
- Different /parts/ of a recipe
|
||
|
||
- Process :: a program in execution.
|
||
- Start off with one thread.
|
||
- Can /fork/.
|
||
- Communicate via inter-process communication (IPC).
|
||
|
||
- Thread :: sequence of instructions that can be managed independently by scheduler.
|
||
- Share resources ∴ less resource intensive.
|
||
- Can communicate via shared resources.
|
||
|
||
* 23. Processes and Threads
|
||
|
||
[[file:images/process-vs-thread.jpg]]
|
||
|
||
Source: https://stackoverflow.com/questions/16354460/forking-vs-threading/16354658
|
||
|
||
* 24. htop
|
||
|
||
Press ~H~ to show threads.
|
||
|
||
* 25. Languages and Runtimes
|
||
|
||
- ~C~, ~C++~, ~Rust~ allow for fine-grained control of processes and threads.
|
||
#+begin_src C :output raw
|
||
int main() {
|
||
fork();
|
||
|
||
printf("Hello world!\n");
|
||
|
||
return 0;
|
||
}
|
||
#+end_src
|
||
|
||
* 26. JavaScript
|
||
- Single-threaded (for application code).
|
||
- Runtimes implement concurrency via an event loop.
|
||
|
||
[[file:images/nodejs-event-loop.png]]
|
||
|
||
Source: [[https://medium.com/preezma/node-js-event-loop-architecture-go-deeper-node-core-c96b4cec7aa4]]
|
||
|
||
* 27. Go
|
||
Runtime multi-threading, multiplexes goroutines onto threads.
|
||
|
||
[[file:images/go-scheduler.png]]
|
||
|
||
Source: https://freethreads.net/2019/01/23/go-runtime-scheduler-design-internals/
|
||
|
||
* 28. Quick Review
|
||
|
||
- Sequential versus Concurrent Steps
|
||
- Compare with recipes.
|
||
- Concurrency versus Parallelism
|
||
- Parallelism: multiple cooks, simultaneously /doing/.
|
||
- Concurrency: one or more cooks, simultaneously /dealing/ with many tasks.
|
||
- Operating System Scheduler
|
||
- Processes and Threads
|
||
- Languages and Runtimes
|
||
|
||
* 29. Part 2: Using Concurrency
|
||
|
||
Typical use cases:
|
||
- Network I/O
|
||
- Requests between clients and servers.
|
||
- Requests to a database.
|
||
- Requests between microservices.
|
||
- Requests to external APIs.
|
||
|
||
- User interfaces
|
||
- Capturing user input
|
||
- Rendering output
|
||
|
||
- Similar, but independent computations.
|
||
|
||
- Solving problems that are best expressed with concurrency.
|
||
|
||
* 30. JavaScript
|
||
|
||
What is the output of the following? Why?
|
||
#+begin_src javascript
|
||
console.log('start');
|
||
|
||
setTimeout(() => console.log('1'), 0);
|
||
|
||
console.log('middle');
|
||
|
||
setTimeout(() => console.log('2'), 0);
|
||
|
||
console.log('end');
|
||
#+end_src
|
||
|
||
* 31. Go Examples
|
||
|
||
1. Shared Memory with Mutex (Mutual Exclusion)
|
||
- Pitfall: synchronization, debugging.
|
||
2. Message Passing (Channels and Goroutines)
|
||
- Pitfall: resources usage, complexity.
|
||
|
||
* 32. Mutex Example
|
||
Concurrent updates to a bank account balance.
|
||
|
||
Code:
|
||
[[file:cmd/mutex/main.go]]
|
||
|
||
Run: [[file:cmd/mutex/run.sh]]
|
||
- Without mutex.
|
||
- With mutex.
|
||
|
||
* 33. Data Races
|
||
[[file:images/data-race.jpg]]
|
||
|
||
* 34. Mutex
|
||
|
||
/Mutual Exclusion/
|
||
|
||
- Purpose: protect a critical section of code.
|
||
- Effect: serialization.
|
||
|
||
* 35. Race Detector
|
||
|
||
- Without mutex.
|
||
- With mutex.
|
||
|
||
* 36. Tradeoffs: Validating Correctness, Testing, Debugging
|
||
|
||
- Non-deterministic behavior
|
||
- Works fine on one run, fatal bug in another.
|
||
- Up to how the scheduler(s) order things at execution time.
|
||
- Concealed bugs
|
||
|
||
* 37. Message Passing
|
||
|
||
Proverbs:
|
||
#+begin_quote
|
||
Don't communicate by sharing memory, share memory by communicating.
|
||
|
||
Channels orchestrate, mutexes serialize.
|
||
#+end_quote
|
||
|
||
* 38. Requirements
|
||
|
||
- Read CSV of zip codes
|
||
- 50 in our example
|
||
- Full list is ~42,000
|
||
|
||
- Get temperature with OpenWeatherMap API
|
||
- 60 requests per minute
|
||
- Need to rate limit our API requests.
|
||
|
||
- Display running list of top ten temperatures.
|
||
|
||
* 39. Concurrency Hallmarks
|
||
|
||
- Repeatedly processing similar data (lines of CSV)
|
||
- Network I/O
|
||
- UI I/O
|
||
|
||
* 40. Message Passing Example
|
||
|
||
Run: [[file:cmd/csp/run.sh]]
|
||
|
||
* 41. Code Walkthrough
|
||
|
||
[[file:cmd/csp/main.go]]
|
||
|
||
* 42. Channels and Goroutines
|
||
[[file:images/message-passing-example-diagram.jpg]]
|
||
|
||
* 43. Channels and Goroutines
|
||
[[file:images/go-scheduler.jpg]]
|
||
|
||
* 44. Scaling
|
||
|
||
What if:
|
||
- We wanted to process the entire list of 42,000 zip codes?
|
||
- We had access to the Enterprise tier of the Open Weather Map API (200,000 requests per minute).
|
||
|
||
Could we just get rid of the limiter?
|
||
|
||
* 45. Resources
|
||
|
||
Probably not, because:
|
||
|
||
- OS limit on open files (~ulimit~)
|
||
- Ask me how I know...
|
||
- Server load
|
||
- Network load
|
||
- CPU
|
||
- Ranking goroutine becomes a chokepoint.
|
||
- Better data structures and algorithms could help here.
|
||
- Goroutines are cheap, but not free.
|
||
|
||
* 46. Possible Solutions
|
||
|
||
- Use a semaphore to limit concurrency.
|
||
- Buffers
|
||
|
||
* 47. Designing and Managing Concurrency
|
||
|
||
- Size of input matters.
|
||
- Resources
|
||
- Pipelines
|
||
- Chokepoints
|
||
- Backpressure
|
||
- Complexity
|
||
|
||
* 48. Quick Review
|
||
|
||
Techniques for implementing and managing concurrency:
|
||
- Event loop
|
||
- Mutexes
|
||
- Communication (channels and goroutines)
|
||
|
||
Concurrency Pitfalls:
|
||
- Synchronization
|
||
- Resource usage
|
||
- Debugging
|
||
- Complexity
|
||
- Need to think differently about concurrent code.
|
||
|
||
* 49. Part 3: Infrastructure Concurrency
|
||
|
||
Concurrent applications running concurrently on distributed computers.
|
||
|
||
- Client Application
|
||
- Backend Application
|
||
- Databases
|
||
- Host and Network Infrastructure
|
||
|
||
* 50. Example
|
||
[[file:images/infrastructure-data-race.jpg]]
|
||
|
||
* 51. Review
|
||
|
||
- What concurrency is
|
||
- Sequential versus concurrent
|
||
- Comparison with recipes
|
||
- Compared to parallelism
|
||
- How it is implemented
|
||
- Operating System Scheduler and Interrupts
|
||
- Languages and Runtimes
|
||
|
||
* 52. Review
|
||
|
||
- When, why, and how to use it
|
||
- I/O + UI
|
||
- Similar, independent computations
|
||
- Expressiveness
|
||
|
||
- Pitfalls
|
||
- Synchronization
|
||
- Resources Usage
|
||
- Debugging
|
||
- Complexity
|
||
|
||
* 53. Review
|
||
|
||
- Infrastructure
|
||
- Data Race
|
||
|
||
* 54. Conclusion
|
||
|
||
- Goals:
|
||
- Give you a rough sense of what concurrency is.
|
||
- Provide context to help you understand what's going on.
|
||
- Practical exposure, to expose some of the trade-offs.
|
||
|
||
- Awareness and Intuition
|
||
|
||
- Remember:
|
||
- /Premature optimization is the root of all evil/.
|
||
- Tradeoffs and economics (resources, costs, developer time).
|
||
- Learning is continuous...
|
||
|
||
* 55. Further Reading
|
||
|
||
- [[https://slikts.github.io/concurrency-glossary/][Concurrency Glossary]]
|
||
- [[https://blog.golang.org/waza-talk][Concurrency is not parallelism]]
|
||
- [[https://en.wikipedia.org/wiki/Executable_and_Linkable_Format][Executable and Linkable Format]]
|
||
- [[https://www.youtube.com/watch?v=KBZlN0izeiY][GopherCon 2017: Kavya Joshi - Understanding Channels]]
|
||
- [[http://web.mit.edu/6.005/www/fa15/][MIT 6.005: Software Construction]]
|
||
- Chapters 19, 20, 22, 23
|
||
- [[https://www.gopl.io/][The Go Programming Language]]
|
||
- [[https://en.wikipedia.org/wiki/Concurrency_(computer_science)][Wikipedia: Concurrency]]
|
||
- [[https://en.wikipedia.org/wiki/Concurrency_control][Wikipedia: Concurrency Control]]
|
||
- [[https://en.wikipedia.org/wiki/Concurrent_computing][Wikipedia: Concurrent Computing]]
|
||
- [[https://en.wikipedia.org/wiki/Deadlock][Wikipedia: Deadlock]]
|
||
- [[https://en.wikipedia.org/wiki/Dining_philosophers_problem][Wikipedia: Dining Philosophers Problem]]
|
||
- [[https://www.backblaze.com/blog/whats-the-diff-programs-processes-and-threads/][What’s the Diff: Programs, Processes, and Threads]]
|
||
- [[https://www.youtube.com/watch?v=8aGhZQkoFbQ&t=13s][What the heck is the event loop anyway? | Philip Roberts]]
|
||
- [[https://en.wikipedia.org/wiki/Heisenbug][Wikipedia: Heisenbug]]
|
||
- [[https://en.wikipedia.org/wiki/Interrupt][Wikipedia: Interrupt]]
|
||
- [[https://en.wikipedia.org/wiki/Libuv][Wikipedia: Libuv]]
|
||
- [[https://en.wikipedia.org/wiki/Process_(computing)][Wikipedia: Process]]
|
||
- [[https://en.wikipedia.org/wiki/Scheduling_(computing)][Wikipedia: Scheduling]]
|
||
- [[https://en.wikipedia.org/wiki/Therac-25][Wikipedia: Therac-25]]
|
||
- [[https://en.wikipedia.org/wiki/Thread_(computing)][Wikipedia: Thread]]
|
||
- [[https://github.com/danicat/pacgo][pacgo: A Pac Man clone written in Go]]
|
||
- [[https://medium.com/preezma/node-js-event-loop-architecture-go-deeper-node-core-c96b4cec7aa4][node.js event loop architecture]]
|