talks/concurrency/slides.org

* 1. Concurrency

- David Wen Riccardi-Zhu
- Senior Software Engineer @ Good Uncle
- dwrz@dwrz.net

Slides and code: https://github.com/dwrz/talks/concurrency.

* 2. Why

You'll likely encounter it at work:

- Client requests to your backend
- Database and API requests made by your backend
- Frontend rendering

* 3. Why

Performance Benefits

- Typically outweigh gains from optimized data structures and algorithms.
  - I/O versus CPU bound applications.

- Hardware limitations.
  - Rate of single core performance improvement is slowing.
  - More cores.

* 4. Why

Expressive

* 5. Why

Difficult to do correctly.

Easy to do incorrectly.

* 6. Goals

- Give you a rough sense of what concurrency is.
- Provide context to help you understand how it's made possible.
- Practical demonstration, to expose some of the trade-offs.

* 7. Three Parts

1. Fundamentals
2. Application
3. Infrastructure

* 8. Definition

/Concurrency/
1. A running together in place or time.
2. Simultaneous occurrence.

OK starting point, but:

Time, and simultaneity, is /complicated/.

A second is a duration, not an instant.

Two things can happen in the same second and not be simultaneous.

With computers, this nuance matters.

* 9. Time

|----------------------+----------------+----------------|
| System Event         | Actual Latency | Scaled Latency |
|----------------------+----------------+----------------|
| One CPU cycle        | 0.4 ns         | 1 s            |
| Level 1 cache access | 0.9 ns         | 2 s            |
| Level 2 cache access | 2.8 ns         | 7 s            |
| Level 3 cache access | 28 ns          | 1 min          |
| Main memory access   | ~100 ns        | 4 min          |
| NVMe SSD I/O         | ~25 μs         | 17 hrs         |
| SSD I/O              | 50–150 μs      | 1.5–4 days     |
| Rotational disk I/O  | 1–10 ms        | 1–9 months     |
| Internet: SF to NYC  | 65 ms          | 5 years        |
| Internet: SF to HK   | 141 ms         | 11 years       |
|----------------------+----------------+----------------|

Source: [[https://www.prowesscorp.com/computer-latency-at-a-human-scale/]]

* 10. Example

- Get the current temperature of ten most populous cities in the US.

- First, /sequentially./
- Then, /concurrently/.

[[file:cmd/temperatures/run.sh]]

* 11. Example

[[file:images/concurrent-requests.jpg]]

* 12. Cooking Recipes

- Step-by-step
- Concurrent:
  - Handle other tasks while:
    - Preheating oven
    - Boiling water
    - Waiting for timers
  - Switching back to a task when it is ready for more work.

* 13. Multitasking

Computers can "simultaneously" work on:

- Different /parts/ of a recipe
- Different /recipes/

* 14. htop

* 15. How?

Combination of:
- Operating System *Scheduler*
- Hardware (/interrupts/)
  - Keyboard
  - Filesystem
  - Network Card
  - Timer

* 16. Cooperative Scheduling
[[file:images/cooperative-scheduling.jpg]]

* 17. Preemptive Scheduling
[[file:images/preemptive-scheduling.jpg]]

* 18. Preemptive Scheduling
[[file:images/preemptive-scheduling-tree.jpg]]

* 19. Multi Core
[[file:images/preemptive-scheduling-multi-core.jpg]]

* 20. Concurrency versus Parallelism

- Parallelism :: performing computations at the same time.
  - Multiple cores.
  - Multiple computers.
  - Kind of like having multiple cooks in a kitchen.
    - Can do different things at the same time.
    - Can do one job faster (peeling a lot of potatoes).

- Concurrency :: performing computations over inter-leaving, non-sequential time periods.
  - Single or multi-core.

* 21. Processes and Threads
A little more nuance.

- Different /recipes/
- Different /parts/ of a recipe

- Process :: a program in execution.
  - Unique resources (memory, code, files, etc.).
  - Start off with one thread.
  - Can /fork/.
  - Communicate via inter-process communication (IPC).

- Thread :: sequence of instructions that can be managed independently by scheduler.
  - Share resources ∴ less resource intensive.
  - Can communicate via shared resources.

* 22. Processes and Threads

[[file:images/process-vs-thread.jpg]]

Source: https://stackoverflow.com/questions/16354460/forking-vs-threading/16354658

* 23. htop

Press ~H~ to show threads.

* 24. Languages and Runtimes

- ~C~, ~C++~, ~Rust~ allow for fine-grained control of processes and threads.
#+begin_src C :output raw
int main() {
	fork();

	printf("Hello world!\n");

	return 0;
}
#+end_src

* 25. JavaScript
- Single-threaded (for application code).
- Runtimes implement concurrency via an event loop.

[[file:images/nodejs-event-loop.png]]

Source: [[https://medium.com/preezma/node-js-event-loop-architecture-go-deeper-node-core-c96b4cec7aa4]]

* 26. Aside: Event Loop

What is the output of the following? Why?
#+begin_src javascript
console.log('start');

setTimeout(() => console.log('1'), 0);

console.log('middle');

setTimeout(() => console.log('2'), 0);

console.log('end');
#+end_src

* 27. Go
*Runtime scheduler* manages multiple threads.
[[file:images/go-scheduler.png]]

Source: https://freethreads.net/2019/01/23/go-runtime-scheduler-design-internals/

* 28. Quick Review

- Sequential versus Concurrent Steps
  - Compare with recipes.
- Concurrency versus Parallelism
  - Parallelism: multiple cooks, simultaneously /doing/.
  - Concurrency: one or more cooks, /dealing/ with many tasks.
- Operating System Scheduler
- Processes and Threads
- Languages and Runtimes

* 29. Part 2: Using Concurrency

Typical use cases:
- Network I/O between:
   - Clients and servers.
   - Services and database(s).
   - Microservices.
   - Service and external APIs.

- User interfaces
  - Capturing user input
  - Rendering output

- Similar, but independent computations.

- Solving problems that are best expressed with concurrency.

* 30. Go Examples

1. Shared Memory with Mutex (Mutual Exclusion)
   - Pitfall: synchronization, debugging.

2. Message Passing (Channels and Goroutines)
   - Pitfall: resource usage, complexity.

* 31. Mutex Example
Concurrent updates to a bank account balance.

Code:
[[file:cmd/mutex/main.go]]

Run: [[file:cmd/mutex/run.sh]]
  - Without mutex.

* 32. Data Races
[[file:images/data-race.jpg]]

* 33. Mutex

/Mutual Exclusion/

- Purpose: protect a critical section of code.
- Effect: serialization.

Run: [[file:cmd/mutex/run.sh]]
  - With mutex.

* 34. Race Detector

How can we be sure?

Race detector can give us /some/ confidence (but no guarantees).

- Without mutex.
- With mutex.

* 35. Tradeoffs: Validating Correctness, Testing, Debugging

- Non-deterministic behavior
  - Works fine on one run, fatal bug in another.
  - Up to how the scheduler(s) order things at execution time.
- Concealed bugs

* 36. Message Passing

Proverbs:
#+begin_quote
Don't communicate by sharing memory, share memory by communicating.

Channels orchestrate, mutexes serialize.
#+end_quote

* 37. Program Requirements

- Read CSV of zip codes
  - 50 in our example
  - Full list is ~42,000

- Get temperature with OpenWeatherMap API
  - 60 requests per minute
  - Need to rate limit our API requests.

- Display running list of top ten temperatures.

* 38. Concurrency Hallmarks

- Repeatedly processing similar data (lines of CSV)
- Network I/O
- UI I/O

* 39. Message Passing Example

Run: [[file:cmd/csp/run.sh]]

* 40. Code Walkthrough

[[file:cmd/csp/main.go]]

* 41. Channels and Goroutines
[[file:images/message-passing-example-diagram.jpg]]

* 42. Channels and Goroutines
[[file:images/go-scheduler.jpg]]

* 43. Scaling

What if:
- We wanted to process the entire list of 42,000 zip codes?
- We had access to the Enterprise tier of the Open Weather Map API (200,000 requests per minute).

Could we just get rid of the limiter?

* 44. Resources

Probably not, because:

- OS limit on open files (~ulimit~)
  - Ask me how I know...
- Server load
- Network load
- CPU
  - Ranking goroutine becomes a chokepoint.
  - Better data structures and algorithms could help here.
- Goroutines are cheap, but not free.

* 45. Possible Solutions

- Use a semaphore to limit concurrency.
- Buffers

* 46. Designing and Managing Concurrency

- Size of input matters.
- Resources
- Pipelines
- Chokepoints
- Backpressure
- Complexity

* 47. Quick Review

Techniques for implementing and managing concurrency:
- Event loop
- Mutexes
- Communication (channels and goroutines)

Concurrency Pitfalls:
- Synchronization
- Resource usage
- Debugging
- Complexity
  - Need to think differently about concurrent code.

* 48. Part 3: Infrastructure Concurrency

Concurrent applications running concurrently on distributed computers.

- Client Application
- Backend Application
- Databases
- Host and Network Infrastructure

* 49. Example
[[file:images/infrastructure-data-race.jpg]]

* 50. Review

- What concurrency is
  - Sequential versus concurrent
  - Comparison with recipes and cooks
  - Compared to parallelism
- How it is implemented
  - Operating System Scheduler and Interrupts
  - Languages and Runtimes

* 51. Review

- When, why, and how to use it
  - I/O + UI
  - Similar, independent computations
  - Expressiveness

- Pitfalls
  - Synchronization
  - Resource Usage
  - Debugging
  - Complexity

* 52. Review

- Infrastructure
  - Data Race

* 53. Conclusion

- Goals:
  - Give you a rough sense of what concurrency is.
  - Provide context to help you understand what's going on.
  - Practical exposure, to expose some of the trade-offs.

- Awareness and Intuition

- Remember:
  - /Premature optimization is the root of all evil/.
  - Tradeoffs and economics (resources, costs, developer time).
  - Learning is continuous...

* 54. Further Reading

- [[https://slikts.github.io/concurrency-glossary/][Concurrency Glossary]]
- [[https://blog.golang.org/waza-talk][Concurrency is not parallelism]]
- [[https://en.wikipedia.org/wiki/Executable_and_Linkable_Format][Executable and Linkable Format]]
- [[https://www.youtube.com/watch?v=KBZlN0izeiY][GopherCon 2017: Kavya Joshi - Understanding Channels]]
- [[http://web.mit.edu/6.005/www/fa15/][MIT 6.005: Software Construction]]
  - Chapters 19, 20, 22, 23
- [[https://www.gopl.io/][The Go Programming Language]]
- [[https://en.wikipedia.org/wiki/Concurrency_(computer_science)][Wikipedia: Concurrency]]
- [[https://en.wikipedia.org/wiki/Concurrency_control][Wikipedia: Concurrency Control]]
- [[https://en.wikipedia.org/wiki/Concurrent_computing][Wikipedia: Concurrent Computing]]
- [[https://en.wikipedia.org/wiki/Deadlock][Wikipedia: Deadlock]]
- [[https://en.wikipedia.org/wiki/Dining_philosophers_problem][Wikipedia: Dining Philosophers Problem]]
- [[https://www.backblaze.com/blog/whats-the-diff-programs-processes-and-threads/][What’s the Diff: Programs, Processes, and Threads]]
- [[https://www.youtube.com/watch?v=8aGhZQkoFbQ&t=13s][What the heck is the event loop anyway? | Philip Roberts]]
- [[https://en.wikipedia.org/wiki/Heisenbug][Wikipedia: Heisenbug]]
- [[https://en.wikipedia.org/wiki/Interrupt][Wikipedia: Interrupt]]
- [[https://en.wikipedia.org/wiki/Libuv][Wikipedia: Libuv]]
- [[https://en.wikipedia.org/wiki/Process_(computing)][Wikipedia: Process]]
- [[https://en.wikipedia.org/wiki/Scheduling_(computing)][Wikipedia: Scheduling]]
- [[https://en.wikipedia.org/wiki/Therac-25][Wikipedia: Therac-25]]
- [[https://en.wikipedia.org/wiki/Thread_(computing)][Wikipedia: Thread]]
- [[https://github.com/danicat/pacgo][pacgo: A Pac Man clone written in Go]]
- [[https://medium.com/preezma/node-js-event-loop-architecture-go-deeper-node-core-c96b4cec7aa4][node.js event loop architecture]]