talks/concurrency/slides.org

• 1. Concurrency
• 2. Why
• 3. Why
• 4. Why
• 5. Why
• 6. Goals
• 7. Three Parts
• 8. Definition
• 9. Time
• 10. Example
• 11. Example
• 12. Cooking Recipes
• 13. Multitasking
• 14. htop
• 15. How?
• 16. Cooperative Scheduling
• 17. Preemptive Scheduling
• 18. Preemptive Scheduling
• 19. Multi Core
• 20. Concurrency versus Parallelism
• 21. Processes and Threads
• 22. Processes and Threads
• 23. htop
• 24. Languages and Runtimes
• 25. JavaScript
• 26. Aside: Event Loop
• 27. Go
• 28. Quick Review
• 29. Part 2: Using Concurrency
• 30. Go Examples
• 31. Mutex Example
• 32. Data Races
• 33. Mutex
• 34. Race Detector
• 35. Tradeoffs: Validating Correctness, Testing, Debugging
• 36. Message Passing
• 37. Program Requirements
• 38. Concurrency Hallmarks
• 39. Message Passing Example
• 40. Code Walkthrough
• 41. Channels and Goroutines
• 42. Channels and Goroutines
• 43. Scaling
• 44. Resources
• 45. Possible Solutions
• 46. Designing and Managing Concurrency
• 47. Quick Review
• 48. Part 3: Infrastructure Concurrency
• 49. Example
• 50. Review
• 51. Review
• 52. Review
• 53. Conclusion
• 54. Further Reading

1. Concurrency

• David Wen Riccardi-Zhu
• Senior Software Engineer @ Good Uncle
• dwrz@dwrz.net

Slides and code: https://github.com/dwrz/talks/concurrency.

2. Why

You'll likely encounter it at work:

• Client requests to your backend
• Database and API requests made by your backend
• Frontend rendering

3. Why

Performance Benefits

• Typically outweigh gains from optimized data structures and algorithms.

  • I/O versus CPU bound applications.

• Hardware limitations.

  • Rate of single core performance improvement is slowing.
  • More cores.

4. Why

Expressive

5. Why

Difficult to do correctly.

Easy to do incorrectly.

6. Goals

• Give you a rough sense of what concurrency is.
• Provide context to help you understand how it's made possible.
• Practical demonstration, to expose some of the trade-offs.

7. Three Parts

1. Fundamentals
2. Application
3. Infrastructure

8. Definition

Concurrency

1. A running together in place or time.
2. Simultaneous occurrence.

OK starting point, but:

Time, and simultaneity, is complicated.

A second is a duration, not an instant.

Two things can happen in the same second and not be simultaneous.

With computers, this nuance matters.

9. Time

System Event	Actual Latency	Scaled Latency
One CPU cycle	0.4 ns	1 s
Level 1 cache access	0.9 ns	2 s
Level 2 cache access	2.8 ns	7 s
Level 3 cache access	28 ns	1 min
Main memory access	~100 ns	4 min
NVMe SSD I/O	~25 μs	17 hrs
SSD I/O	50–150 μs	1.5–4 days
Rotational disk I/O	1–10 ms	1–9 months
Internet: SF to NYC	65 ms	5 years
Internet: SF to HK	141 ms	11 years

Source: https://www.prowesscorp.com/computer-latency-at-a-human-scale/

10. Example

• Get the current temperature of ten most populous cities in the US.
• First, sequentially.
• Then, concurrently.

cmd/temperatures/run.sh

11. Example

12. Cooking Recipes

• Step-by-step

• Concurrent:

  • Handle other tasks while:

    • Preheating oven
    • Boiling water
    • Waiting for timers
  • Switching back to a task when it is ready for more work.

13. Multitasking

Computers can "simultaneously" work on:

• Different parts of a recipe
• Different recipes

14. htop

15. How?

Combination of:

• Operating System Scheduler

• Hardware (interrupts)

  • Keyboard
  • Filesystem
  • Network Card
  • Timer

16. Cooperative Scheduling

17. Preemptive Scheduling

18. Preemptive Scheduling

19. Multi Core

20. Concurrency versus Parallelism

Parallelism

performing computations at the same time.

• Multiple cores.
• Multiple computers.

• Kind of like having multiple cooks in a kitchen.

  • Can do different things at the same time.
  • Can do one job faster (peeling a lot of potatoes).

Concurrency

performing computations over inter-leaving, non-sequential time periods.

• Single or multi-core.

21. Processes and Threads

A little more nuance.

• Different recipes
• Different parts of a recipe

• Process :: a program in execution.

  • Unique resources (memory, code, files, etc.).
  • Start off with one thread.
  • Can fork.
  • Communicate via inter-process communication (IPC).

• Thread :: sequence of instructions that can be managed independently by scheduler.

  • Share resources ∴ less resource intensive.
  • Can communicate via shared resources.

22. Processes and Threads

Source: https://stackoverflow.com/questions/16354460/forking-vs-threading/16354658

23. htop

Press H to show threads.

24. Languages and Runtimes

• C, C++, Rust allow for fine-grained control of processes and threads.

int main() {
	fork();

	printf("Hello world!\n");

	return 0;
}

25. JavaScript

• Single-threaded (for application code).
• Runtimes implement concurrency via an event loop.

Source: https://medium.com/preezma/node-js-event-loop-architecture-go-deeper-node-core-c96b4cec7aa4

26. Aside: Event Loop

What is the output of the following? Why?

console.log('start');

setTimeout(() => console.log('1'), 0);

console.log('middle');

setTimeout(() => console.log('2'), 0);

console.log('end');

27. Go

Runtime scheduler manages multiple threads.

Source: https://freethreads.net/2019/01/23/go-runtime-scheduler-design-internals/

28. Quick Review

• Sequential versus Concurrent Steps

  • Compare with recipes.

• Concurrency versus Parallelism

  • Parallelism: multiple cooks, simultaneously doing.
  • Concurrency: one or more cooks, dealing with many tasks.
• Operating System Scheduler
• Processes and Threads
• Languages and Runtimes

29. Part 2: Using Concurrency

Typical use cases:

• Network I/O between:

  • Clients and servers.
  • Services and database(s).
  • Microservices.
  • Service and external APIs.

• User interfaces

  • Capturing user input
  • Rendering output
• Similar, but independent computations.
• Solving problems that are best expressed with concurrency.

30. Go Examples

1. Shared Memory with Mutex (Mutual Exclusion)

   • Pitfall: synchronization, debugging.

2. Message Passing (Channels and Goroutines)

   • Pitfall: resource usage, complexity.

31. Mutex Example

Concurrent updates to a bank account balance.

Code: cmd/mutex/main.go

Run: cmd/mutex/run.sh

• Without mutex.

32. Data Races

33. Mutex

Mutual Exclusion

• Purpose: protect a critical section of code.
• Effect: serialization.

Run: cmd/mutex/run.sh

• With mutex.

34. Race Detector

How can we be sure?

Race detector can give us some confidence (but no guarantees).

• Without mutex.
• With mutex.

35. Tradeoffs: Validating Correctness, Testing, Debugging

• Non-deterministic behavior

  • Works fine on one run, fatal bug in another.
  • Up to how the scheduler(s) order things at execution time.
• Concealed bugs

36. Message Passing

Proverbs:

Don't communicate by sharing memory, share memory by communicating.

Channels orchestrate, mutexes serialize.

37. Program Requirements

• Read CSV of zip codes

  • 50 in our example
  • Full list is ~42,000

• Get temperature with OpenWeatherMap API

  • 60 requests per minute
  • Need to rate limit our API requests.
• Display running list of top ten temperatures.

38. Concurrency Hallmarks

• Repeatedly processing similar data (lines of CSV)
• Network I/O
• UI I/O

39. Message Passing Example

Run: cmd/csp/run.sh

40. Code Walkthrough

cmd/csp/main.go

41. Channels and Goroutines

42. Channels and Goroutines

43. Scaling

What if:

• We wanted to process the entire list of 42,000 zip codes?
• We had access to the Enterprise tier of the Open Weather Map API (200,000 requests per minute).

Could we just get rid of the limiter?

44. Resources

Probably not, because:

• OS limit on open files (ulimit)

  • Ask me how I know…
• Server load
• Network load

• CPU

  • Ranking goroutine becomes a chokepoint.
  • Better data structures and algorithms could help here.
• Goroutines are cheap, but not free.

45. Possible Solutions

• Use a semaphore to limit concurrency.
• Buffers

46. Designing and Managing Concurrency

• Size of input matters.
• Resources
• Pipelines
• Chokepoints
• Backpressure
• Complexity

47. Quick Review

Techniques for implementing and managing concurrency:

• Event loop
• Mutexes
• Communication (channels and goroutines)

Concurrency Pitfalls:

• Synchronization
• Resource usage
• Debugging

• Complexity

  • Need to think differently about concurrent code.

48. Part 3: Infrastructure Concurrency

Concurrent applications running concurrently on distributed computers.

• Client Application
• Backend Application
• Databases
• Host and Network Infrastructure

49. Example

50. Review

• What concurrency is

  • Sequential versus concurrent
  • Comparison with recipes and cooks
  • Compared to parallelism

• How it is implemented

  • Operating System Scheduler and Interrupts
  • Languages and Runtimes

51. Review

• When, why, and how to use it

  • I/O + UI
  • Similar, independent computations
  • Expressiveness

• Pitfalls

  • Synchronization
  • Resource Usage
  • Debugging
  • Complexity

52. Review

• Infrastructure

  • Data Race

53. Conclusion

• Goals:

  • Give you a rough sense of what concurrency is.
  • Provide context to help you understand what's going on.
  • Practical exposure, to expose some of the trade-offs.
• Awareness and Intuition

• Remember:

  • Premature optimization is the root of all evil.
  • Tradeoffs and economics (resources, costs, developer time).
  • Learning is continuous…

54. Further Reading

• Concurrency Glossary
• Concurrency is not parallelism
• Executable and Linkable Format
• GopherCon 2017: Kavya Joshi - Understanding Channels

• MIT 6.005: Software Construction

  • Chapters 19, 20, 22, 23
• The Go Programming Language
• Wikipedia: Concurrency
• Wikipedia: Concurrency Control
• Wikipedia: Concurrent Computing
• Wikipedia: Deadlock
• Wikipedia: Dining Philosophers Problem
• What’s the Diff: Programs, Processes, and Threads
• What the heck is the event loop anyway? | Philip Roberts
• Wikipedia: Heisenbug
• Wikipedia: Interrupt
• Wikipedia: Libuv
• Wikipedia: Process
• Wikipedia: Scheduling
• Wikipedia: Therac-25
• Wikipedia: Thread
• pacgo: A Pac Man clone written in Go
• node.js event loop architecture