talks/concurrency/slides.org

• 1. Concurrency
• 2. Why
• 3. Why
• 4. Why
• 5. Why
• 6. Goals
• 7. Three Parts
• 8. Definition
• 9. Time
• 10. Example
• 11. Example
• 12. Cooking Recipes
• 13. Multitasking
• 14. htop
• 15. How?
• 17. Cooperative Scheduling
• 18. Preemptive Scheduling
• 19. Preemptive Scheduling
• 20. Multi Core
• 21. Concurrency versus Parallelism
• 22. Processes and Threads
• 23. Processes and Threads
• 24. htop
• 25. Languages and Runtimes
• 26. nodejs
• 27. Go
• 28. Quick Review
• 29. Part 2: Using Concurrency
• 30. JavaScript
• 31. Go Examples
• 32. Mutex Example
• 33. Data Races
• 34. Mutex
• 35. Race Detector
• 36. Tradeoffs: Validating Correctness, Testing, Debugging
• 37. Message Passing
• 38. Requirements
• 39. Concurrency Hallmarks
• 40. Message Passing Example
• 41. Code Walkthrough
• 42. Channels and Goroutines
• 43. Channels and Goroutines
• 44. Scaling
• 45. Resources
• 46. Possible Solutions
• 47. Designing and Managing Concurrency
• 48. Quick Review
• 49. Part 3: Infrastructure Concurrency
• 50. Example
• 51. Review
• 52. Review
• 53. Review
• 54. Conclusion
• 55. Further Reading

1. Concurrency

• David Wen Riccardi-Zhu
• Senior Software Engineer @ Good Uncle
• dwrz@dwrz.net

Slides and code: https://github.com/dwrz/talks/concurrency.

2. Why

You'll likely encounter it at work:

• Client requests to your backend
• Database and API requests made by your backend
• Frontend rendering

3. Why

Performance Benefits

• I/O versus CPU bound applications.
• Typically outweigh gains from optimized data structures and algorithms.

• Hardware limitations.

  • Rate of single core performance improvement is slowing.
  • More cores.

4. Why

Expressive

5. Why

Difficult to do correctly.

Easy to do incorrectly.

6. Goals

• Give you a rough sense of what concurrency is.
• Provide context to help you understand how it's made possible.
• Practical demonstration, to expose some of the trade-offs.

7. Three Parts

1. Fundamentals
2. Application
3. Infrastructure

8. Definition

Concurrency

1. A running together in place or time.
2. Simultaneous occurrence.

OK starting point, but:

Time, and simultaneity, is complicated.

A second is a duration, not an instant.

Two things can happen in the same second and not be simultaneous.

In computer science, this nuance matters.

9. Time

System Event	Actual Latency	Scaled Latency
One CPU cycle	0.4 ns	1 s
Level 1 cache access	0.9 ns	2 s
Level 2 cache access	2.8 ns	7 s
Level 3 cache access	28 ns	1 min
Main memory access	~100 ns	4 min
NVMe SSD I/O	~25 μs	17 hrs
SSD I/O	50–150 μs	1.5–4 days
Rotational disk I/O	1–10 ms	1–9 months
Internet: SF to NYC	65 ms	5 years
Internet: SF to HK	141 ms	11 years

Source: https://www.prowesscorp.com/computer-latency-at-a-human-scale/

10. Example

• Get the current temperature of ten most populous cities in the US.
• First, sequentially.
• Then, concurrently.

cmd/temperatures/run.sh

11. Example

12. Cooking Recipes

• Step-by-step

• Concurrent

  • Pre-heat oven
  • Boil water
  • Timers

13. Multitasking

Computers can "simultaneously" work on:

• Different parts of a recipe
• Different recipes

14. htop

15. How?

• Operating System's Scheduler

• Hardware

  • Interrupts

17. Cooperative Scheduling

18. Preemptive Scheduling

19. Preemptive Scheduling

20. Multi Core

21. Concurrency versus Parallelism

Parallelism

performing computations at the same moment.

• Multiple cores.
• Multiples computers.

• Kind of like having multiple cooks in a kitchen.

  • Can do different things at the same time.
  • Can do one job faster (peeling a lot of potatoes).

Concurrency

performing computations over inter-leaving, non-sequential time periods.

• Single or multi-core.

22. Processes and Threads

A little more nuance.

• Different recipes
• Different parts of a recipe

• Process :: a program in execution.

  • Start off with one thread.
  • Can fork.
  • Inter-Process Communication

• Thread :: sequence of instructions that can be managed independently by scheduler.

  • Share resources ∴ less resource intensive.
  • Can communicate via shared resources.

23. Processes and Threads

Source: https://stackoverflow.com/questions/16354460/forking-vs-threading/16354658

24. htop

Press H to show threads.

25. Languages and Runtimes

• C, C++, Rust allow for fine-grained control of processes and threads.

int main() {
	fork();

	printf("Hello world!\n");

	return 0;
}

26. nodejs

• Single-threaded (for application code).
• Runtimes implement concurrency via an event loop.

Source: https://medium.com/preezma/node-js-event-loop-architecture-go-deeper-node-core-c96b4cec7aa4

27. Go

Runtime multi-threading, multiplexes goroutines onto threads.

Source: https://freethreads.net/2019/01/23/go-runtime-scheduler-design-internals/

28. Quick Review

• Sequential versus Concurrent Steps

  • Compare with recipes.

• Concurrency versus Parallelism

  • Multiple cooks.
• Operating System Scheduler
• Processes and Threads
• Languages and Runtimes

29. Part 2: Using Concurrency

Typical use cases:

• Network I/O

  • Requests between clients and servers.
  • Requests to a database.
  • Requests between microservices.
  • Requests to external APIs.

• User interfaces

  • Capturing user input
  • Rendering output
• Similar, but independent computations.
• Solving problems that are best expressed with concurrency.

30. JavaScript

What is the output of the following? Why?

console.log('start');

setTimeout(() => console.log('1'), 0);

console.log('middle');

setTimeout(() => console.log('2'), 0);

console.log('end');

31. Go Examples

1. Shared Memory with Mutex (Mutual Exclusion)

   • Pitfall: synchronization, debugging.

2. Message Passing (Channels and Goroutines)

   • Pitfall: resources usage, complexity.

32. Mutex Example

Concurrent updates to a bank account balance.

Code: cmd/mutex/main.go

Run: cmd/mutex/run.sh

• Without mutex.
• With mutex.

33. Data Races

34. Mutex

Mutual Exclusion

• Purpose: protect a critical section of code.
• Effect: serialization.

35. Race Detector

• Without mutex.
• With mutex.

36. Tradeoffs: Validating Correctness, Testing, Debugging

• Non-deterministic behavior

  • Works fine on one run, fatal bug in another.
  • Up to how the scheduler(s) order things at execution time.
• Concealed bugs

37. Message Passing

Proverbs:

Don't communicate by sharing memory, share memory by communicating.

Channels orchestrate, mutexes serialize.

38. Requirements

• Read CSV of zip codes

  • 50 in our example
  • Full list is ~42,000

• Get temperature with OpenWeatherMap API

  • 60 requests per minute
  • Need to rate limit our API requests.
• Display running list of top ten temperatures.

39. Concurrency Hallmarks

• Repeatedly processing similar data (lines of CSV)
• Network I/O
• UI I/O

40. Message Passing Example

Run: cmd/csp/run.sh

41. Code Walkthrough

cmd/csp/main.go

42. Channels and Goroutines

43. Channels and Goroutines

44. Scaling

What if:

• We wanted to process the entire list of 42,000 zip codes?
• We had access to the Enterprise tier of the Open Weather Map API (200,000 requests per minute).

Could we just get rid of the limiter?

45. Resources

Probably not, because:

• OS limit on open files (ulimit)

  • Ask me how I know…
• Server load
• Network load

• CPU

  • Ranking goroutine becomes a chokepoint.
  • Better data structures and algorithms could help here.
• Goroutines are cheap, but not free.

46. Possible Solutions

• Use a semaphore to limit concurrency.
• Buffers

47. Designing and Managing Concurrency

• Size of input matters.
• Resources
• Pipelines
• Chokepoints
• Backpressure
• Complexity

48. Quick Review

Techniques for implementing and managing concurrency:

• Event loop
• Mutexes
• Communication (channels and goroutines)

Concurrency Pitfalls:

• Synchronization
• Resource usage
• Debugging

• Complexity

  • Need to think differently about concurrent code.

49. Part 3: Infrastructure Concurrency

Concurrent applications running concurrently on distributed computers.

• Client Application
• Backend Application
• Databases
• Host and Network Infrastructure

50. Example

51. Review

• What concurrency is

  • Sequential versus concurrent
  • Comparison with recipes
  • Compared to parallelism

• How it is implemented

  • Operating System Scheduler and Interrupts
  • Languages and Runtimes

52. Review

• When, why, and how to use it

  • I/O + UI
  • Similar, independent computations
  • Expressiveness

• Pitfalls

  • Synchronization
  • Resources Usage
  • Debugging
  • Complexity

53. Review

• Infrastructure

  • Data Race

54. Conclusion

• Goals:

  • Give you a rough sense of what concurrency is.
  • Provide context to help you understand what's going on.
  • Practical exposure, to expose some of the trade-offs.
• Awareness and Intuition

• Remember:

  • Premature optimization is the root of all evil.
  • Tradeoffs and economics (resources, costs, developer time).
  • Learning is continuous…

55. Further Reading

• Concurrency Glossary
• Concurrency is not parallelism
• Executable and Linkable Format
• GopherCon 2017: Kavya Joshi - Understanding Channels

• MIT 6.005: Software Construction

  • Chapters 19, 20, 22, 23
• The Go Programming Language
• Wikipedia: Concurrency
• Wikipedia: Concurrency Control
• Wikipedia: Concurrent Computing
• Wikipedia: Deadlock
• Wikipedia: Dining Philosophers Problem
• What’s the Diff: Programs, Processes, and Threads
• What the heck is the event loop anyway? | Philip Roberts
• Wikipedia: Heisenbug
• Wikipedia: Interrupt
• Wikipedia: Libuv
• Wikipedia: Process
• Wikipedia: Scheduling
• Wikipedia: Therac-25
• Wikipedia: Thread
• pacgo: A Pac Man clone written in Go
• node.js event loop architecture