Preface xxix

Acknowledgments xxxv

About the Author xxxvii



Chapter 1: Introduction 1

1.1 Systems Performance 1

1.2 Roles 2

1.3 Activities 3

1.4 Perspectives 4

1.5 Performance Is Challenging 5

1.6 Latency 6

1.7 Observability 7

1.8 Experimentation 13

1.9 Cloud Computing 14

1.10 Methodologies 15

1.11 Case Studies 16

1.12 References 19



Chapter 2: Methodologies 21

2.1 Terminology 22

2.2 Models 23

2.3 Concepts 24

2.4 Perspectives 37

2.5 Methodology 40

2.6 Modeling 62

2.7 Capacity Planning 69

2.8 Statistics 73

2.9 Monitoring 77

2.10 Visualizations 79

2.11 Exercises 85

2.12 References 86



Chapter 3: Operating Systems 89

3.1 Terminology 90

3.2 Background 91

3.3 Kernels 111

3.4 Linux 114

3.5 Other Topics 122

3.6 Kernel Comparisons 124

3.7 Exercises 124

3.8 References 125



Chapter 4: Observability Tools 129

4.1 Tool Coverage 130

4.2 Tool Types 133

4.3 Observability Sources 138

4.4 sar 160

4.5 Tracing Tools 166

4.6 Observing Observability 167

4.7 Exercises 168

4.8 References 168



Chapter 5: Applications 171

5.1 Application Basics 172

5.2 Application Performance Techniques 176

5.3 Programming Languages 182

5.4 Methodology 186

5.5 Observability Tools 199

5.6 Gotchas 213

5.7 Exercises 216

5.8 References 217



Chapter 6: CPUs 219

6.1 Terminology 220

6.2 Models 221

6.3 Concepts 223

6.4 Architecture 229

6.5 Methodology 244

6.6 Observability Tools 254

6.7 Visualizations 288

6.8 Experimentation 293

6.9 Tuning 294

6.10 Exercises 299

6.11 References 300



Chapter 7: Memory 303

7.1 Terminology 304

7.2 Concepts 305

7.3 Architecture 311

7.4 Methodology 323

7.5 Observability Tools 328

7.6 Tuning 350

7.7 Exercises 354

7.8 References 355



Chapter 8: File Systems 359

8.1 Terminology 360

8.2 Models 361

8.3 Concepts 362

8.4 Architecture 372

8.5 Methodology 383

8.6 Observability Tools 391

8.7 Experimentation 411

8.8 Tuning 414

8.9 Exercises 419

8.10 References 420



Chapter 9: Disks 423

9.1 Terminology 424

9.2 Models 425

9.3 Concepts 427

9.4 Architecture 435

9.5 Methodology 449

9.6 Observability Tools 458

9.7 Visualizations 487

9.8 Experimentation 490

9.9 Tuning 493

9.10 Exercises 495

9.11 References 496



Chapter 10: Network 499

10.1 Terminology 500

10.2 Models 501

10.3 Concepts 503

10.4 Architecture 509

10.5 Methodology 524

10.6 Observability Tools 533

10.7 Experimentation 562

10.8 Tuning 567

10.9 Exercises 574

10.10 References 575



Chapter 11: Cloud Computing 579

11.1 Background 580

11.2 Hardware Virtualization 587

11.3 OS Virtualization 605

11.4 Lightweight Virtualization 630

11.5 Other Types 634

11.6 Comparisons 634

11.7 Exercises 636

11.8 References 637



Chapter 12: Benchmarking 641

12.1 Background 642

12.2 Benchmarking Types 651

12.3 Methodology 656

12.4 Benchmark Questions 667

12.5 Exercises 668

12.6 References 669



Chapter 13: perf 671

13.1 Subcommands Overview 672

13.2 One-Liners 674

13.3 perf Events 679

13.4 Hardware Events 681

13.5 Software Events 683

13.6 Tracepoint Events 684

13.7 Probe Events 685

13.8 perf stat 691

13.9 perf record 694

13.10 perf report 696

13.11 perf script 698

13.12 perf trace 701

13.13 Other Commands 702

13.14 perf Documentation 703

13.15 References 703



Chapter 14: Ftrace 705

14.1 Capabilities Overview 706

14.2 tracefs (/sys) 708

14.3 Ftrace Function Profiler 711

14.4 Ftrace Function Tracing 713

14.5 Tracepoints 717

14.6 kprobes 719

14.7 uprobes 722

14.8 Ftrace function_graph 724

14.9 Ftrace hwlat 726

14.10 Ftrace Hist Triggers 727

14.11 trace-cmd 734

14.12 perf ftrace 741

14.13 perf-tools 741

14.14 Ftrace Documentation 748

14.15 References 749



Chapter 15: BPF 751

15.1 BCC 753

15.2 bpftrace 761

15.3 References 782



Chapter 16: Case Study 783

16.1 An Unexplained Win 783

16.2 Additional Information 792

16.3 References 793



Appendix A: USE Method: Linux 795

Appendix B: sar Summary 801

Appendix C: bpftrace One-Liners 803

Appendix D: Solutions to Selected Exercises 809

Appendix E: Systems Performance Who's Who 811



Glossary 815

Index 825