Why Do We Need Multiple Processors?

Goal: do more work in less time, while using energy efficiently.

• For decades, chips got faster by raising the clock rate, until the power wall: too much power and heat (Chapter 1).
• New plan: put more processors (cores) on a chip instead of making one core faster.
• Small, efficient cores give better performance per watt than one big, hot core.
• Analogy: instead of one worker rushing through four jobs, hire four workers to share them.

Uniprocessor, Multicore, Cluster

A multicore chip is one kind of multiprocessor. A cluster links whole computers together.

Key Vocabulary

Why Use Multiple Processors?

• Higher performance: more work done at the same time.
• Better energy efficiency: many small cores beat one big core in performance per watt.
• Handle more tasks: a server can serve many users at once.
• Better availability: if one of 4 processors fails, the other 3 can keep working (just more slowly).

Two Ways to Use Many Processors

• Task-level (job-level) parallelism: run different, independent programs at the same time. Easy, with no changes to the programs.
• Parallel processing program: one program split across several processors. This is harder, because the programmer must divide the work.

Concurrency vs. Parallelism

• Concurrency (a property of the software): several tasks are in progress during overlapping periods. They can take turns on one core.
• Parallelism (a property of the hardware): several operations execute at the same instant on multiple cores.
• A program can be concurrent without running in parallel.

Sequential vs. Concurrent Software

Hardware and Software Are Separate Questions

1. Hardware: can the computer execute several operations at the same time? (serial vs. parallel)
2. Software: is the program written as one sequence or as many tasks? (sequential vs. concurrent)

The challenge: make both kinds of software run fast on parallel hardware.

Quick Quiz

1. A computer has four cores on one chip. What is it called?
2. A single-core processor switches between a browser and a music player. Concurrency or parallelism?
3. Several separate computers communicate over a network to solve a big problem. What is this called?
4. True or false: To benefit from a multiprocessor, an application must be concurrent.

The Reporter Analogy

Hardware gives us the opportunity for parallelism. Software must use it, and that is the hard part.

Eight reporters want to write one news story 8× faster.

More reporters (processors) → these problems get worse. Parallel programs must be both correct and efficient.

Amdahl’s Law

The part of a program you cannot speed up limits the total speed-up.

\[ \text{Time after} = \frac{\text{Time affected}}{\text{Amount of improvement}} + \text{Time unaffected} \]

If the original time is 1 and a fraction \(F\) can run in parallel on \(P\) processors:

\[ \text{Speed-up} = \frac{1}{(1 - F) + \dfrac{F}{P}} \]

• \((1 - F)\) = the sequential part, which no amount of processors can shrink.
• As \(P \to \infty\): \(\text{Speed-up} \to \dfrac{1}{1 - F}\)

Amdahl’s Law: The Picture

Even with only 1% sequential code, 128 processors give less than 60×.

Example 1: Speed-up Challenge

Goal: a speed-up of 90× with 100 processors. How much of the program can be sequential?

• Set up: \(\;90 = \dfrac{1}{(1 - F) + \dfrac{F}{100}}\)
• Flip both sides: \(\;(1 - F) + 0.01F = \dfrac{1}{90} = 0.0111\)
• Simplify: \(\;1 - 0.99F = 0.0111 \;\Rightarrow\; F = \dfrac{0.9889}{0.99} \approx 0.999\)
• So the sequential part must be only about 0.1% of the original time!

Example 2: Make the Problem Bigger

Sum 10 scalar variables (sequential) + sum two matrices (parallel). One addition takes time \(t\).

Strong vs. Weak Scaling

Real-world example: a bank benchmark (TPC-C) must add more customer accounts to report more transactions per minute. Customers don’t use the ATM 100× more often just because the bank bought a faster computer.

Example 3: Load Balancing

Same 100×100 problem on 100 processors. Ideal: each processor does 1% of the work → 91×.

What if one processor gets more than its share?

Check Yourself

True or false: Strong scaling is not bound by Amdahl’s law.

Shared Memory Multiprocessor (SMP)

• One address space: any processor can load or store any memory location.
• Processors communicate implicitly through shared variables.
• Hardware cache coherence keeps every core’s view of memory consistent (Chapter 5).
• Most multicore chips work this way.

UMA vs. NUMA

Synchronization and Locks

When processors share data, they must coordinate.

• Synchronization: coordinating two or more processes so that one does not use data before another has finished writing it.
• Lock: only one processor at a time can hold it.
  • Processor acquires the lock → uses the shared data → releases the lock.
  • Others wait until the lock is released.

Example: Sum 100,000 Numbers on an SMP

100 processors, numbered Pn = 0 … 99. Each sums its own 1,000 numbers.

Step 1: each processor sums its own slice

sum[Pn] = 0;
for (i = 1000*Pn; i < 1000*(Pn+1); i = i + 1)
    sum[Pn] = sum[Pn] + A[i];     /* sum the assigned area */

• All data is in one shared array A; each processor just starts at a different address.
• Now we have 100 partial sums. How do we combine them?

Step 2: Reduction (Divide and Conquer)

A reduction combines many values into one.

100 partial sums need only \(\lceil\log_2 100\rceil = 7\) rounds instead of 99 sequential additions.

Reduction Code

half = 100;                       /* 100 processors */
repeat
    synch();                      /* wait for partial sums to finish */
    if (half % 2 != 0 && Pn == 0)
        sum[0] = sum[0] + sum[half-1];   /* odd case: P0 takes the extra one */
    half = half / 2;              /* dividing line on who sums */
    if (Pn < half)
        sum[Pn] = sum[Pn] + sum[Pn+half];
until (half == 1);                /* final sum is in sum[0] */

• synch() is a barrier: no processor continues until all arrive.
• Without it, a processor might read a partial sum before it is ready.
• i and half must be private variables, one copy per processor.

Check Yourself

True or false: Shared memory multiprocessors cannot take advantage of job-level parallelism.

Message-Passing Multiprocessor

• Each processor has a private address space.
• To share data, a processor must explicitly send a message, and another must receive it.
• send / receive also act as synchronization: the receiver waits until the message arrives.

Clusters

A cluster is a set of ordinary computers connected by standard network switches and cables.

• Each node runs its own copy of the operating system.
• Used for web servers, search engines, email, databases and science. Grid computing spreads them worldwide (e.g., SETI@home).

Example: Memory Efficiency

• Shared memory: 1 machine with 20 GB; the OS uses 1 GB.
• Cluster: 5 machines with 4 GB each; each OS uses 1 GB.

How much more memory is available for user programs with shared memory?

Example: Sum 100,000 Numbers with Message Passing

Step 1: One processor sends 1,000 numbers to each of the 100 nodes.

Step 2: Every node sums its local array:

sum = 0;
for (i = 0; i < 1000; i = i + 1)   /* loop over local array */
    sum = sum + AN[i];

Now each of the 100 nodes holds one partial sum in its own memory.

Message Passing: The Reduction

Step 3: combine the partial sums by sending messages:

limit = 100; half = 100;              /* 100 processors */
repeat
    half = (half + 1) / 2;            /* send vs. receive dividing line */
    if (Pn >= half && Pn < limit) send(Pn - half, sum);
    if (Pn < (limit / 2))          sum = sum + receive();
    limit = half;                     /* upper limit of senders */
until (half == 1);                    /* final sum is on P0 */

Upper half sends, lower half receives and adds: same divide-and-conquer idea, but with messages.

Shared Memory vs. Message Passing

Check Yourself

1. True or false: Like SMPs, message-passing computers rely on locks for synchronization.
2. True or false: Unlike SMPs, message-passing computers need multiple copies of the parallel program and the operating system.

What Is Hardware Multithreading?

Idea: when one thread stalls (e.g., waiting on a cache miss), let another thread use the processor.

To make this possible, the hardware keeps a separate copy of each thread’s state:

• its own register file
• its own program counter (PC)

Memory is shared through virtual memory.

Three Styles of Multithreading

Example: Intel’s Nehalem cores run 2 threads with SMT (Intel calls it Hyper-Threading).

Who Uses the Issue Slots?

A 4-wide superscalar processor running four threads (illustrative).

Reading the Picture

• No multithreading: a long stall leaves the whole processor idle, and limited ILP wastes slots.
• Coarse-grained: switches after a stall, so fewer fully empty cycles, but switching still costs cycles.
• Fine-grained: a different thread every cycle, so almost no fully empty cycles, but each cycle still only uses one thread’s instructions.
• SMT: many threads fill the slots in the same cycle, so the fewest wasted slots.

Check Yourself

1. True or false: Both multithreading and multicore rely on parallelism to get more efficiency from a chip.
2. True or false: Simultaneous multithreading uses threads to improve resource utilization of a dynamically scheduled, out-of-order processor.

Flynn’s Classification

Classify hardware by the number of instruction streams and data streams:

• SPMD (Single Program, Multiple Data) is the usual way to program a MIMD machine: write one program, run it on all processors, and use if (Pn == …) so processors can do different parts.
• The sum examples earlier were SPMD programs!

SIMD: One Instruction, Many Data

A single SIMD instruction can, for example, add 64 pairs of numbers at once using 64 ALUs.

• All units run the same instruction from one PC, which feels like ordinary sequential programming.
• Only one copy of the code and one control unit are needed.
• Works best on arrays in for loops → data-level parallelism.
• Works worst on switch / case statements: units with the “wrong” case sit idle, so you get about \(1/n\) performance for \(n\) cases.

SIMD in x86: Multimedia Extensions (MMX, SSE)

One wide ALU can be split into smaller ALUs:

• Added to speed up multimedia (images, audio, video).
• The data width is encoded in the opcode, so every new width means new instructions → hundreds of SSE instructions.

Vector Architecture

An older, more elegant form of SIMD (Cray supercomputers).

1. Load a whole vector of data from memory into a vector register.
2. Operate on the elements in a pipelined ALU.
3. Store the results back to memory.

Example: 32 vector registers, each holding 64 elements of 64 bits.

Example: DAXPY (Y = a × X + Y)

\(X\) and \(Y\) are vectors of 64 doubles; \(a\) is a scalar.

      l.d   $f0,a($sp)     ;load scalar a
      addiu r4,$s0,#512    ;upper bound
loop: l.d   $f2,0($s0)     ;load x(i)
      mul.d $f2,$f2,$f0    ;a * x(i)
      l.d   $f4,0($s1)     ;load y(i)
      add.d $f4,$f4,$f2    ;a*x(i) + y(i)
      s.d   $f4,0($s1)     ;store y(i)
      addiu $s0,$s0,#8     ;next x
      addiu $s1,$s1,#8     ;next y
      subu  $t0,r4,$s0     ;compute bound
      bne   $t0,$zero,loop ;done?

l.d     $f0,a($sp)  ;load scalar a
lv      $v1,0($s0)  ;load vector x
mulvs.d $v2,$v1,$f0 ;vector * scalar
lv      $v3,0($s1)  ;load vector y
addv.d  $v4,$v2,$v3 ;add y to product
sv      $v4,0($s1)  ;store result

Why Vectors Are Fast (and Save Power)

• One instruction = a whole loop → far less instruction fetch and decode.
• Elements in a vector are independent → no hazard checks inside a vector.
• Hazards are checked once per vector, not once per element → about 64× fewer stalls than the scalar loop in DAXPY.
• Known memory access pattern → memory latency is paid once per vector.
• No loop branch → no control hazards.

Check Yourself

True or false: As exemplified in the x86, multimedia extensions can be thought of as a vector architecture with short vectors that supports only sequential (contiguous) vector data transfers.

How GPUs Differ from CPUs

GPUs grew out of video games: drawing millions of independent pixels.

• CPU + GPU = a heterogeneous system (not all processors are the same).
• Don’t forget the time to copy data between CPU and GPU memory.

NVIDIA GPU Vocabulary (Tesla, 2008)

Example: GeForce 8800 GTX by the Numbers

Peak compute: 16 multiprocessors, 8 SPs each, 1 multiply-add (= 2 FLOPs) per SP per clock, 1.35 GHz

\[ 16 \times 8 \times 2 \times 1.35 \text{ GHz} = 345.6 \text{ GFLOP/s} \]

Peak memory bandwidth: 6 memory partitions, 8 bytes wide, 2 transfers per clock, 0.9 GHz

\[ 6 \times 8 \text{ B} \times 2 \times 0.9 \text{ GHz} = 86.4 \text{ GB/s} \]

Branch Divergence

Threads in a warp share one instruction fetch unit.

if (x[i] > 0)
    y[i] = a;      // some threads take this path ...
else
    y[i] = b;      // ... others take this one

• If threads in a warp take different paths, the warp runs both paths, one after another, with some threads idle.
• Best performance: all 32 threads in a warp follow the same path.
• This is the same weakness as SIMD with switch statements.

Check Yourself

True or false: GPUs rely on graphics DRAM chips to reduce memory latency and thereby increase performance on graphics applications.

Connecting the Cores

Multicore chips need a network on chip. The design trade-off is cost vs. performance:

• Cost: number of switches, links per switch, wires per link, link length
• Performance: latency, throughput, contention, fault tolerance, power

Measuring a Network

Comparing Topologies

\(P\) = number of processors; each link has bandwidth 1.

• Ring: cheap, and many messages can travel at once, but messages may hop through other nodes.
• Fully connected: fastest but very expensive (links grow as \(P^2\)).
• 2-D mesh and n-cube sit in between and appear in real machines.

Multistage Networks: Crossbar vs. Omega

Switches sit between processors instead of at each node.

Parallel Benchmarks

Fixed rules keep benchmarks fair but block innovation; the Berkeley patterns allow any implementation.

The Roofline Model: Key Quantity

Arithmetic intensity (AI) = how much computation you do per byte you bring from memory:

\[ \text{AI} = \frac{\text{floating-point operations}}{\text{bytes transferred to/from main memory}} \]

• Low AI (e.g., sparse matrix): mostly moving data → memory is the limit.
• High AI (e.g., dense matrix multiply): lots of math per byte → compute is the limit.

The Roofline Model

\[ \text{Attainable GFLOP/s} = \min\left(\text{Peak memory BW} \times \text{AI},\;\; \text{Peak floating-point performance}\right) \]

Example machine: AMD Opteron X2 (peak 16 GFLOP/s, memory bandwidth 16 GB/s).

Reading the Roofline

• The slanted roof = memory limit. The flat roof = compute limit.
• Draw a vertical line at your kernel’s AI, and performance can’t go above the roof there.
  • Kernel 1 (AI = 0.5) hits the slanted roof → memory-bound → at most \(16 \times 0.5 = 8\) GFLOP/s.
  • Kernel 2 (AI = 4) hits the flat roof → compute-bound → at most 16 GFLOP/s.
• Ridge point (where the roofs meet):
  • Far left → almost any program can reach peak.
  • Far right → only programs with very high AI can reach peak.

Your Turn: Roofline Practice

A machine has peak = 32 GFLOP/s and memory bandwidth = 8 GB/s.

1. Where is the ridge point?
2. What is the attainable performance for a kernel with AI = 0.25?
3. … and for a kernel with AI = 8?

Optimizing with the Roofline: Ceilings

If your program is far below the roof, the roofline tells you what to fix first.

You can also raise AI itself, for example with better cache use, so fewer bytes come from memory.

Real Stuff: Four Multicores (2008)

*Fully optimized. SpMV = sparse matrix × vector; LBMHD = fluid dynamics on a grid.

• Highest peak → lowest real performance: the Xeon’s memory system was the bottleneck.
• UltraSPARC T2 was the easiest to program well; Cell was the fastest but hardest.

Fallacies

Fallacy: “Amdahl’s law doesn’t apply to parallel computers.”

• One group claimed to “break” Amdahl’s law with linear speed-up on 1000 processors.
• They used weak scaling: they did 1000× more work in the same time, not the same work 1000× faster.
• Amdahl’s law still applies, but bigger problems help.

Fallacy: “Peak performance tracks observed performance.”

• The Xeon had the highest peak and the slowest results.
• Marketing multiplies peak per core by the number of cores, which assumes perfect speed-up!

Pitfall

Pitfall: Not rewriting software for a multiprocessor.

• An OS protected the whole page table with one lock. That was fine on one CPU, but on many CPUs all page allocations were serialized.
• Fix: use finer-grained locks (lock smaller pieces).

Summary

• Power wall → the industry bet its future on multicore. Programmers must think in parallel.
• Multicore = cores on one chip; cluster = computers on a network. Concurrency ≠ parallelism.
• Amdahl’s law: the sequential part limits speed-up; weak scaling and load balancing help.
• Shared memory (implicit, easy to port) vs. message passing (explicit, scalable, available).
• Multithreading (fine, coarse, SMT) keeps a core busy when a thread stalls.
• SIMD / vector / GPU exploit data-level parallelism in loops.
• Networks: judge them by bisection bandwidth.
• Roofline: performance ≤ min(memory BW × AI, peak FLOPs).

The Most Important Lesson

Hardware provides the opportunity for parallelism. Software must be designed to take advantage of it.

• More cores only help if the work can be divided, balanced and coordinated.
• Keep the sequential part small (Amdahl’s law), and know whether your program is limited by compute or memory (roofline).

Exit Ticket

1. A program is 20% sequential. What is the maximum speed-up with infinitely many processors?
2. Name one advantage of a cluster over a shared memory multiprocessor.
3. A kernel has AI = 0.5 on the Opteron X2 roofline (16 GB/s, 16 GFLOP/s). Is it memory-bound or compute-bound?