Parallelism is coming to everything from laptops to high-end servers. It will exist at multiple levels with multiple cores per chip, multiple sockets per box, and multiple boxes per system (cluster). While only some programmers will need to worry about clusters, all programmers will need to worry about shared memory multiprocessors. And the way you program these systems is with multi-threading.

---

By Tim Mattson

Get the basics about parallel algorithms, parallel programming APIs, and the tools required to start writing your own parallel programs.

Today’s leading hardware designs depend on parallelism; that is, multiple activities running at the same time. Performance inside a CPU depends on instruction level parallelism. All the major CPU vendors are putting multiple cores on each die. Typical servers have multiple sockets per box. Clusters of servers and grids of systems spanning the globe are becoming commonplace. Parallelism has gone main stream and if software developers want to stay relevant, they had better learn to deal with it.

A good place to start is by taking a closer look at parallel hardware. Supercomputing designers have been building high performance computers by connecting smaller computers into larger systems since the early 80’s. Most of these designs fall into one of two camps: distributed memory multiple instruction multiple data (MIMD) or shared memory MIMD architectures. In MIMD, each processing element has its own stream of instructions and its own stream of data. We call the system a "shared memory" system when there is a single address space that all the processing elements share. If the memories are distinct and the processing elements can only interact through the networks connecting them, we call the system a "distributed memory" system.