I am honored to be with you today at your commencement from one of the finest universities in the world. I never graduated from college. Truth be told, this is the closest I’ve ever gotten to a college graduation. Today I want to tell you three stories from my life. That’s it. No big deal. Just three stories.

The first story is about connecting the dots.

I dropped out of Reed College after the first 6 months, but then stayed around as a drop-in for another 18 months or so before I really quit. So why did I drop out?

It started before I was born. My biological mother was a young, unwed college graduate student, and she decided to put me up for adoption. She felt very strongly that I should be adopted by college graduates, so everything was all set for me to be adopted at birth by a lawyer and his wife. Except that when I popped out they decided at the last minute that they really wanted a girl. So my parents, who were on a waiting list, got a call in the middle of the night asking: “We have an unexpected baby boy; do you want him?” They said: “Of course.” My biological mother later found out that my mother had never graduated from college and that my father had never graduated from high school. She refused to sign the final adoption papers. She only relented a few months later when my parents promised that I would someday go to college.

And 17 years later I did go to college. But I naively chose a college that was almost as expensive as Stanford, and all of my working-class parents’ savings were being spent on my college tuition. After six months, I couldn’t see the value in it. I had no idea what I wanted to do with my life and no idea how college was going to help me figure it out. And here I was spending all of the money my parents had saved their entire life. So I decided to drop out and trust that it would all work out OK. It was pretty scary at the time, but looking back it was one of the best decisions I ever made. The minute I dropped out I could stop taking the required classes that didn’t interest me, and begin dropping in on the ones that looked interesting.

It wasn’t all romantic. I didn’t have a dorm room, so I slept on the floor in friends’ rooms, I returned coke bottles for the 5¢ deposits to buy food with, and I would walk the 7 miles across town every Sunday night to get one good meal a week at the Hare Krishna temple. I loved it. And much of what I stumbled into by following my curiosity and intuition turned out to be priceless later on. Let me give you one example:

Reed College at that time offered perhaps the best calligraphy instruction in the country. Throughout the campus every poster, every label on every drawer, was beautifully hand calligraphed. Because I had dropped out and didn’t have to take the normal classes, I decided to take a calligraphy class to learn how to do this. I learned about serif and san serif typefaces, about varying the amount of space between different letter combinations, about what makes great typography great. It was beautiful, historical, artistically subtle in a way that science can’t capture, and I found it fascinating.

None of this had even a hope of any practical application in my life. But ten years later, when we were designing the first Macintosh computer, it all came back to me. And we designed it all into the Mac. It was the first computer with beautiful typography. If I had never dropped in on that single course in college, the Mac would have never had multiple typefaces or proportionally spaced fonts. And since Windows just copied the Mac, it’s likely that no personal computer would have them. If I had never dropped out, I would have never dropped in on this calligraphy class, and personal computers might not have the wonderful typography that they do. Of course it was impossible to connect the dots looking forward when I was in college. But it was very, very clear looking backwards ten years later.

Again, you can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future. You have to trust in something — your gut, destiny, life, karma, whatever. This approach has never let me down, and it has made all the difference in my life.

My second story is about love and loss.

I was lucky — I found what I loved to do early in life. Woz and I started Apple in my parents garage when I was 20. We worked hard, and in 10 years Apple had grown from just the two of us in a garage into a $2 billion company with over 4000 employees. We had just released our finest creation — the Macintosh — a year earlier, and I had just turned 30. And then I got fired. How can you get fired from a company you started? Well, as Apple grew we hired someone who I thought was very talented to run the company with me, and for the first year or so things went well. But then our visions of the future began to diverge and eventually we had a falling out. When we did, our Board of Directors sided with him. So at 30 I was out. And very publicly out. What had been the focus of my entire adult life was gone, and it was devastating.

I really didn’t know what to do for a few months. I felt that I had let the previous generation of entrepreneurs down – that I had dropped the baton as it was being passed to me. I met with David Packard and Bob Noyce and tried to apologize for screwing up so badly. I was a very public failure, and I even thought about running away from the valley. But something slowly began to dawn on me — I still loved what I did. The turn of events at Apple had not changed that one bit. I had been rejected, but I was still in love. And so I decided to start over.

I didn’t see it then, but it turned out that getting fired from Apple was the best thing that could have ever happened to me. The heaviness of being successful was replaced by the lightness of being a beginner again, less sure about everything. It freed me to enter one of the most creative periods of my life.

During the next five years, I started a company named NeXT, another company named Pixar, and fell in love with an amazing woman who would become my wife. Pixar went on to create the worlds first computer animated feature film, Toy Story, and is now the most successful animation studio in the world. In a remarkable turn of events, Apple bought NeXT, I returned to Apple, and the technology we developed at NeXT is at the heart of Apple’s current renaissance. And Laurene and I have a wonderful family together.

I’m pretty sure none of this would have happened if I hadn’t been fired from Apple. It was awful tasting medicine, but I guess the patient needed it. Sometimes life hits you in the head with a brick. Don’t lose faith. I’m convinced that the only thing that kept me going was that I loved what I did. You’ve got to find what you love. And that is as true for your work as it is for your lovers. Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do. If you haven’t found it yet, keep looking. Don’t settle. As with all matters of the heart, you’ll know when you find it. And, like any great relationship, it just gets better and better as the years roll on. So keep looking until you find it. Don’t settle.

My third story is about death.

When I was 17, I read a quote that went something like: “If you live each day as if it was your last, someday you’ll most certainly be right.” It made an impression on me, and since then, for the past 33 years, I have looked in the mirror every morning and asked myself: “If today were the last day of my life, would I want to do what I am about to do today?” And whenever the answer has been “No” for too many days in a row, I know I need to change something.

Remembering that I’ll be dead soon is the most important tool I’ve ever encountered to help me make the big choices in life. Because almost everything — all external expectations, all pride, all fear of embarrassment or failure – these things just fall away in the face of death, leaving only what is truly important. Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose. You are already naked. There is no reason not to follow your heart.

About a year ago I was diagnosed with cancer. I had a scan at 7:30 in the morning, and it clearly showed a tumor on my pancreas. I didn’t even know what a pancreas was. The doctors told me this was almost certainly a type of cancer that is incurable, and that I should expect to live no longer than three to six months. My doctor advised me to go home and get my affairs in order, which is doctor’s code for prepare to die. It means to try to tell your kids everything you thought you’d have the next 10 years to tell them in just a few months. It means to make sure everything is buttoned up so that it will be as easy as possible for your family. It means to say your goodbyes.

I lived with that diagnosis all day. Later that evening I had a biopsy, where they stuck an endoscope down my throat, through my stomach and into my intestines, put a needle into my pancreas and got a few cells from the tumor. I was sedated, but my wife, who was there, told me that when they viewed the cells under a microscope the doctors started crying because it turned out to be a very rare form of pancreatic cancer that is curable with surgery. I had the surgery and I’m fine now.

This was the closest I’ve been to facing death, and I hope it’s the closest I get for a few more decades. Having lived through it, I can now say this to you with a bit more certainty than when death was a useful but purely intellectual concept:

No one wants to die. Even people who want to go to heaven don’t want to die to get there. And yet death is the destination we all share. No one has ever escaped it. And that is as it should be, because Death is very likely the single best invention of Life. It is Life’s change agent. It clears out the old to make way for the new. Right now the new is you, but someday not too long from now, you will gradually become the old and be cleared away. Sorry to be so dramatic, but it is quite true.

Your time is limited, so don’t waste it living someone else’s life. Don’t be trapped by dogma — which is living with the results of other people’s thinking. Don’t let the noise of others’ opinions drown out your own inner voice. And most important, have the courage to follow your heart and intuition. They somehow already know what you truly want to become. Everything else is secondary.

When I was young, there was an amazing publication called The Whole Earth Catalog, which was one of the bibles of my generation. It was created by a fellow named Stewart Brand not far from here in Menlo Park, and he brought it to life with his poetic touch. This was in the late 1960′s, before personal computers and desktop publishing, so it was all made with typewriters, scissors, and polaroid cameras. It was sort of like Google in paperback form, 35 years before Google came along: it was idealistic, and overflowing with neat tools and great notions.

Stewart and his team put out several issues of The Whole Earth Catalog, and then when it had run its course, they put out a final issue. It was the mid-1970s, and I was your age. On the back cover of their final issue was a photograph of an early morning country road, the kind you might find yourself hitchhiking on if you were so adventurous. Beneath it were the words: “Stay Hungry. Stay Foolish.” It was their farewell message as they signed off. Stay Hungry. Stay Foolish. And I have always wished that for myself. And now, as you graduate to begin anew, I wish that for you.

Stay Hungry. Stay Foolish.

Thank you all very much.

Unix/Linux Administration
Logical Volume Management Guide

©2005 Wayne Pollock, Tampa Florida USA.
All Rights Reserved.

Adapted from the LVM How-to page
from The Linux Documentation Project website.

LVM (Logical Volume Management) Overview:

In the olden days computer disks were small compared to the data set sizes that were needed.  A solution is to make several physical disks appear virtually (or logically) as a single much larger disk.  A large filesystem could then be created on that virtual disk.

Later hard disk sizes grew large enough that it made sense to do the opposite: make one physical disk appear as several virtual disks.  Each virtual disk holds a filesystem independently of the others.  Today such virtual disks are called disk “partitions“.

Now we have come full circle.  Large data warehouse applications require very large filesystems to hold the database data files.  To support this sort of application the old idea of combining several disks into one has been resurrected.  Novell Netware supported this feature since the 1990s.  The physical disk (or selected disk partitions) are formatted to be “physical volume segments“.  All added physical volume segments become part of a single large virtual disk.  The administrator can then create logical “volumes“, that is, a filesystem.  The exciting part is that if some volume is low on space, you can extend the virtual disk by adding another physical volume segment to it, and then increase the (logical) volume’s size.  This operation is fast and doesn’t disturb the existing data or other partitions (or volumes)!

The modern Unix (and Linux) version of this idea is called “Logical Volume Management” (or “LVM”).  LVM allows the administrator to

• use and allocate disk space more efficiently and flexibly
• move logical volumes between different physical devices
• have very large logical volumes span a number of physical devices
• take snapshots of whole filesystems easily, allowing on-line backup of those filesystems
• replace on-line drives without interrupting services

Linux also supports software RAID, which, like LVM, can be used to provide disk striping.  The two systems are independent of each other.  So you can use RAID to provide striping and use that RAID volume as a physical volume for LVM.  There is no reason to use both software RAID and LVM, although it can be done.  However it does make good sense to use hardware RAID and LVM together.

As a quick example of what can be done, suppose you need to grow some filesystem by 500 MB.  Without LVM, you must either have a spare 500 MB on the same physical disk or the entire filesystem must be moved to another (larger) disk.  Even if you have the space, all the other partitions (and their filesystems) must be moved, so the 500 MB is at the end of the partition you wish to grow.  Then all the data must be backed up, the partition changed, and the old data restored.  While today there are tools such as parted that can manage this for regular partitions (not LVs), using LVM you can quickly and safely grow a filesystem in this simple way:

  root# umount /dev/vg1/lv1
  root# e2fsadm /dev/vg1/lv1 -L+500M
  root# mount /dev/vg1/lv1

The free space can be anywhere, even on another disk!

LVM for Linux has had two major versions (so far).  Both have many more features and controls than the older Novell scheme.  While the additional controls and features aren’t always needed, you still must understand them in order to setup and manage LVM.  LVM is also available in recent Solaris and other Unix systems, however different terms may be used for the same concepts.  The Veritas Volume Manager, (“VVM” or “VxVM”) is a proprietary but popular logical volume manager from Veritas (now part of Symantec).  It is available for Windows, AIX, Solaris, Linux, and HP-UX (where it is the system default LVM).  Also take a look at the Enterprise Volume Management System (a sourceforge.net project).

Basic Concepts and Terminology:

Figure 1

With LVM, physical volume segments are simply called “physical volumes” (or “PVs“).  These PVs are usually entire disks but may be disk partitions.  The PVs in turn are combined to create one or more large virtual disks called “volume groups” (or “VGs“).  While you can create many VGs, one may be sufficient.  A VG can grow or shrink by adding or removing PVs from it.  VGs appear to be block devices, similar to other disks such as /dev/hda.  In fact each VG can be referred to by the name “/dev/VG_name“.

Once you have one or more volume groups you can create one or more virtual partitions called “logical volumes” (or “LVs“).  Note each LV must fit entirely within a single VG.  LVs appear to be block devices similar to disk partitions such as /dev/hda1, with entries named “/dev/VG_name/LV_name“.  LVs have a number of parameters that can be set (and later most can be changed) that can affect disk I/O performance, including extent size, chunk size, stripe size and stripe set size, and read-ahead.  These are discussed below.

Table 1: LVM Acronyms

Finally, you can create any type of filesystem you wish on the logical volume, including as swap space.  Note that some filesystems are more useful with LVM than others.  For example not all filesystems support growing and shrinking.  ext2, ext3, xfs, and reiserfs do support such operations and would be good choices.  The relationship between PVs, VGs, and LVs is illustrated in figure 1.  Table 1 summarizes these acronyms.

Extents:

When creating a volume group from one or more physical volumes, you must specify the size of the “extents” of each of the physical volumes that make up the VG.  Each extent is a single contiguous chunk of disk space, typically 4M in size, but can range from 8K to 16G in powers of 2 only.  (Extents are analogous to disk blocks or clusters.)  The significance of this is that the size of logical volumes are specified as a number of extents.  Logical volumes can thus grow and shrink in increments of the extent size.  A volume group’s extent size cannot be changed after it is set.

The system internally numbers the extents for both logical and physical volumes.  These are called logical extents (or LEs) and physical extents (or PEs), respectively.  When a logical volume is created a mapping is defined between logical extents (which are logically numbered sequentially starting at zero) and physical extents (which are also numbered sequentially).

To provide acceptable performance the extent size must be a multiple of the actual disk cluster size (i.e., the size of the smallest chunk of data that can be accessed in a single disk I/O operation).  In addition some applications (such as Oracle database) have performance that is very sensitive to the extent size.  So setting this correctly also depends on what the storage will be used for, and is considered part of the system administrator’s job of tuning the system.

Booting Consideration with LVM:

LVM can be use with RAID.  LVM can be used to hold all filesystems.  However special considerations apply when using LVs for the boot and root filesystems.  This is because the BIOS code in the ROM of the motherboard of your computer must be able to locate and load the kernel.  So if the boot partition was a LV, the BIOS would need to know about PVs, VGs, and LVs, and it probably doesn’t.  Unless you are using some custom BIOS you must not make the bootable partition an LV.

The root partition (if it isn’t also the boot partition) may be a logical volume.  However this means the kernel must access the root partition before it can load any (e.g., LVM) kernel modules.  Thus the modules for LVM must be compiled into the kernel.  This is rarely the case with standard distributions!  (There is a similar issue with SCSI drivers, as most kernels only compile in the IDE drivers.)  For this reason, as well as allowing a filesystem to be accessed by another operating system (yes there are ext2 drivers available for Windows), some system administrators prefer to make the root filesystem on a regular partition rather than on a logical volume.  Note in this case you can make a single root+boot partition.

The solution to using a logical volume for your root filesystem (as it is with SCSI) is either to build a custom kernel with the correct drivers compiled in, or to make sure the system loads a RAM disk initially, known as initrd, which contains all the correct modules.  This RAM disk then loads the system as normal, and goes away.  Creating a ramdisk on Linux is simple using the mkinitrd script.  Just run this command (as root). You need to know the kernel version, and then you must update grub.conf to use the ramdisk:

/root# KERNEL_VERSION=`uname -r`
/root# mkinitrd -v initrd.$KERNEL_VERSION $KERNEL_VERSION
/root# mv initrd.$KERNEL_VERSION /boot
/root# vi /boot/grub/grub.conf  # or /boot/grub/menu.lst

Before logical volumes can be mounted, the LVM driver must be loaded (or compiled in) to the kernel.  Next all physical volumes on all available drives must be found and examined, in order to determine all the volume groups.  Finally the volume groups must be activated, which causes the kernel to recognize the various block devices.  Only then can the filesystems within logical volumes be mounted. So most systems add code similar to the following to the boot up scripts (typically the rc.sysinit script):

# LVM2 initialization
if [ -x /sbin/lvm.static ]  # Check for LVM v2
then
   # make sure device mapper (LVM2) kernel module is loaded:
   if ! grep -q "device-mapper" /proc/devices 2>/dev/null
   then
      modprobe dm-mod >/dev/null 2>&1
   fi
   # Cleanup and then recreate device mapper control file:
   /bin/rm -f /dev/mapper/control
   echo "mkdmnod" | /sbin/nash --quiet >/dev/null 2>&1
   if [ -c /dev/mapper/control ]  # if LVM2 is loaded:
   then  # Check for any physical volumes:
      if /sbin/lvm.static vgscan > /dev/null 2>&1
      then
         echo "Setting up Logical Volume Management:"
         # Activate volume groups and re-create all /dev entries:
         /sbin/lvm.static vgchange -a y && /sbin/lvm vgmknodes
      fi
   fi
fi

You may want to edit the file /etc/init.d/halt to deactivate the volume groups at shutdown.  However this shouldn’t be necessary when using LVM version 2.  To deactivate volume groups, insert the following near the end of this file (just after the filesystems are mounted read-only and before the comment that says “Now halt or reboot“):

# Deactivate LVM:
if [ -x /sbin/lvm.static ]  # Check for LVM v2
then
   echo "Deactivating LVM volume groups:"
   /sbin/lvm.static vgchange -a n
fi

Like all storage devices data may become corrupted over time.  LVM provides a command “vgck” you can use to periodically check the consistency of your volume groups.  It may pay to add this command to the bootup scripts.

Linear and Striped Mapping:

Let’s suppose we have a volume group called VG1, and this volume group has a physical extent size of 4M.  Suppose too this volume group is composed of one disk partition /dev/hda1 and one whole disk /dev/hdb.  These will become physical volumes PV1 and PV2 (more meaningful names for a particular scenario can be given if desired).

The PVs are different sizes and we get 99 (4M) extents in PV1 and 248 extents in PV2, for a total of 347 extents in VG1.  Now any number of LVs of any size can be created from the VG, as long as the total number of extents of all LVs sums to no more than 347.  To make the LVs appear the same as regular disk partitions to the filesystem software, the logical extents are numbered sequentially within the LV.  However some of these LEs may be stored in the PEs on PV1 and others on PV2.  For instance LE[1] of some LV in VG1 could map onto PE[51] of PV1, and thus data written to the first 4M of the LV is in fact written to the 51st extent of PV1.

When creating LVs an administrator can choose between two general strategies for mapping logical extents onto physical extents:

1. Linear mapping will assign a range of PE’s to an area of an LV in order (e.g., LE 1–99 map to PV1‘s PEs, and LE 100–347 map onto PV2‘s PEs).
2. Striped mapping will interleave the disk blocks of the logical extents across a number of physical volumes.  You can decide the number of PVs to stripe across (the stripe set size), as well as the size of each stripe.

When using striped mapping, all PVs in the same stripe set need to be the same size.  So in our example the LV can be no more than 198 (99 + 99) extents in size.  The remaining extents in PV2 can be used for some other LVs, using linear mapping.

The size of the stripes is independent of the extent size, but must be a power of 2 between 4K and 512K.  (This value n is specified as a power of 2 in this formula: (2^n) × 1024 bytes, where 2 ≤ n ≤ 9.)  The stripe size should also be a multiple of the disk sector size, and finally the extent size should be a multiple of this stripe size.  If you don’t do this, you will end up with fragmented extents (as the last bit of space in the extent will be unusable).

Tables 2 and 3 below illustrate the differences between linear and striped mapping.  Suppose you use a stripe size of 4K, an extent size of 12K, and a stripe set of 3 PVs (PVa, PVb, and PVc), each of which is 100 extents.  Then the mapping for an LV ( whose extents we’ll call LV1, LV2, …) to PVs (whose extents we’ll call PVa1, PVa2, …, PVb1, PVb2, …, PVc1, PVc2, …) might look something like the following.  (In this table the notation means volume_name extent_number . stripe_number):

Tables 2 and 3: Linear versus Striped Mapping

In certain situations striping can improve the performance of the logical volume but it can be complex to manage.  However note that striped mapping is useless and will in fact hurt performance, unless the PVs used in the stripe set are from different disks, preferably using different controllers.

(In version 1 of LVM LVs created using striping cannot be extended past the PVs on which they were originally created.  In the current version (LVM 2) striped LVs can be extended by concatenating another set of devices onto the end of the first set.  However this could lead to a situation where (for example) a single LV ends up as a 2 stripe set, concatenated with a linear (non-striped) set, and further concatenated with a 4 stripe set!

Snapshots:

A wonderful facility provided by LVM is a snapshot.  This allows an administrator to create a new logical volume which is an exact copy of an existing logical volume (called the original), frozen at some point in time.  This copy is read-only.  Typically this would be used when (for instance) a backup needs to be performed on the logical volume but you don’t want to halt a live system that is changing the data.  When done with the snapshot the system administrator can just unmount it and then remove it.  This facility does require that the snapshot be made at a time when the data on the logical volume is in a consistent state, but the time the original LV must be off-line is much less than a normal backup would take to complete.

In addition the copy typically only needs about 20% or less of the disk space of the original.  Essentially, when the snapshot is made nothing is copied.  However as the original changes, the updated disk blocks are first copied to the snapshot disk area before being written with the changes.  The more changes are made to the original, the more disk space the snapshot will need.

When creating logical volumes to be used for snapshots, you must specify the chunk size.  This is the size of the data block copied from the original to the snapshot volume.  For good performance this should be set to the size of the data blocks written by the applications using the original volume.  While this chunk size is independent of both the extent size and the stripe size (if striping is used), it is likely that the disk block (or cluster or page) size, the stripe size, and the chunk size should all be the same.  Note the chunk size must be a power of 2 (like the stripe size), between 4K and 1M.  (The extent size should be a multiple of this size.)

You should remove snapshot volumes as soon as you are finished with them, because they take a copy of all data written to the original volume and this can hurt performance.  In addition, if the snapshot volume fills up errors will occur.

LVM Administration — Commands and Procedures:

The lvm command permits the administrator to perform all LVM operations using this one interactive command, which includes built-in help and will remember command line arguments used from previous commands for the current command.  However each LVM command is also available as a stand-alone command (that can be scripted).  These are discussed briefly below, organized by task.  See the man page for the commands (or use the built-in help of lvm) for complete details.

Plan the Disk Layout:

Disk I/O is often the determining factor in overall system performance.  If your system has multiple disks and controllers, the correct strategy is to have them all used in parallel (that is, simultaneously) as much of the time as possible.  In addition you should aim to place files in such a way as to minimize disk head movement (and thus minimize seek time).

While the performance improvements are real they often aren’t significant.  Whether or not to worry about these issues depends upon the current performance, file size, type, and access patterns, and which applications are running.  Some ways to maximize performance are:

• To support a large number of users, say on a development system, you can use linear mapping of your logical volumes to a large number of physical volumes on different disks, hopefully using different controllers as well.
• On a production system running several different services, the files for each service should be placed on different physical disks (so for example the web server I/O won’t interfere with the FTP server I/O).
• On a database server the files for the following should be placed on different disks: tables, their indexes, log files, and frequently used tables (that don’t get cached in RAM).
• If possible it pays to keep the operating system files (i.e., boot, root, /var/log, swap) on one disk and the rest of the system on another, so the system’s disk I/O doesn’t interfere with the service’s I/O.
• Large files are often accessed sequentially.  Such access is most efficient when the data blocks of the file are contiguous, at the outer edge of the disk (cylinder zero).  In addition striping across different disks is very helpful in minimizing disk head movement and allowing parallel read/write operations.
• Large sequentially accessed files, such as those on a web or FTP server can also benefit from a LVM feature called read ahead.  This allows the next block to be read from disk to RAM at the same time as the current block, useful when the data is contiguous.  (Non-LVM disk read-ahead is also settable in Linux, by changing some parameters in /proc/sys/vm.)
• On the other hand, randomly accessed files benefit from a more central placement on a disk, as this will tend to minimize disk seek time.
• Using striped mapping can speed up access to large, sequentially accessed files, as more disks (and controllers) are used simultaneously.  Smaller, randomly accessed files don’t benefit much from striping.
• To avoid striping performance problems you should format one PV per whole disk.  LVM can’t tell if two PVs are on the same physical disk or not, so if you create a multiple PVs per disk and then create striped LVs, the stripes could be on different partitions on the same disk.  This would result in a decrease in performance rather than an increase.  If you do format a disk with multiple PVs, make sure no two of them are added to the same volume group if striped mapping will be used!
• For a production system it is recommended that you create one PV per whole disk, for Administrative convenience.  It’s easier to keep track of the hardware in a system if each real disk only appears once.  This becomes particularly true if a disk fails.

Other factors affect the disk I/O performance.  Often these other factors overshadow any performance gains from careful disk layout.  One example is the disk scheduler in the kernel.  The 2.6 version of the Linux kernel comes with two different schedulers you can select between.  In addition the kernel supports features such as read-ahead independently of any LVM settings.

Another issue is the disk controller settings.  Sometimes the disk schedules disk I/O regardless of the kernel.  Disks also have configurable settings, including DMA, buffering, etc.  These can be changed with “hdparam” and other utilities.

Other factors that affect performance include bus type and speed, and what other devices are attached to that bus.

Format Physical Volumes (PVs)

To initialize a disk or disk partition as a physical volume you just run the “pvcreate” command on the whole disk.  For example:

  pvcreate /dev/hdb

This creates a volume group descriptor at the start of the second IDE disk.  You can initialize several disks and/or partitions at once.  Just list all the disks and partitions on the command line you wish to format as PVs.

Sometimes this procedure may not work correctly, depending on how the disk (or partition) was previously formatted.  If you get an error that LVM can’t initialize a disk with a partition table on it, first make sure that the disk you are operating on is the correct one!  Once you have confirmed that /dev/hdb is the disk you really want to reformat, run the following dd command to erase the old partition table:

# Warning DANGEROUS!
# The following commands will destroy the partition table on the
# disk being operated on.  Be very sure it is the correct disk!
dd if=/dev/zero of=/dev/hdb bs=1k count=1 blockdev \
    --rereadpt /dev/hdb

For partitions run “pvcreate” on the partition:

    pvcreate /dev/hdb1

This creates a volume group descriptor at the start of the /dev/hdb1 partition.  (Note that if using LVM version 1 on PCs with DOS partitions, you must first set the partition type to “0x8e” using fdisk or some other similar program.)

Create Volume Groups (VGs)

Use the “vgcreate” program to group selected PVs into VGs, and to optionally set the extent size (the default is 4MB).  The following command creates a volume group named “VG1” from two disk partitions from different disks:

    vgcreate VG1 /dev/hda1 /dev/hdb1

Modern systems may use “devfs” or some similar system, which creates symlinks in /dev for detected disks.  With such systems names like “/dev/hda1” are actually the symlinks to the real names.  You can use either the symlink or the real name in the LVM commands, however the older version of LVM demanded you use the real device names, such as /dev/ide/host0/bus0/target0/lun0/part1 and /dev/ide/host0/bus0/target1/lun0/part1.

You can also specify the extent size with this command using the “-s size” option, if the 4Mb default not what you want.  The size is a value followed by one of k (for kilobytes), m (megabytes), g (gigabytes), or t (tetrabytes).  In addition you can put some limits on the number of physical or logical volumes the volume can have.  You may want to change the extent size for performance, administrative convenience, or to support very large logical volumes.  (Note there may be kernel limits and/or application limits on the size of LVs and files on your system.  For example Linux 2.4 kernel has a max size of 2TB.)

The “vgcreate” command adds some information to the headers of the included PVs.  However the kernel modules needed to use the VGs as disks aren’t loaded yet, and thus the kernel doesn’t “see” the VGs you created.  To make the VGs visible you must activate them.  Only active volume groups are subject to changes and allow access to their logical volumes.

To activate a single volume group VG1, use the command:

    vgchange -a y /dev/VG1

(“-a” is the same as “--available“.)  To active all volume groups on the system use:

    vgchange -a y

Create Logical Volumes (LVs)

Creating a logical volume in some VG is the most complex part of LVM setup, due to the many options available.  The basic command syntax is:

    lvcreate options size VG_name

Where size is either “-l num_extents” or “-L num_bytes“, where num_bytes is a number followed by one of k, m, g, or t.  If this second form is used you may not get an LV of that exact size, as LVs are always a whole number of extents.  You can also use “--extents” for “-l” or “--size” for “-L“.

One of the most common options is “-n name” (you can use “--name” for “-n“) to specify a name for the logical volume.  If you don’t use this option than the LVs are named automatically “lvol1“, “lvol2“, “lvol3“, etc.

Other options include “-C y” (or “--contiguous y“) to create an LV with contiguous allocation, “-i num_stripes -I stripe_size” to create an LV with striped mapping (stripe_size is a number between 2 and 9, as described above), and “-r num_sectors” (or “--readahead num_sectors“) to set the amount of read ahead to a value between 2 and 120.

Another form of this command is used to create snapshot volumes.  This will be discussed below.

Some examples of creating logical volumes LV1 and LV2 from the volume group VG1:

# To create a 20GB linear LV named "LV1" for some VG named
# "VG1" and its block device special file "/dev/VG1/LV1":
lvcreate -L 20g -n LV1 VG1

# To create a LV of 100 extents with 2 stripes and stripe size 4 KB:
lvcreate -i 2 -I 4 -l 100 -n LV2 VG1

If you want to create an LV that uses the entire VG, use the “vgdisplay” command to find the “Total PE” size, then use that when running lvcreate.

Once the LVs have been created you can format them with filesystems (or as swap space) using standard tools such as “mkfs“.  If the new filesystem can be successfully mounted, a final step is to edit the /etc/fstab file and possibly the rc.sysinit file, so that the volumes are mounted automatically at boot time.  It may also be necessary to setup an initial ramdisk for booting (if the “root” filesystem is built on a logical volume).

Create and Use a Snapshot

To create a snapshot of some existing LV, a form of the lvcreate command is used:

root# lvcreate size option -s -n name existing_LV

where size is as discussed previously, “-s” (or “--snapshot“) indicates a snapshot LV, “-n name” (or “--name name“) says to call the snapshot LV name.  The only option allowed is “-c chunk_size” (or “--chunksize chunk_size“), where chunk_size is specified as a power of 2 in this formula: (2^chunk_size) × 1024 bytes, where 2 ≤ chunk_size ≤ 10.)

Suppose you have a volume group VG1 with a logical volume LV1 you wish to backup using a snapshot.  you can estimate the time the backup will take, and the amount of disk writes that will take place during that time (plus a generous fudge factor), say 300MB.  Then you would run the command:

root# lvcreate -l 300m -s -n backup LV1

to create a snapshot logical volume named /dev/VG1/backup which has read-only access to the contents of the original logical volume named /dev/VG1/LV1 at point in time the snapshot was created.  Assuming the original logical volume contains a file system you now mount the snapshot logical volume on some (empty) directory, then backup the mounted snapshot while the original filesystem continues to get updated.  When finished, unmount the snapshot and delete it (or it will continue to grow as LV1 changes, and eventually run out of space).

Note: If the snapshot is of an XFS filesystem, the xfs_freeze command should be used to quiesce the filesystem before creating the snapshot (if the filesystem is mounted):

/root# xfs_freeze -f /mnt/point;
/root# lvcreate -L 300M -s -n backup /dev/VG1/LV1
/root# xfs_freeze -u /mnt/point
Warning	Full snapshot are automatically disabled

Now create a mount-point (an empty directory) and mount the volume:

/root# mkdir /mnt/dbbackup
/root# mount /dev/VG1/backup /mnt/dbbackup
mount: block device /dev/ops/dbbackup is write-protected, mounting read-only

If you are using XFS as the filesystem you will need to add the “nouuid” option to the mount command as follows:

/root# mount /dev/VG1/backup /mnt/dbbackup -o nouuid,ro

Do the backup, say by using tar to some “DDS4″ or “DAT” tape backup device:

/root# tar -cf /dev/rmt0 /mnt/dbbackup
tar: Removing leading `/' from member names

When the backup has finished you unmount the volume and remove it from the system:

root# umount /mnt/dbbackup
root# lvremove /dev/VG1/backup
lvremove -- do you really want to remove "/dev/VG1/backup"? [y/n]: y
lvremove -- doing automatic backup of volume group "VG1"
lvremove -- logical volume "/dev/VG1/backup" successfully removed

Examining LVM Information

To see information about some VG use:

vgdisplay some_volume_group
vgs some_volume_group

To see information about some PV use the command:

pvdisplay some_disk_or_partition # e.g., /dev/hda1
pvs some_disk_or_partition

To see information about some LV use:

lvdisplay some-logical-volume
lvs some-logical-volume

The man pages for these commands provides further details.

Grow VGs, LVs, and Filesystems

To grow a filesystem, you must install a new hard disk (unless you have free space available), format it as a PV,add that PV to your VG, then add the space to your LV, and finally use the filesystem tools to grow it.  (Not all filesystem allow or come with tools to grow and shrink them!)

VGs are resizable (spelled in Linux as “resizeable”) by adding or removing PVs from them.  However by default they are created as fixed in size.  To mark a VG as resizable use the command:

root# vgchange -x y  #or --resizeable y

Once this is done add a PV (say “hdb2″) to some VG (say “VG1″) with the command:

root# vgextend VG1 /dev/hdb2

Next, extend an LV with the “lvextend” command.  This command works almost the same as the “lvcreate” command, but with a few different options.  When specifying how much to increase the size of the LV, you can either specify how much to grow the LV with “+size” or you can specify the new (absolute) size (by omitting the plus sign).  So to extend the LV “LV1″ on VG “VG1″ by 2GB, use:

root# lvextend -L +2G /dev/VG1/LV1

You could also use:

root# lvresize -L +2G /dev/VG1/LV1

It would be a good idea to use the same mapping as the original LV, or you will have strange performance issues!  Also note this command can be used to extend a snapshot volume if necessary.

After you have extended the logical volume the last step is to increase the file system size.  How you do this depends on the file system you are using.  Most filesystem types come with their own utilities to grow/shrink filesystems, if they allow that.  These utilities usually grow to fill the entire partition or LV, so there is no need to specify the filesystem size.

Some common filesystem utilities are (assume we are expanding the /home filesystem in LV1 on VG1):

• EXT2 and EXT3:
  EXT2/3 filesystems must be unmounted before they can be resized.  The commands to use are: 

  root# umount /home # /home is the mount point for /dev/VG1/LV1
  root# fsck -f /home # required!
  root# resize2fs /dev/VG1/LV1 # grow FS to fill LV1.
  root# mount /home

• Reiserfs:
  The Reiserfs file system can be safely resized while mounted.  If unmounted resizing is preferred, first umount and afterward mount the filesystem.  For online resizing just use: 

  root# resize_reiserfs -f /dev/VG1/LV1

• XFS:
  The XFS file systems must be mounted to be resized, and the mount-point is specified rather than the device name: 

  root# xfs_growfs /home

• JFS:
  Like XFS, the JFS file system must be mounted to be resized and the mount-point is specified rather than the device name.  JFS doesn’t have a special utility for resizing, but the mount command has an option that can be used: 

  root# mount -o remount,resize /home

  In some cases the exact number of blocks must be specified.  (A kernel bug in some older Linux versions prevents the LV size from being determined automatically.)  For example to resize a JFS file system that has a 4KB block size (the default) to 4GB, you must use 1M 4KB-blocks.  Now “1M” is 2 raised to the power of 20 (=1048576), so use:

  root# mount -o remount,resize=1048576 /home

Shrink VGs, LVs, and Filesystems

To shrink a filesystem, you perform the same steps for growing one but in reverse order.  You first shrink the filesystem, then remove the space from the LV (and put it back into the VG).  Other LVs in the same VG can now use that space.  To use it in another VG, you must remove the corresponding PV from the one VG and add it to the other VG.

To shrink a LV you must first shrink the filesystem in that LV.  This can be done with the resize2fs for EXT2/3, or resize_reiserfs for ReiserFS (doing this off-line is safer but not required).  There are similar tools for other filesystem types.  Here’s an example of shrinking /home by 1 GB:

# df
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             145M   16M  122M  12% /boot
/dev/mapper/vg01-lv01  49G  3.7G   42G   9% /home
...
# umount /home
# fsck -f /home # required!
fsck 1.38 (30-Jun-2005)
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/home: 32503/6406144 files (0.3% non-contiguous), 1160448/12845056 blocks
# resize2fs -p /dev/vg01/lv01 48G
resize2fs 1.38 (30-Jun-2005)
Resizing the filesystem on /dev/vg01/lv01 to 12799788 (4k) blocks.
Begin pass 3 (max = 9)
Scanning inode table          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/vg01/lv01 is now 12799788 blocks long.

Currently XFS and JFS filesystem types do not support shrinking.  If a newer version of these filesystems will support this, mount may have been updated to support these filesystem types.  (And if not a new tool may be released.)  For such filesystems you can resize them the hard way: Backup the data using some archive tool (e.g., cpio, tar, star, or you can copy the data to some other disk).  Then delete the filesystem in the LV, then shrink the LV, then recreate the new (smaller) filesystem, and finally restore the data.

Once the filesystem has been shrunk it is time to shrink the logical volume.  You can use either the lvreduce command or the lvresize command.  Continuing from the example above:

# lvresize -L -1G /dev/vg01/lv01
  Rounding up size to full physical extent 96.00 MB
  WARNING: Reducing active logical volume to 48 GB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce lv01? [y/n]: y
  Reducing logical volume lv01 to 48 GB
  Logical volume lv01 successfully resized
# mount /home

To shrink a VG (say “VG1“), a PV (say “hdc“) can be removed from it if none of that PV’s extents (the PEs) are in use by any LV.  Run the command:

root# vgreduce VG1 /dev/hdc

You might want to do this to upgrade or replace a worn-out disk.  If the PV is in use by some LV, you must first migrate the data to another available PV within the same VG.  To move all the data from a PV (say “hdb2“) to any unused, large enough PV within that VG, use the command:

root# pvmove /dev/hdb2

Delete LVs and VGs

A logical volume (say “LV3” on the volume group “VG2“) must be unmounted before it can be removed.  The steps for this are simple:

root# umount /dev/VG2/LV3
root# lvremove /dev/VG2/LV3

Before a volume group (say “VG2“) is removed you must first deactivate it.  This is done with the command:

root# vgchange -a n VG2

Now the VG can be removed.  This of course will destroy all LVs within it.  The various PVs that made up that VG can then be re-assigned to some other VGs.  Remove (a non-active) volume group with:

root# vgremove VG2

Summary and Examples

In the following examples assume that LVM2 is installed and up to date, and the boot scripts have been modified already if needed.  The first example includes some commentary and some command output; the second is much shorter but uses the long option names just for fun.

Home directory Example

In this example we will create a logical volume to hold the “/home” partition for a multi-media development system.  The system will use a standard EXT3 filesystem of 60 GB, built using 3 25GB SCSI disks (and no hardware RAID).  Since multi-media uses large files it makes sense to use stripe mapping and read-ahead.  We will call the volume group “vg1” and the logical volume “home“:

1. Initialize the disks as PVs:
   /root# pvcreate /dev/sda /dev/sdb /dev/sdc
2. Create a Volume Group, then check it’s size:

   /root# vgcreate vg1 /dev/sda /dev/sdb /dev/sdc
   /root# vgdisplay
   vgdisplay
   --- Volume Group ---
   VG Name	          vg1
   VG Access             read/write
   VG Status             available/resizable
   VG #                  1
   MAX LV                256
   Cur LV                0
   Open LV               0
   MAX LV Size           255.99 GB
   Max PV                256
   Cur PV                3
   Act PV                3
   VG Size               73.45 GB
   PE Size               4 MB
   Total PE              18803
   Alloc PE / Size       0 / 0
   Free  PE / Size       18803/ 73.45 GB
   VG UUID               nP2PY5-5TOS-hLx0-FDu0-2a6N-f37x-0BME0Y

3. Create a 60 GB logical volume, with stripe set of 3 PVs and stripe size of 4 (which shows 2^4 KB = 16KB):

   /root# lvcreate -i 3 -I 4 -L 60G -n home vg1
   lvcreate -- rounding 62614560 KB to stripe boundary size 62614560 KB / 18803 PE
   lvcreate -- doing automatic backup of "vg1"
   lvcreate -- logical volume "/dev/vg1/home" successfully created

4. Create an EXT3 filesystem in the new LV:

   /root# mkfs -t ext3 /dev/vg1/home

5. Test the new FS:

   /root# mount /dev/vg1/home /mnt
   /root# df | grep /mnt
   /root# umount /dev/vg1/home

6. Update /etc/fstab with the revised entry for /home.
7. Finally, don’t forget to update the system journal.

Oracle Database Example

In this example we will create 2 LVs for an Oracle database.  Oracle manages its own striping and read-head/caching, so we won’t use these LVM features.  However using hardware RAID is useful, so we will use two RAID 10 disks, hdb and hdc.  The tables will use one logical volume called “tables” on one disk and the indexes and control files will be on a second LV called “indexes” on the other disk.  Both LVs will exist in the VG called “db“.  Both filesystems will be XFS, for good performance for large database files:

/root# pvcreate /dev/hdb /dev/hdc
/root# vgcreate db /dev/hdb /dev/hdc
/root# lvcreate --size 200G --name tables db
/root# lvcreate --size 200G --name indexes db
/root# mkfs -t xfs /dev/db/tables
/root# mkfs -t xfs /dev/db/indexes
/root# vi /etc/fstab
/root# vi ~/system-journal

• OpenMP 2.5 is supported by the GNU Compiler Collection (GCC) since version 4.2. Add the commandline option -fopenmp to enable it.
• OpenMP 3.0 is supported by the GNU Compiler Collection (GCC) since version 4.4. Add the commandline option -fopenmp to enable it.
• OpenMP 2.5 is supported by Microsoft Visual C++ 2005 (cl). Add the commandline option /openmp to enable it.
• OpenMP 2.5 is supported by Intel C Compiler (icc) since version 10.1. Add the commandline option -openmp to enable it. Add the -openmp-stubs option instead to enable the library without actual parallel execution.
• OpenMP 3.0 is supported by Intel C Compiler (icc) since version 11.0. Add the commandline option -openmp to enable it. Add the -openmp-stubs option instead to enable the library without actual parallel execution.

Note: If your GCC complains that “-fopenmp” is valid for D but not for C++ when you try to use it, or does not recognize the option at all, your GCC version is too old. If your linker complains about missing GOMP functions, you forgot to specify “-fopenmp” in the linking.More information: http://openmp.org/wp/openmp-compilers/

  #include <cmath>
  int main()
  {
    const int size = 256;
    double sinTable[size];

    #pragma omp parallel for
    for(int n=0; n<size; ++n)
      sinTable[n] = std::sin(2 * M_PI * n / size);

    // the table is now initialized
  }

 int a, b=0;
 #pragma omp parallel for private(a) shared(b)
 for(a=0; a<50; ++a)
 {
   #pragma omp atomic
   b += a;
 }

This example explicitly specifies that a is private (each thread has their own copy of it) and that b is shared (each thread accesses the same variable).

The difference between private and firstprivate

 #include <string>
 #include <iostream>

 int main()
 {
     std::string a = "x", b = "y";
     int c = 3;

     #pragma omp parallel private(a,c) shared(b) num_threads(2)
     {
         a += "k";
         c += 7;
         std::cout << "A becomes (" << a << "), b is (" << b << ")\n";
     }
 }

This will output the string “k”, not “xk”. At the entrance of the block, a becomes a new instance of std::string, that is initialized with the default constructor; it is not initialized with the copy constructor.Internally, the program becomes like this:

 int main()
 {
     std::string a = "x", b = "y";
     int c = 3;

     OpenMP_thread_fork(2);
     {                  // Start new scope
         std::string a; // Note: It is a new local variable.
         int c;         // This too.
         a += "k";
         c += 7;
         std::cout << "A becomes (" << a << "), b is (" << b << ")\n";
     }                  // End of scope for the local variables
     OpenMP_join();
 }

In the case of primitive (POD) datatypes (int, float, char* etc.), the private variable is uninitialized, just like any declared but not initialized local variable. It does not contain the value of the variable from the surrounding context. Therefore, the increment of c is moot here; the value of the variable is still undefined. (Unfortunately GCC, as of version 4.2.1, does not yet warn about uninitialized values in situations like this.)If you actually need a copy of the original value, use the firstprivate clause instead.

 #include <string>
 #include <iostream>

 int main()
 {
     std::string a = "x", b = "y";
     int c = 3;

     #pragma omp parallel firstprivate(a,c) shared(b) num_threads(2)
     {
         a += "k";
         c += 7;
         std::cout << "A becomes (" << a << "), b is (" << b << ")\n";
     }
 }

Now the output becomes “A becomes (xk), b is (y)”.

The lastprivate clause

• In a loop construct (for directive), the last value is the value assigned by the thread that handles the last iteration of the loop. Values assigned during other iterations are ignored.
• In a sections construct (sections directive), the last value is the value assigned in the last section denoted by the section directive. Values assigned in other sections are ignored.

Example:

 #include <stdio.h>
 int main()
 {
    int done = 4, done2 = 5;

     #pragma omp parallel for lastprivate(done, done2) num_threads(2) schedule(static)
     for(int a=0; a<8; ++a)
     {
       if(a==2) done=done2=0;
       if(a==3) done=done2=1;
     }
     printf("%d,%d\n", done,done2);
 }

This program outputs “4196224,-348582208″, because internally, this program became like this:

 #include <stdio.h>
 int main()
 {
    int done = 4, done2 = 5;
    OpenMP_thread_fork(2);
    {
        int this_thread = omp_get_thread_num(), num_threads = 2;
        int my_start = (this_thread  ) * 8 / num_threads;
        int my_end   = (this_thread+1) * 8 / num_threads;

        int priv_done, priv_done2; // not initialized, because firstprivate was not used

        for(int a=my_start; a<my_end; ++a)
        {
            if(a==2) priv_done=priv_done2=0;
            if(a==3) priv_done=priv_done2=1;
        }
        if(my_end == 
        {
           // assign the values back, because this was the last iteration
           done  = priv_done;
           done2 = priv_done2;
        }
    }
    OpenMP_join();
 }

As one can observe, the values of priv_done and priv_done2 are not assigned even once during the course of the loop that iterates through 4…7. As such, the values that are assigned back are completely bogus.Therefore, lastprivate cannot be used to e.g. fetch the value of a flag assigned randomly during a loop. Use reduction for that, instead.Where this behavior can be utilized though, is in situations like this (from OpenMP manual):

 void loop()
 {
   int i;
   #pragma omp for lastprivate(i)
   for(i=0; i<get_loop_count(); ++i) // note: get_loop_count() must be a pure function.
       { ... }

   printf("%d\n", i); // this shows the number of loop iterations done.
 }

The default clause

 int a, b=0;
 // This code won't compile: It will require that
 // it will be explicitly specified whether a is shared or private.
 #pragma omp parallel default(none) shared(b)
 {
   b += a;
 }

The default clause can also be used to set that all variables are shared by default (default(shared)).

Note: Because different compilers have different ideas about which variables are implicitly private or shared, and for which it is an error to explicitly state the private/shared status, it is recommended to use the default(none) setting only during development, and drop it in production/distribution code.

The reduction clause

 int factorial(int number)
 {
   int fac = 1;
   #pragma omp parallel for reduction(*:fac)
   for(int n=2; n<=number; ++n)
     fac *= n;
   return fac;
 }

• At the beginning of the parallel block, a private copy is made of the variable and preinitialized to a certain value .
• At the end of the parallel block, the private copy is atomically merged into the shared variable using the defined operator.

(The private copy is actually just a new local variable by the same name and type; the original variable is not accessed to create the copy.)The syntax of the clause is:

  reduction(operator:list)

where list is the list of variables where the operator will be applied to, and operator is one of these:

To write the factorial function (shown above) without reduction, it probably would look like this:

 int factorial(int number)
 {
   int fac = 1;
   #pragma omp parallel for
   for(int n=2; n<=number; ++n)
   {
     #pragma omp atomic
     fac *= n;
   }
   return fac;
 }

However, this code would be less optimal than the one with reduction: it misses the opportunity to use a local (possible register) variable for the cumulation, and needlessly places load/synchronization demands on the shared memory variable. In fact, due to the bottleneck of that atomic variable (only one thread may access it simultaneously), it would completely nullify the gains of parallelism in that loop.The version with reduction is equivalent to this code (illustration only):

 int factorial(int number)
 {
   int fac = 1;
   #pragma omp parallel
   {
     int fac_private = 1; /* This value comes from the table shown above */
     #pragma omp for nowait
     for(int n=2; n<=number; ++n)
       fac_private *= n;
     #pragma omp atomic
     fac *= fac_private;
   }
   return fac;
 }

Note how it moves the atomic operation out from the loop.The restrictions in reduction and atomic are very similar: both can only be done on POD types; neither allows overloaded operators, and both have the same set of supported operators.As an example of how the reduction clause can be used to produce semantically different code when OpenMP is enabled and when it is disabled, this example prints the number of threads that executed the parallel block:

 int a = 0;
 #pragma omp parallel reduction (+:a)
 {
   a = 1; // Assigns a value to the private copy.
   // Each thread increments the value by 1.
 }
 printf("%d\n", a);

If you preinitialized “a” to 4, it would print a number >= 5 if OpenMP was enabled, and 1 if OpenMP was disabled.
Note: If you really need to detect whether OpenMP is enabled, use the _OPENMP #define instead. To get the number of threads, use omp_get_num_threads() instead.

Execution synchronization

 #pragma omp parallel
 {
   /* All threads execute this. */
   SomeCode();

   #pragma omp barrier

   /* All threads execute this, but not before
    * all threads have finished executing SomeCode().
    */
   SomeMoreCode();
 }

 #pragma omp parallel
 {
   #pragma omp for
   for(int n=0; n<10; ++n) Work();

   // This line is not reached before the for-loop is completely finished
   SomeMoreCode();
 }

 // This line is reached only after all threads from
 // the previous parallel block are finished.
 CodeContinues();

 #pragma omp parallel
 {
   #pragma omp for nowait
   for(int n=0; n<10; ++n) Work();

   // This line may be reached while some threads are still executing the for-loop.
   SomeMoreCode();
 }

 // This line is reached only after all threads from
 // the previous parallel block are finished.
 CodeContinues();

 #pragma omp parallel
 {
   Work1();
   #pragma omp single
   {
     Work2();
   }
   Work3();
 }

 #pragma omp parallel
 {
   Work1();

   // This...
   #pragma omp master
   {
     Work2();
   }

   // ...is practically identical to this:
   if(omp_get_thread_num() == 0)
   {
     Work2();
   }

   Work3();
 }

 #pragma omp parallel for
 for(int y=0; y<25; ++y)
 {
   #pragma omp parallel for
   for(int x=0; x<80; ++x)
   {
     tick(x,y);
   }
 }

 #pragma omp parallel for
 for(int y=0; y<25; ++y)
 {
   #pragma omp for // ERROR, nesting like this is not allowed.
   for(int x=0; x<80; ++x)
   {
     tick(x,y);
   }
 }

 #pragma omp parallel for collapse(2)
 for(int y=0; y<25; ++y)
   for(int x=0; x<80; ++x)
   {
     tick(x,y);
   }

 #pragma omp parallel for
 for(int pos=0; pos<(25*80); ++pos)
 {
   int x = pos%80;
   int y = pos/80;
   tick(x,y);
 }

 omp_set_nested(1);
 #pragma omp parallel for
 for(int y=0; y<25; ++y)
 {
   #pragma omp parallel for
   for(int x=0; x<80; ++x)
   {
     tick(x,y);
   }
 }

 for(int y=0; y<25; ++y)
 {
   #pragma omp parallel for
   for(int x=0; x<80; ++x)
   {
     tick(x,y);
   }
 }

  #include <stdio.h>
  #include <sys/wait.h>
  #include <unistd.h>

  void a()
  {
    #pragma omp parallel num_threads(2)
    {
      puts("para_a"); // output twice
    }
    puts("a ended"); // output once
  }
  void b()
  {
    #pragma omp parallel num_threads(2)
    {
      puts("para_b");
    }
    puts("b ended");
  }

  int main() {
   a();   // Invokes OpenMP features (parent process)
   int p = fork();
   if(!p)
   {
     b(); // ERROR: Uses OpenMP again, but in child process
     _exit(0);
   }
   wait(NULL);
   return 0;
  }

 /* Returns any position from the haystack where the needle can
  * be found, or NULL if no such position exists. It is not guaranteed
  * to find the first matching position; it only guarantees to find
  * _a_ matching position if one exists.
  */
 const char* FindAnyNeedle(const char* haystack, size_t size, char needle)
 {
   for(size_t p = 0; p < size; ++p)
     if(haystack[p] == needle)
     {
       /* This breaks out of the loop. */
       return haystack+p;
     }
   return NULL;
 }

 const char* FindAnyNeedle(const char* haystack, size_t size, char needle)
 {
   const char* result = NULL;
   bool done = false;
   #pragma omp parallel for
   for(size_t p = 0; p < size; ++p)
   {
     #pragma omp flush(done)
     if(!done)
     {
       /* Do work only if no thread has found the needle yet. */
       if(haystack[p] == needle)
       {
         /* Inform the other threads that we found the needle. */
         done = true;
         #pragma omp flush(done)
         result = haystack+p;
       }
     }
   }
   return result;
 }

 const char* FindAnyNeedle(const char* haystack, size_t size, char needle)
 {
   const char* result = NULL;
   #pragma omp parallel for
   for(size_t p = 0; p < size; ++p)
     if(haystack[p] == needle)
       result = haystack+p;
   return result;
 }

 const char* FindAnyNeedle(const char* haystack, size_t size, char needle)
 {
   const char* result = NULL;
   bool done = false;
   #pragma omp parallel
   {
     int this_thread = omp_get_thread_num(), num_threads = omp_get_num_threads();
     size_t beginpos = (this_thread+0) * size / num_threads;
     size_t endpos   = (this_thread+1) * size / num_threads;
     // watch out for overflow in that multiplication.

     for(size_t p = beginpos; p < endpos; ++p)
     {
       /* End loop if another thread already found the needle. */
       #pragma omp flush(done)
       if(done) break;

       if(haystack[p] == needle)
       {
         /* Inform the other threads that we found the needle. */
         done = true;
         #pragma omp flush(done)
         result = haystack+p;
         break;
       }
     }
   }
   return result;
 }

 const char* FindAnyNeedle(const char* haystack, size_t size, char needle)
 {
   const char* result = NULL;
   bool done = false;
   size_t beginpos = 0, n_per_loop = 4096 * omp_get_max_threads();
   while(beginpos < size && !done)
   {
     size_t endpos = std::min(size, beginpos + n_per_loop);
     #pragma omp parallel for reduction(|:done)
     for(size_t p = beginpos; p < endpos; ++p)
       if(haystack[p] == needle)
       {
         /* Found it. */
         done = true;
         result = haystack+p;
       }
     beginpos = endpos;
   }
   return result;
 }

1. ^ Trumpler and Weaver (1962), pp. 32–33.