CSE 522S: Studio 3

Userspace Benchmarking

"I wish it need not have happened in my time," said Frodo.
"So do I," said Gandalf, "and so do all who live to see such times. But that is not for them to decide. All we have to decide is what to do with the time that is given, us."

—The Fellowship of the Ring , Book 1, Chapter 2

Benchmarking programs is a common task that drives decisions about where and how to invest the effort of software developers. This task can be very simple to very complex, depending on desired accuracy and the kind of code being profiled. In an OS, code that is run very infrequently or at a low priority doesn't need to be validated to as high a standard as code that runs very frequently (such as the system timer interrupt handler) or at high priority (such as real-time priority). Benchmarking in userspace serves as an introduction to the concepts behind benchmarking in the kernel, and it allows us to measure the impact of the kernel on userspace performance.

In this studio, you will:

1. Benchmark programs using coarse command line tools
2. Benchmark programs using Linux's clock functions

Please complete the required exercises below, as well as any optional enrichment exercises that you wish to complete.

As you work through these exercises, please record your answers, and when finished email your results to dferry@email.wustl.edu with the phrase Userspace Benchmarking in the subject line.

Make sure that the name of each person who worked on these exercises is listed in the first answer, and make sure you number each of your responses so it is easy to match your responses with each exercise.

Required Exercises

1. As the answer to the first exercise, list the names of the people who worked together on this studio.

2. First, we need some programs to benchmark. I've put together a code package that includes four programs for us to use. Download it here. Unzip the package, build the programs with the provided makefile, and run each one a few times. As the answer to this exercise, list what each program does.

3. We can do some very coarse grained benchmarking straight from the command line, using a bash command named time. This is actually a special command built into the bash shell, so its documentation can be found under man bash. As the answer to this exercise, use this command to make a few timings of the test programs and copy the results here.

4. The time command outputs three different timings. List them here and explain the difference.

5. Compare the results of the following two commands. What do you observe about the relationship between the user and real timings? Explain your observations.
   • time ./dense_mm 400
   • time ./parallel_dense_mm 400

6. Execute the following command. What do you observe about the relationship between the user and sys timings? Look at the code in sing.c and explain your observations.
   • time ./sing 1000

7. Now we're going to switch to using the C API for Linux's clocks. First, we'll look at exactly what clocks are available and get some info about each with the function clock_getres. You can access the documentation for this function in the manual pages: man clock_getres. Warning: Internet versions of man pages are often not up to date. Use the version on your machine.

   Look through the clocks available at the clock_getres man page. For this exercise, give two clock names. For the first, give a clock that would be well suited for userspace benchmarking, and explain why. For the second, give a clock that would be poorly suited for userspace benchmarking, and explain why.

8. Next, use clock_getres to write a short program called getres.cthat gives the resolutions for several different clock types. This function requires using a structure called a timespec which is also documented at the same man page, and is the basic data structure used to report timing information from the kernel to userspace.

   As the answer to this exercise, copy and paste your program output. Include at least one _COARSE clock type.

9. Explain what is meant by the resolution values returned from your program.

10. Given what you know about Linux timers, the HZ variable, and jiffies from chapter 11 in our book, speculate about the difference between CLOCK_MONOTONIC and CLOCK_MONOTONIC_COARSE.

11. Write a second short program that uses the function clock_gettime to figure out how long a call to clock_gettime takes. As the answer to this exercise, report this value.

12. Make a copy of parallel_dense_mm.c. First modify it so that you time the critical computational loop with the clock CLOCK_MONOTONIC_RAW. Then modify it again so that the program takes a second parameter and executes the timed segment multiple times. Your program should output the minimum, average, and maximum timings over all timed iterations.

    Leave the answer to this exercise blank

13. Run your program for 100 iterations with matrix size 100. What are the reported values? Speculate about the total distribution of values- based on the minimum, avgerage, and maximum timings, what do you think a common running time actually is? Give an explanation for the distribution of values.

Things to turn in

In addition to the answers above, please submit:

• getres.c
• Your copy of parallel_dense_mm.c