Chapter 19.  Synthetic Benchmarks: Evaluating Code Blocks and Functions

19. Synthetic Benchmarks: Evaluating Code Blocks and Functions

Chapter 18, "Profiling," DESCRIBES BENCHMARKING entire applications. This is useful for doing comparative analysis between Web pages to identify slow pages and for measuring the effects of application tuning. Similar techniques are useful for evaluating the differences such as the following between different code blocks:

• Is while() faster or slower than foreach() for loops?

• Is substr() faster than strstr() for matching characters at the beginning of a string?

You could go online and search the PHP general archives to look for the answers, or you could pick up a book (like this one) for some sage advice on the subject, but neither of these methods is really sufficient. One of PHP's strengths is the rapid development of the language itself. Performance differences that exist now may be absent from future releases of the language. Nor does this happen only on major releasesthe open-source development model behind PHP means that many problems are addressed when they itch someone enough to need scratching. These are just two examples of code patterns that reversed themselves:

• In every version of PHP until version 4.3, interpolated variables in strings were much slower than concatenating strings. (Refer to the section "Interpolation Versus Concatenation," later in this chapter.)

• Using the built-in parse_url() function is much slower than parsing the URL in the userspace by using preg_match. This was also fixed in version 4.3. (Refer to the section "Adding Custom Timer Information," later in this chapter.)

When you're tuning critical code, it is always preferable to make the comparison and the appropriate code-usage choice yourself, as opposed to relying on someone else's purported benchmarks.

To answer the aforementioned questions and others, you need to write synthetic benchmarks as test cases. Synthetic benchmarks provide a means for testing small portions of code or individual functions to evaluate (and, by comparison, minimize) their resource usage. By incorporating benchmarks into unit tests, you can also track performance changes in libraries over time.

Synthetic benchmarks differ from application benchmarks in that they do not attempt to simulate a realistic use of the application but instead focus simply on measuring the performance of a particular piece of code. Synthetic benchmarks have a rich history in computer science. In the 1950s, programmers used benchmarks with the goal of optimizing physical systems' implementations. One of the original and most famous synthetic benchmarks is the Whetstone benchmark, designed to benchmark floating-point operations. Other common examples include calculating Fibonacci Sequences, using the Towers of Hanoi to test the speed of recursive function calls in a language, and using matrix multiplication to test linear algebra algorithms.

The results of synthetic benchmarks often have little bearing on the overall performance of an application. The real issue is that nothing is intrinsically broken with the idea of benchmarking; rather, it is simply an issue of optimizing the wrong parts of an application. A critical companion to benchmarking is profiling, which allows you to pinpoint the sections of an application that can benefit most from optimization.

In creating a good synthetic benchmark, you need to address the following two issues:

• Does it test what you intend? This might sound obvious, but it is very important to make sure a benchmark is really designed to test what you are looking for. Remember: You are not testing the whole application, but just a small component. If you do not succeed in testing that component alone, you have reduced the relevance of the benchmark.

• Does it use the function the way you will? Algorithms often vary dramatically, depending on the structure of their input. If you know something about the data that you will be passing to the function, it is beneficial to represent that in the test data set. Using a sample of live data is optimal.

Intentionally missing from this list is the question "Is it relevant?" Benchmarking can be a useful exercise in and of itself to help familiarize you with the nuances of PHP and the Zend Engine. Although it might not be useful to optimize array iteration in a seldom-used script, having a general knowledge of the performance idioms of PHP can help you develop a coding style that needs less optimization down the road.