Linux system profiling with perf

Most probably you have heard of dtrace. And not so much about the Linux alternatives of it. Well today I decided to give the Linux perf tool a shot while trying to find performance problem on a heavily loaded LAMP server.

Installing perf

Actually installing perf on Ubuntu 12.04 is really straightforward :

Collecting performance events

After if finished installing you can run a system wide profiling with :

And stop with Ctrl-C, to finish collecting the data.

will show the what processes are using the most CPU:

perf report

Obviously most CPU goes  to apache(mod_php) and and php5 standalone processes. And looking at the names of the functions, on top is the garbage collection. If you don’t have the debug symbols  installed then you won’t see this breakdown by function but instead just total process percentage which is not so useful.

To install debug symbols install the php5-dbg package, and update the perf buildid database.

Of course just knowing which function is taking up a lot of CPU is very useful but even more useful is to have the backtrace of who called it. Luckily perf does that. Just add -g to the record command:

And after with :

you can expand the function to see the callers with E and C shortcuts. Or use :

To see it as ASCII tree in the console like this:

perf report call tree

The profiling can be done only for a single running process :

Or for a single execution of a command:

Also there is simple statistics mode :

This is the default set of collected performance events, but you can sample more with the -e switch. For a list of all supported events run:

You can find more details and examples on the project wiki .