dtrace'ing paging to disk

How to know which processes are paging to/from disk (as opposed to other VMM management) in OS X, and how much exactly:

sudo dtrace -n maj_fault'{@[execname] = count()}'

reference (with examples and other options): http://wikis.sun.com/display/DTrace/vminfo+Provider

I had been meaning to look for ways to do this, and tried some of the tools included in OS X (Activity Monitor, top, Instruments, vm_stat, vmmap, top, ...). But nothing really helped, and/or seemed to miss the exact level of information I was targeting (only resulting in real I/O; relationship between I/O and process; realtime... ). Finally I had the inspiration to google for "dtrace pagefaults". Bingo.
(dtrace in this example isn't realtime, but is the best approximation so far, and I'm guessing some tuning should fix it. Heck, it's a one-liner!)

Learning dtrace is still something I'd love to do, and once again it is tempting me to let it jump to the front of queue...

(Mhm, will Oracle's Java for OS X support dtrace too?)

Oh, and of course Firefox was by far the greatest pagefaulter, even with the great improvements in the v9 betas. (I actually saw it diminish its VM usage by some hundreds of MB when it was freshly launched!... though after some days it has gone back to its habit of hovering over 20% CPU and 2 GB VM even while idle)
But it's interesting that the second offender is Skype, even if more than one order of magnitude less than Firefox; but also one order of magnitude greater than the 3rd and rest of offenders. Interesting because it looks very good mannered in Activity Monitor, and it was unused during all of the measuring time. Maybe its P2P routing thing...? Would be good to keep an eye on it.


  1. Some interesting one-liners in http://www.nbl.fi/~nbl97/solaris/dtrace/index.html#line .

    In particular, it will be interesting how this works as a substitute for lsof | grep procname:
    dtrace -qn 'syscall::open*:entry{ printf("%s %s\n",execname,copyinstr(arg0)); }'

  2. vmmap -w -resident firefox also seems interesting. After all, DTrace only can report on what is happening at the moment, since by definition when it's disabled it does nothing... (although I guess a probe could just dump the value of some variable already existing in the computer; the MIB-related probes surely do just that). vmmap -resident in fact shows a good picture of which part of Firefox is virtual vs. ... well, resident.

    But looks like getting any deeper would mean debugging Firefox itself.

  3. An interesting test: Firefox active, teoretically idle, with about 1.5 GB in VM. Not used during a couple of hours during which I did other things in the computer (text editing, browsing with Safari).

    After these couple of hours, Safari has grown to about 1.5 GB too (Safari + WebContent). Safari has been active for as long as Firefox, more or less. Firefox has about 100 tabs open, Safari never had more than about 30, and the average would be about 15.
    Both have AdBlockers, FlashBlockers and additionally Firefox has NoScript. Which sometimes feels like it hurts as much as it helps.

    So that's on one hand. On the other hand, the idling Firefox has managed to keep about 1.4 of its 1.5 GB loaded in memory (Real Memory vs Virtual Memory as shown in Activity Monitor). Meanwhile, Safari, being the actively used browser, keeps in real memory less than 800 MB of its 1.5 virtual GB.

    And the dtrace paging script shows that Firefox paged almost as much as Safari during these 2 hours. So, even though Firefox was using very little CPU, it has been slowing the rest of the computer through memory pressure. And Safari complied.

    It's interesting because yesterday I had tried hiding Firefox and stopping it with kill -SIGSTOP. That helps memory-wise; after some time the Activity Monitor showed that Firefox had been paged out of Real Memory almost completely (less than 1 MB remained!). But the WindowServer process was having strange, persistent peaks of activity (almost 100% for whole seconds, so maxing out one core); I felt it was weirding out because of the stopped Firefox, so I reactivated it with -SIGCONT. And the WindowServer peaks went away, but immediately Firefox's Real Memory started growing up. At the moment I was trying to use Firefox, so that was expected.

    But given its paging behaviour now when it was supposed to sit quietly, it looks like Firefox not only is a godless memory hog; it is an actively, aggressively pushy godless memory hog.

    I really, really hope that Firefox's MemoryShrink efforts do work, because this is mind-boggling. 4 cores, discrete GPU, 4 GB RAM and I am feeling that I need a SSD?? WTF?

    Yesterday I had to use for some minutes the iBook G4 with OS X 10.4 Tiger. With Safari 4 and Firefox 9. I was amazed at how snappy it felt. 1 core, 1.2 GHz I think, 1.5 GB IIRC, no fan. And amazingly snappy.

    Not good.

    Chrome was even worse (much worse) than Firefox the last time I tried it, but that was maybe 3 months ago. Maybe it's time to try again.