Skip to content →

Measure Jetpack – Progress Update

Since people are asking questions and even writing about it, it is probably way past time to give a decent progress update on the Measure Jetpack project.

I haven’t talked about it much since I announced my intentions back in October, mainly because there was not much to report. Things were moving slugishly for several reasons. In November I started working with Codeable (affiliate link) and my free time quickly evaporated. The students that I have drafted for this project had an exam period and without me, pushing them hard, did not make much progress, too.

But now we are back and we intend to move faster. There is one wonderful and powerful thing, that will make both a student and a freelancer pick up the pace – a deadline. And we have one coming up – if we want to make it to one of the conferences that we originally planned to attend, we have to deliver an article by mid February. So you can look forward to having some results by then at the latest.

Dataset

Now, about our current status. Early in the planing stages we realized that if the run our tests on „Hello World“ WordPress installation, they will not show much in terms of performance for modules like Related posts. So we decided to try and build a decent set of data for our test site. And we did. We now have 500 posts, complete with thumbnails, categories, tags, etc. For content, we started by taking country and capital descriptions from Wikipedia and then moved on to other major cities. We also added some pages (20), some comments (50), created 3 categories and 48 different tags. Every post has a thumbnail, so there are 500 media items in there, too.

Since we intend our results to be replicable, we are going to release this dataset on GitHub for anyone to use, after we are done tinkering with it and run our tests. It should be useful not only to try and replicate our results, but also for all kinds of other WordPress testing/benchmarking purposes.

Metrics

Here is a list of metrics that we currently plan to measure during our testing:

Server Side

  • Peak Memory Usage
  • PHP Execution Time
  • SQL Read Query Count
  • SQL Write Query Count
  • SQL Execution Time
  • Local Script/Style Request Count
  • Remote Sricpt/Style Request Count
  • Remote HTTP GET Request Count
  • Remote HTTP GET Request Count

Browser Side

  • Number of HTTP requests
  • Total Load Time
  • Total Size of Loaded Website

If you have other suggestions here, please let me know, so I can include them before we start running our tests.

Tools

Our system for measuring performance is still a work in progress, but we have it in broad strokes. At first, I was planning to use John Blackbourn’s excellent Query Monitor plugin and Pingdom’s Website Speed Test tool to measure performance, but that would have meant a lot of manual work and would not allow for fully testing both anonymous and logged-in state of the website.

So with a goal to automate the process as much as possible, I set out to build my own measuring tools in a form of a small and simple must-use plugin. It uses a lot of the same methods for measuring stuff as Query Monitor does, but it also allows to turn the measuring on/off via GET arguments and stores the results. I tried to keep the footprint of the plugin as light as possible, it currently is only just over 100 lines long and does most of its work on the shutdown hook, when WordPress is pretty much done.

This plugin uses SAVEQUERIES constant to tell $wpdb to log all queries it makes, PHP’s memory_get_peak_usage(false) to get memory usage, and WordPress’ built-in stop_timer() function to measure PHP execution time. $wp_styles and $wp_scripts are used to determine and count what scripts and styles were loaded in a particular pageview. The only measurement that happens before shutdown hook is counting remote HTTP requests via HTTP API – it hooks into pre_http_request for that.

For the browser side, I am going to use PhantomJS/CasperJS. Currently, this side is not yet fully built out, but it looks promising. It will not only allow me to observe how many actuall resources are loaded, an what is the total page size and render time, but also to fully automate the whole test scenario (more about that later).

Hardware

The test will be run on a 10$ Digital Ocean (again, affiliate link) droplet, with nothing else on it but the testing site. We’ll use the latest avialable WordPress version (probably 4.4.1, but you never know) and LEMP stack. I’ll provide actual version numbers for each component later on, but it will be the latest versions available via Ubuntu repos. I am not going to do any performance tinkering and will provide the nginx configuration file.

Method

The process of testing will be as follows:

  1. Restore WordPress to it’s original state from a backup copy;
  2. A plugin/Jetpack module is enabled.
  3. A testing scenario is run:
    1. Visit to the front page
    2. Visit to an archive page
    3. Visit to a single post
    4. User logs in.
    5. Visit to the front page
    6. Visit to an archive page
    7. Visit to a single post
    8. Visit WP Admin Dashboard
    9. Visit New Post page
    10. Publish a new post
  4. 3rd step is repeated 7 times;
  5. The best and worst result are thown out, averages are calculated from the rest;

This is where we are at the moment. Our current plan is to have everything ready by the end of the week and finish running the tests by the end of January. And then we’ll have a couple of weeks to comb through the raw data and publish some results.

Finally, I’d like to thank people who kept our coffee mugs full: Status301, Postmatic and Matt Mullenweg. If you want to join them, please use a form below.

Don’t forget to share this via , , Google+, LinkedIn and Reddit.

 

Published in Research