Three weeks ago, my colleague Petri presented our (that is, me, Petri, and Teemu who did most of the work) paper titled “Quality Of WordPress Plug-Ins: An Overview of Security and User Ratings” at Workshop on Security and Privacy in Social Networks. The main purpose of the paper was to analyze a set of WordPress plugins to find out if plugin ratings predict the number of potential security vulnerabilities in them. In order to do that, we downloaded the source code and ratings and download counts of a set of random plugins (322 in total). We then ran the RIPS PHP security vulnerability scanner to find potential security holes in the code. The 322 plugins had:

• 3,792,711 downloads
• 2,783 user ratings
• 179,393 lines of PHP code

Of the 322 plugins, 127 had a potential security vulnerability according to RIPS. In total, 860 vulnerabilities were identified.

When comparing this with the ratings of the plugins, we found only a small, negative correlation between the rating of the plugin and the number of potential vulnerabilities. To put it another way, plugin ratings are not a good measure of the security of the plugin.

There are, of course, several problems with our analysis and conclusion. Biggest of those is that the sample size was really small compared to the total number of plugins (over 21,000 at the moment). To improve the results, we are currently analyzing a bigger sample (100% of them, to be exact).

If you are interested in more details, the full paper can be downloaded here.