Vulnerability detection tools are frequently

considered the silver-bullet for detecting vulnerabilities in web

services. However, research shows that the effectiveness of

most of those tools is very low and that using the wrong tool

may lead to the deployment of services with undetected

vulnerabilities. In this paper we propose a benchmarking

approach to assess and compare the effectiveness of

vulnerability detection tools in web services environments. This

approach was used to define a concrete benchmark for SQL

Injection vulnerability detection tools. This benchmark is

demonstrated by a real example of benchmarking several

widely used tools, including four penetration-testers, three

static code analyzers, and one anomaly detector. Results show

that the benchmark accurately portrays the effectiveness of

vulnerability detection tools and suggest that the proposed

approach can be applied in the field.