• Subscribe to the low volume list for updates.

Nessus, OpenVAS and Nexpose VS Metasploitable

The following article shows results from a test in which I have chosen to target three different vulnerability scanners in a "black box" test against a Metasploitable version 2 Virtualbox. In such a test the vulnerability scanner run against a target with no prior knowledge or credentialed access to the system.

In this high-level comparison of Nessus, Nexpose, and OpenVAS, I have not attempted a detailed metric-based analysis with the reason being it would be difficult to get a conclusive result due to the large differences in detection and the categorization of vulnerabilities by the different solutions.

Background Info

The testing deliberately focuses on network vulnerability scanning capabilities rather than looking at the web application vulnerability detection in detail. Here at Hacker Target, we believe a network vulnerability scanner must be capable of identifying poorly configured services, default services that have poor security, and software with known security vulnerabilities.

Notes on the Vulnerability Scanner Testing

  • Apart from NMAP, external tools that OpenVAS can use have not been installed. These external tools are mostly web application vulnerability detection tools, including wapiti, Arachni, Nikto and Dirb.
  • OpenVAS version 5 has been tested with the full scan profile. Ports were all TCP ports scanned with Nmap and top 100 UDP ports.
  • Nessus version 5 was launched using the External network scan profile. It was also tested with Internal Network Scan however, results were similar.
  • The Nexpose scanner was executed with the Full audit profile.
  • No tweaking of default scan profiles was undertaken.
  • No credentials were used during the scan. It was an external network service focused scan.

These results are only a quick overview. I have not followed up every discovered vulnerability to determine false positives and false negatives.

Edit 1st of September 2012 (clarification of scanner versions and plugins used)
Nessus : The home feed was used for the Nessus testing. According to the Tenable website The Nessus HomeFeed gives you the ability to scan your personal home network (up to 16 IP addresses) with the same high-speed, in-depth assessments and agentless scanning convenience that ProfessionalFeed subscribers enjoy.. Note when using the Nessus scanner with the home feed it cannot be used in a professional or commercial environment.
OpenVAS : The default OpenVAS 5 open source signatures and software was used. This is free to use under the GNU General Public License (GNU GPL).
Nexpose : The community version of Nexpose was tested. According to the Rapid7 website " Nexpose Community Edition is powered by the same scan engine as award-winning Nexpose Enterprise Edition and offers many of the same features." With this version you can scan up to 32 IP addresses.

And now for the results.....

Nessus 5
External Network Profile
Critical 3
High 6
Medium 22
Low 8
Info 137
OpenVAS 5
Full Audit Scan Profile
High 38
Medium 24
Low 36
Log 44
Nexpose
Full Audit Scan Profile
Critical 49
Severe 103
Moderate 18

These total numbers, without any context around the categorization of findings or the accuracy of the results, provides us little value, except to highlight the wide variation in results from the different scanners.

Analysing a specific sample of Security Issues

In order to look at some more meaningful results, I have examined a sample set of exploitable and mis-configured services on the Metasploitable system.

This is only a sample of exploitable services on the target host. There are many more vulnerabilities present on the system; both network services and web application security holes.

At the last minute I decided to include Nmap with its NSE scripts against the Metasploitable host. The results were interesting to say the least, while not a full blown vulnerability scanner the development of the NSE scripting ability in Nmap makes this powerful tool even more capable.

the numbers get interesting...

These are the numbers of vulnerabilities correctly discovered and rated by each vulnerability scanner from the sample set of exploitable services.

Nessus OpenVAS NexPose Nmap
7 7 7 6

7 out of 15 security holes identified

Security Issue Nessus OpenVAS Nexpose Nmap
FTP 21
Anonymous FTP Access
FTP 21
VsFTPd Smiley Face Backdoor
FTP 2121
ProFTPD Vulnerabilities
SSH 22
Weak Host Keys
PHP-CGI
Query String Parameter Injection
CIFS
Null Sessions
INGRESLOCK 1524
known backdoor drops to root shell
NFS 2049
/* exported and writable
MYSQL 3306
weak auth (root with no password)
RMI REGISTRY 1099
Insecure Default Config
DISTCCd 3632
distributed compiler
POSTGRESQL 5432
weak auth (postgresql)
VNC 5900
weak auth (password)
IRC 6667
Unreal IRCd Backdoor
Tomcat 8180
weak auth (tomcat/tomcat)

Notes about the sample set of tests

  • All the above vulnerabilities and mis-configurations, except for Anonymous FTP, can be exploited to gain shells on the system (in most cases with root privileges) using Metasploit or other methods.
  • There are a number of examples where the scanners do not detect weak or default credentials. While not specifically testing passwords, if MySQL is being checked for weak credentials why not other services?
  • Items such as the INGRESLOCK backdoor and the Unreal IRCd vulnerability are fairly obscure, however, this makes them good examples for testing overall capability.
  • The Metasploitable version 2 release page has good examples of exploiting many of the mis-configurations in this list. This highlights not only how a poorly configured service can lead to a root shell but also the fact that vulnerability scanners need to be able to detect these types of security related mis-configurations.

These scans were conducted in a black box manner, when running internal scans it is recommended to perform credential supplied scanning. This means providing the vulnerability scanning tool with valid Windows domain, SSH, or other valid authorisation so it can perform checks against the local system. This is of most value when looking for missing patches in an operating system or third party software and detecting installed applications.

Conclusion

An organisation wishing to secure its IT infrastructure needs to implement Vulnerability scanning as it is essential to Security Control.

Vulnerability scanning is recommended by the SANS Institute as a Critical Control and US-based NIST as a Security Management Control.

The results shown in this article show significant variation in discovered security vulnerabilities by different tools. It may be helpful to compare vulnerability scanners to anti-virus solutions. Both are important to security control and will enhance an organisation's security posture. However, as with anti-virus, a vulnerability scanner will not find all the bad things.

The following is common knowledge for most in the security industry who perform network vulnerability testing;

  • Check results for accuracy -> false positives.
  • Actively look for things that were missed -> false negatives.

A recommended approach to vulnerability scanning

  • Tune the vulnerability scan profiles to suit your requirements
  • Perform a detailed analysis of the results
  • Run secondary tools such as Nmap, a secondary vulnerability scanning solution and/or specialised tools. The use of multiple tools will provide a greater level of coverage and assist in confirming discovered vulnerabilities.

Performing internal focused testing in conjunction with external facing vulnerability scans adds value when working to secure Internet connected networks or servers.

Assess the risk and work on mitigation.

Remove limits with a full membership.

We host OpenVAS, Nmap and other Vulnerability Scanners.

Trusted tools. Hosted for easy access.

20 Comments

  • Ted
    Where you using the commercial versions of Nessus and Nexpose in your test? Do you hav any plans to test other commercial scanners? It would be great if the community could help out.
    • Home feed of Nessus and the Community version of Nexpose, however I believe the plugins are the same for both with only a delayed release. All vulnerabilities in the sample set were months or years old. I may look into other products when I get some time. :)
      • dre g
        Look into some of the open-source third-party tools out there, too. I would be curious to see Nessus vs. Nessus Pro vs. NeXpose Comm vs. NeXpose Pro vs. nmap with default nse scripts vs. nmap with an open-source third-party nse script like vulscan. It would also be interesting to see how these fair in the sectoolmarket.com test criteria and grounds (i.e. wavsep.googlecode.com). Shay Chen has done some interesting work there, and some of the Nessus numbers are pretty good.
  • vm auditor
    This is unfair to Nessus. Did you use the Professional feed or did you use the Home feed? Why did you use the External Network Profle and the rest you did a Full Audit? You should have created a Full Audit Profile with Nessus or use the Internal Network Audit to be FAIR. This is a very bias and not well though out review. I'm very disappointed you also did not detail the configurations of your scanners, such as range of ports scanned and did you use credentials (from your results, no you did not). Totally unfair and bias against Nessus
    • Thank you for your feedback and comments.
      Did you use the Professional feed or did you use the Home feed?
      Home Feed, my understanding is that the only difference between the two is that the plugins for the professional feed are released earlier than the home feed. All the vulnerabilities tested are months or years old so there should be no difference between the results.
      Why did you use the External Network Profle and the rest you did a Full Audit?
      Nessus does not have a Full Audit profile in a default configuration, they have Internal Network Scan and External Network Scan, I tried both and discovered more vulnerabilities with the External Network Scan Profile.
      I'm very disappointed you also did not detail the configurations of your scanners, such as range of ports scanned and did you use credentials (from your results, no you did not).
      My comments and notes do state that I only used the default profiles and I did not use credentials". I also recommend that people do perform credential based scans on Internal network scanning. Of course I could create a custom scan and tick all the boxes, however I wanted this test to be performed with the default configurations. Unfortunately many organisations only use default profiles. In creating this test my intention is not to attack any particular product, my aim was to highlight the fact that out of the box current vulnerability scanners are far from perfect. Operators of the tools should have a good understanding of the product and in many cases use multiple tools to confirm discovered vulnerabilities and find others that were missed. If a scanner only performs well with "credential scans", then perhaps it is not a network vulnerability scanner but closer to a software audit tool.
      • jimbean
        All aside, it doesn't matter which feed was used and if the review's biased or not. The goal of the review is to remind "point and click lovers" to use their frontal lobe and not muscle memory while tunning, anaylizing or exploring anything relative to vulnerability scanners. In fact, three important points are made at the end of the review and they are to: - Tune scanner security policies - Analyze the results - Run a variety of tools
      • bukovinai
        Hi The exploitable vulnerability don't 15 but much more.... (a lot) regards.
  • Paul Asadoorian
    vm auditor makes two great points: 1) Since Nessus did not have a Full Audit policy, you just used one of the other policies available. These policies are not meant to accomplish the goals you set out for in this test (I helped write them and define their purpose). 2) You did not use credentialed scans, which eliminates a huge result set and can even be used to weed out false positives found by all the tools in the test. In any case, I wrote an article with some suggestions for a better comparison, including a downloadable Nessus policy titled "Full Thorough Audit (slow)" You can find it here: http://pauldotcom.com/2012/08/the-right-way-to-configure-nes.html I hope you find it useful, and feel free to hit me up with any questions/comments/suggestions. BTW, in my scan, Nessus finds the ProFTD vulnerability on port 2121 and the Unreal IRCd backdoor ;) Cheers, Paul Asadoorian Product Evangelist Tenable Network Security
    • Paul, great to get feedback from someone so familiar with the Nessus scanner.
      1) Since Nessus did not have a Full Audit policy, you just used one of the other policies available. These policies are not meant to accomplish the goals you set out for in this test (I helped write them and define their purpose).
      The goal of determining how well 3 different vulnerability scanners perform against remotely exploitable network services using default scan profiles was accomplished. Anyone reading this now has an understanding that using only one product with a default scanning profile has limited coverage. In my experience I have seen many organisations where the operations staff run Nessus or another scanner using the default scan profiles, fix any discovered vulnerabilities and trust the network is secure.
      2) You did not use credentialed scans, which eliminates a huge result set and can even be used to weed out false positives found by all the tools in the test.
      I do recommended using credential scans. Even so before this testing I was under the impression that credential scans were used for detecting client side vulnerabilities and missing patches (stuff the network based scanner could not see). Since I have learnt something here (it is important to use credentials when testing network services), maybe others have too.
      In any case, I wrote an article with some suggestions for a better comparison, including a downloadable Nessus policy titled "Full Thorough Audit (slow)"
      You have some good product specific information in your article. I would counter that using a fully locked and loaded configuration in a test such as this would be akin to stacking the deck. I doubt many would get that configuration past a change control board when testing a network of production systems.
    • Ronnie Redd
      The page your are looking for does not exist. Please try using the search below: Did a search for "Full Thorough Audit" returns no results.
      • obelix
        It's now available at http://securityweekly.com/2012/08/24/the-right-way-to-configure-nes/
  • Dave Breslin
    "In creating this test my intention is not to attack any particular product, my aim was to highlight the fact that out of the box current vulnerability scanners are far from perfect" If this had been the sole intention and aim it could have been proved with using one vendor's scanner using a mixture of custom and out of the box scan policies, and been in the process a very educational article. Instead its clearly aimed at being a product comparison, just look at the title; "Nessus, OpenVAS and Nexpose VS Metasploitable".
    • Hi Dave, Thanks for your comments, its great to get more feedback from the Tenable? Team.
      If this had been the sole intention and aim it could have been proved with using one vendor's scanner using a mixture of custom and out of the box scan policies, and been in the process a very educational article.
      I disagree, I think if I had of performed a test using Nessus only and found that it discovered 7 out of 15 remotely exploitable network services using a default scan profile that would have been more of an attack. Instead I found that three different vulnerability scanning solutions all were limited in the detection of a sample set of remotely exploitable services using default scan profiles. This should be taken as not an attack but an educational piece, anyone reading this now knows that tweaking the default profiles and using credentials is not optional - it really is required in order to get greater coverage.
  • Mark Webster
    Thanks for the review,I have been using security scanners for years. I started out with the original ISS Scanner, I used to work for ISS. Then got into Nessus and have been using it for years. This opened me up to OpenVAS and now Nexpose. I will be checking those out. Guys don't forget about Web / Application Scanners Like HP Web Inspect, these guys were originally developers / security experts for ISS that broke off many years back and eventually got bought by HP. Lot of talent there too.
  • Takfly
    Cheers dude, I found your review extremely helpful.. I find it frustrating that people are attacking your methods for performing the test in the way that you did, you provide a table of comparison which as far as I'm concerned allows the reader to form their own conclusions.. it almost feels as if they are a bunch of Nessus sales folk!! The way I read it was that with each tool, you used the the preset which provided the most comprehensive results. I suppose a simple solution would be for these people to perform their own comparison of the products using their own methodology and then publish the results... Likelihood 0!! Again, Thank you!!
    • asdf
      At least 2 are/were from Tenable, Paul Asadoorian and Dave Breslin. Paul's comment is constructive and doesn't seem to be overly critical. vm auditor and Dave Breslin are much less constructive, given vm auditor's response he/she is also likely with Tenable.
  • Takfly
    tard!
  • WomBat
    dude..very nicely done, I found this very useful..I agree with Takfly disregard those who critisize and ask them to prove where they have done better, I find that usually silences the doomsayers.....
  • a
    There is an academic paper that does the same thing, but in a more rigurous manner: http://www.emeraldinsight.com/doi/abs/10.1108/09685221111173058 "– The purpose of this paper is to evaluate if automated vulnerability scanning accurately identifies vulnerabilities in computer networks and if this accuracy is contingent on the platforms used." "– Both qualitative comparisons of functionality and quantitative comparisons of false positives and false negatives are made for seven different scanners. The quantitative assessment includes data from both authenticated and unauthenticated scans. Experiments were conducted on a computer network of 28 hosts with various operating systems, services and vulnerabilities. This network was set up by a team of security researchers and professionals."
  • guest
    hey peter, thanks for taking the time to review these products, a good read and good recommendations :)