Most organizations extensively use third-party software products and libraries to increase efficiency and decrease costs. As various data breaches and Supply Chain Attacks have shown, this can be dangerous if they fail to keep this software up-to-date and thus free from known vulnerabilities. Therefore, it is crucial to continuously monitor the security of any third party software and software components. This, in turn, requires efficient and precise tools to match specific software versions against known vulnerabilities.
As IT security professionals we regularly encounter this challenge as well: Determining whether the software we discover during engagements has any known vulnerabilities. Retrieving such information is tedious and not as straightforward as one might think. We could not find one software solution that solves this challenge to our needs and satisfaction. Therefore, we developed our own solution, search_vulns, which we believe overcomes the various shortcomings of similar solutions.
In this first blog post we initially detail all of the requirements, a good software solution for this problem should meet. To understand how well our solution performs, we conducted a thorough evaluation of six similar software solutions. This evaluation is based on our requirements and its results are presented in detail next. As we will see, each one of these six solutions has one or more shortcomings that make it less usable, reliable and trustworthy. In our second blog post we present our own tool, search_vulns, which meets all of our requirements and solves their various challenges. We will go in-depth about its inner workings that allow it to outperform similar solutions and become our go-to tool.
Motivation & Requirements
Being pentesters, we are regularly confronted with third party software during our security assessments. As a result, we require an efficient and precise way of determining whether this software is affected by any known vulnerabilities and exploits. While many databases and search engines exist for this purpose, they commonly omit important vulnerabilities (have false negatives) or return vulnerabilities the queried software is not affected by (have false positives). Therefore, we had to utilize various databases and search engines during a pentest, because there was no single database we believed to be precise enough. To conduct more detailed research on this problem, we initially defined our requirements for a good solution:
- Recall and Precision: We need the tool to return all known vulnerabilities that exist for the queried software (recall, no false negatives). At the same time, we do not want the tool to return vulnerabilities that the queried software is not affected by (precision, no false positives).
- Use Multiple Data Sources: There is no single data source that contains all known vulnerabilities. MITRE and their Common Vulnerabilities and Exposures (CVE) program keeps track of many vulnerabilities, but not all of them. Some vulnerabilities might have been reported via GitHub or GitLab and never received a CVE entry. Regarding exploits, there also is no single database. Some databases keep track of some exploits, like the Exploit-DB, 0day.today or PoC-in-GitHub. Ideally, the tool utilizes as many data sources as possible to minimize the need for any further research.
- Accept Flexible Input: The tool should be very flexible in accepting and properly processing input. While a concept for uniquely identifying software exists, namely the Common Platform Enumeration (CPE), it is impossible to know the correct CPE for every software you encounter. In addition, the way a software name is presented can vary depending on the component presenting it. For example, sometimes you find Apache 2.4.25 and sometimes Apache HTTP server 2.4.25. The tool, of course, should work with both variants (and more).
- Be as Up-to-Date as Possible: The tool's data sources must always be up-to-date. Primarily this refers to the actual vulnerability data. However, this can also refer to a database of software names and versions. In addition, the tool should make attempts to deal with slight inconsistencies. For example, the latest vulnerability data may refer to software versions that are not contained in the software database yet. The tool must be able to compensate for this and still return all known vulnerabilities.
- Provide Export Functionalities: Being able to retrieve all known vulnerabilities is great. Not being able to share this information with others not so much. As security professionals we regularly write reports detailing our findings. Having to manually copy tool results and modify them to our needs can be an unnecessary nuisance. Hence, we would like the tool to provide some basic export functionality.
- Be Usable in Automated Workflows, Fully Offline: Automation is a computer person's best friend. Specifically in the IT-security field, vulnerability scanners are regularly used to supplement manual testing. In fact, we also have our own custom toolchain to ensure constant efficiency and quality results. To enhance this toolchain, we want the tool to easily integrate into it.
While we come from the pentesting background, we think a tool that fulfills all of these requirements can be a handy companion in most IT security fields. For example, security auditors, system administrators and security engineers also constantly need to be aware of any known vulnerabilities in the software they monitor. We believe that our tool will be a great benefit to them as well.
Comparison of Existing Tools and Databases
To compare the performance of our new tool (search_vulns) to existing solutions, we evaluated several similar tools and databases with regard to our requirements. These six tools are: nvdtools, cve-search, CVE Quick Search, Snyk DB, VulDB and Rapid7 DB.
Comparison Overview
A brief overview of our results is illustrated in the following table.
| Tool | Result Quality | Data Sources | Input | Result Export | API | Performance | Open Source |
|---|---|---|---|---|---|---|---|
| nvdtools | + | ++ | + | ✗ | ✗ | + | ✓ |
| cve-search | + | + | + | + | ✓ | ++ | ✓ |
| CVE Quick Search | + | + | +++ | + | ✓ | +++ | ✗ |
| Snyk DB | ++ | + | + | ✗ | ✗ | +++ | ✗ |
| VulDB | ++ | +++ | + | + | ✓ | ++ | ✗ |
| Rapid7 DB | ++ | ++ | + | ✗ | ✗ | +++ | ✗ |
| search_vulns | +++ | ++ | +++ | ++ | ✓ | +++ | ✓ |
The last row of this table already shows how well the tool we developed satisfies our requirements. Note that Result Quality combines the requirements Recall and Precision and Being Up-to-Date.
Detailed Comparison
The following sections detail the evaluation results with regard to our agreed-upon requirements. Here, examples are provided and the different tools are compared with each other. Since Common Platform Enumeration (CPE) strings are a central component of these tools and the evaluation, we provide a brief overview beforehand:
A CPE string is a standardized way to describe and identify specific hardware, operating systems, and software applications. From a logical perspective, CPE strings can be compared to Uniform Resource Identifiers (URIs), which are a standardized way to uniquely identify resources. The structure of a CPE string (in the latest revision 2.3) is the following:
cpe:2.3:<part>:<vendor>:<product>:<version>:<update>:<edition>:<language>:<sw_edition>:<target_sw>:<target_hw>:<other>The individual fields are somewhat self explanatory. Still, more information and clear definitions can be obtained from the official specification by NIST. An example CPE for the application OpenManage Server Adminstrator (OMSA) 9.4.0.2 by Dell is:
cpe:2.3:a:dell:openmanage_server_administrator:9.4.0.2:*:*:*:*:*:*:*Recall and Precision
The false positive and false negative rates should be as low possible to provide good results. Not a single solution we evaluated performed truly well here in our eyes. While certain tools returned irrelevant vulnerabilities, others missed important vulnerabilities.
One software that commonly caused incomplete results is Proftpd 1.3.3c (cpe:2.3:a:proftpd:proftpd:1.3.3:c:*:*:*:*:*:*). CVE Quick Search, for example, missed all vulnerabilities from 2010 as well as CVE-2012-6095, CVE-2020-9272 and CVE-2021-46854. Rapid7 also missed these vulnerabilities and more. In comparison, nvdtools returned extraneous results, like CVE-2010-4221 and CVE-2010-3867, which affect only ProFTPD before 1.3.3c, according to their description. Since VulDB or Snyk did not allow searches with a version, they naturally returned undesired results when we searched for just ProFTPD. Also, when searching for the CPE on VulDB, it returned completely unrelated results. Furthermore, VulDB severly limited the number of results and queries for unregistered users. Only cve-search (via CIRCL's online version) returned all relevant vulnerabilities and no others.
Another sample query we used was Thingsboard 3.4.1 (cpe:2.3:a:thingsboard:thingsboard:3.4.1:*:*:*:*:*:*:*). CVE Quick Search did not return any vulnerabilities, presumably because the tool was not maintained regularly. Similarly, CIRCL's version of cve-search missed the relatively new vulnerability CVE-2022-48341. The Rapid7 DB also did not return any vulnerabilities, even when just querying the software without version. Thus, the Rapid7 DB did not seem to track this software. The other two online solutions, Snyk and VulDB, only accepted a search without version. Still, they also missed vulnerabilities when being queried with just Thingsboard. VulDB missed CVE-2022-40004, while Snyk also missed CVE-2020-27687, CVE-2022-31861, CVE-2022-45608 and more.
The third query we include here is Redis 3.2.3. The core challenge with this query is that there are two valid CPEs: cpe:2.3:a:redis:redis:3.2.3:*:*:*:*:*:*:* and cpe:2.3:a:redislabs:redis:3.2.3:*:*:*:*:*:*:*. Any software that is based on matching vulnerabilities with queried CPEs is therefore prone to having issues with this. For nvdtools, cve-search and CVE Quick Search this became immediately apparent, as the user had to choose a CPE they want to retrieve vulnerabilities for. Snyk and VulDB did not appear to have issues with this, but also didn't allow specifically searching for version 3.2.3. The Rapid7 DB allowed searching for this version, but missed several vulnerabilities like CVE-2016-8339, CVE-2018-11218 or CVE-2016-10517.
The problem above is very similar to another: deprecated CPEs. Sometimes the NVD, which keeps a dictionary of all their official CPEs, deprecates one CPE for another. For example, all CPEs for Piwik, like cpe:2.3:a:piwik:piwik:0.4.5:*:*:*:*:*:*:* have been deprecated with ones for Matomo because of a rebranding (see here). As a result, vulnerability databases may miss vulnerabilities if they only search for the old or the new software title. We did not evaluate the six software solutions in this aspect, however, and would just like to mention this challenge here.
The last challenge we highlight here is complex version numbers. For example, there is the software and version Huawei Uma V200R001C00SPC100 with CPE cpe:2.3:a:huawei:uma:v200r001c00spc100:*:*:*:*:*:*:*. If a vulnerability affects version V200R001C00SPC200 and below, it could be difficult to decide automatically whether version V200R001C00SPC100 is affected or not. cve-search, via CIRLC's online version, solved the challenge in this case and returned correct results. CVE Quick Search, on the other hand, failed and did not return any results. The Snyk DB and the Rapid7 DB also didn't return any results, presumably because they didn't track this software. VulDB did not return any results when searching with the version number and missed vulnerabilities when searching without it.
The results of this part of the evaluation are summarized in the following table. A check mark indicates that the tool returned exactly all relevant vulnerabilities for the query, no more and no less. A cross means that the tool returned either too few vulnerabilities or too many.
| Tool | Proftpd 1.3.3c | Thingsboard 3.4.1 | Redis 3.2.3 | Piwik 0.4.5 | Huawei UMA v200r001c00spc100 |
|---|---|---|---|---|---|
| nvdtools | ✗ | ✓ | ✗ | N/A | ✗ |
| cve-search | ✓ | ✗ | ✗ | N/A | ✓ |
| CVE Quick Search | ✗ | ✗ | ✗ | N/A | ✗ |
| Snyk DB | ✗ | ✗ | ✗ | N/A | ✗ |
| VulDB | ✗ | ✗ | ✗ | N/A | ✗ |
| Rapid7 DB | ✗ | ✗ | ✗ | N/A | ✗ |
| search_vulns | ✓ | ✓ | ✓ | ✓ | ✓ |
Having a good recall and precision for vulnerability search is quite challenging and none of the solutions we identified were sufficiently good at it for our standards.
Data Sources
For three of the solutions we evaluated, the National Vulnerability Database's (NVD) vulnerability data was the central data source. In particular, cve-search and CVE Quick Search solely used the NVD's data. nvdtools additionally claims to support using data from Trellix (formerly FireEye), Flexera, iDefense, the Rustsec Advisory-DB, the Snyk DB and vFeed. This was not evaluated in detail, however, because most of these services require a premium account.
The other three solutions, namely the Snyk DB, VulDB and the Rapid7 DB manage and use their own vulnerability database. Snyk is heavily focused on open source software and software distributed by package managers. The details of how Rapid7's DB is built, are unclear. However, since they have a GitHub fork of cvelist and cooperate with Snyk, their DB should include vulnerabilities from these two sources. Lastly, VulDB appears to have the most comprehensive custom database, when inspecting their listing of data sources. The extent to which all of these different sources are thoroughly evaluated and included into their database was not clear.
Input Restrictions
Across all evaluated tools, we identified three different input formats that the solutions would accept: A (partial) Common Platform Enumeration (CPE) string, a software product name, a software product name and version. A CPE string represents a software ID and is difficult to determine without further research. Consequently, for a tool to be easy to use, it must not require a user to enter a CPE string. Furthermore, a tool should allow users to refine their search by providing a software version alongside its title.
The following table illustrates which tool accepted which types of input:
| CPE | Software Title | Software Title + Version | |
|---|---|---|---|
| nvdtools | ✓ | ✗ | ✗ |
| cve-search | ✓ | ✓ | ✗ |
| CVE Quick Search | ✓ | ✓ | ✓ |
| Snyk DB | ✗ | ✓ | ✗ |
| VulDB | ✓ | ✓ | ✗ |
| Rapid7 DB | ✗ | ✓ | ✓ |
| search_vulns | ✓ | ✓ | ✓ |
Similarly to the first table, we decided to jump ahead a little and show how our tool performs.
Data Timeliness
Out of the six tools we evaluated we only classify CVE Quick Search as truly out-ouf-date. As an example, it could not inform us about any vulnerabilities for WordPress 5.8.3, even though this software has several known vulnerabilities dating back to as early as June 2022. A similar behavior was observed when querying the database for the software Thingsboard 3.4.1.
Naturally, nvdtools and cve-search are as up-to-date as the user keeps them, because they are offline tools. Both of them are also under active development, ensuring compatibility with new formats of their data sources. We did, however, observe that CIRCL's version of cve-search was slightly out-of-date as it did not return CVE-2022-48341 for Thingsboard 3.4.1 (see here).
Regarding the remaining three databases, our impression is that they are generally kept up-to-date. However, small imperfections appear to exist. For instance, we discovered that the Snyk DB and the Rapid7 DB do not seem to report CVE-2016-1000027 and CVE-2023-20863 for VMWare Spring Framework 5.3.26 (see here and here, respectively).
During the development of our own tool, we discovered an inconsistency in the NVD's data sources: Its vulnerability data is kept up-to-date quite well, while its dictionary of all existing software generally is not. As a result, software solutions that utilize the NVD's software databases may find that they cannot report any vulnerabilities for newer software. Sticking with the example VMWare Spring Framework 5.3.26 from above, we clearly observed this behavior with CVE Quick Search. It simply did not even allow us to search for this software, even though it exists and has vulnerabilities. Lacking the internal view of how the other solutions work, we can only assume from our observations that this issue may affect them as well to some degree. What to take away from this paragraph is that this issue generally exists and has to be alleviated as best as possible by a robust software solution.
Export of Results
Only three solutions we evaluated supported exporting their results. CVE Quick Search allowed its users to export a list of the identified vulnerabilities in a fixed CSV format. cve-search also allowed a vulnerability export in CSV format, but also in other formats like HTML, JSON or XML. VulDB, in comparison, did not support exporting the individual vulnerabilities. Instead it provides multiple graphs and charts that summarize the vulnerabilities' severity, exploitability, types and CVSS vectors.
nvdtools, the Snyk DB and the Rapid7 DB did not support exporting their results.
Performance
For the two offline solutions, performance strongly depends on the hardware of the underlying system. However, on general purpose notebooks we experienced search times in the magnitude of seconds when using cve-search and minutes when using nvdtools. In addition, we observed memory allocations of several hundred megabytes, making usage in resource-constrained environments difficult.
The online solutions generally responded quickly, at most taking ten seconds to return results. We consider this good and appropriate for regular usage.
Online vs. Offline
Not one of the solutions we evaluated provided both: an offline and online interface. CVE Quick Search, Snyk DB, VulDB, and the Rapid7 DB are only usable online. Out of these, only CVE Quick Search and VulDB provide an API as a premium service, so that they can be used in automated workflows on Internet-connected machines.
In comparison, nvdtools and cve-search are tools that primarily facilitate offline use, as they build a local vulnerability database during their setup. However, these tools can also be wrapped by a web application and provided as an online service, which the Computer Incident Response Center Luxembourg (CIRCL) has accomplished with cve-search.
In summary, we could not find one tool that is centrally developed for online as well as offline use.
Open Source
We are strong supporters of open source projects. As such it was disheartening to see that most established software solutions for this problem are closed source. This applies to all of the online databases we evaluated, including CIRLC's wrapper of cve-search. While there is a GitHub project for the latter, as well as one for Snyk, they are both outdated and no longer maintained.
Summary & Outlook
In summary, as IT security professionals we need an efficient and reliable solution for identifying known vulnerabilities in a software product, given its name and version. We first defined multiple requirements such a solution needs to fulfill. These are for example to return high quality and up-to-date results, use various data sources, accept flexible input and offer export functionalities for results. Thereafter, we evaluated six solutions to determine to what extent they satisfy these requirements. Here we have arrived at the conclusion that not one of these solutions sufficiently meets our requirements. Results often omitted important vulnerabilities or included irrelevant vulnerabilities. In addition, most solutions require a certain input format or cannot be queried with a software version. Furthermore, the exporting capabilities of all tools were quite limited and did not allow us to easily include identified vulnerabilities in our reports.
As a consequence, we started developing our own tool: search_vulns. While our tool is still in development, it already solves many of the challenges we described earlier. For more information on search_vulns and a deep dive into its inner workings, stay tuned for our second blog post.