On the 9th of December 2021, the world became aware of a critical RCE vulnerability in the Log4j open source package that is buried in the software stacks of many organisations (CVE-2021-44228).
Versions of Log4j2 >= 2.0-beta9 and <= 2.16 are all affected by this vulnerability. The vulnerability is easy to exploit and is currently being attacked, with exploitation occurring in the wild.
The CVE is rated 10, and while it is not the first with such a high level, the big problem with this one is the Log4j software is deployed in systems where many will not even realise. It is a Java based dependency of other common software solutions. The scope and impact of this vulnerability won't be fully understood for some time.
Exploitation can occur through a range of vectors, such as stuffing the simple exploit code into a HTTP User Agent, HTTP Referrer, as well as any user supplied input such as web forms and login portals.
Who and What is affected
Product and vendor information can be found at this link that is being updated as new information becomes available.
A small sample of what is affected:
- Minecraft (server and java clients)
- VMware vCenter + many more products
- Apache Tomcat (not by default but if configured)
- Apache Solr
- Logstash
- Elasticsearch
- Graylog
- Security Onion
- Cisco Products (multiple *under investigation)
- UniFi Network Application
- ZAP Proxy
Remediation of CVE-2021-44228
A number of remediation options are available:
Best Option: Patch the Log4j library
Updating Log4j to a secure version (version 2.16.0 2.17.0) is the best way forward. Note that older versions (1.x) are not vulnerable. These have been out of support for many years but are not vulnerable to this issue so are not an immediate priority.
Second Option: Disable the Lookup Function
Unfortunately, there are going to be many outdated Java based systems running in organisations around the world where an upgrade is not an easy option. The system administrators may even be hesitant to touch the system in the event that they break it. Patching is not always an easy fix.
Disabling the Lookup function of the Log4j package is the next best bet. Note that the latest information is that disabling the Lookup function does not mitigate the vulnerability fully. As the issue is still evolving the best strategy is to check the vendor advice. Specific mitigation's may be required, depending on the deployment.
-Dlog4j2.formatMsgNoLookups=true
Firewall Application Server (outbound)
Current exploits require the Log4j server to make a connection to another (attacker controlled) system. If possible, firewall outbound connections from the APP server as this will block many attacks. Note that in this case, DNS ex-filtration of data is still possible if the APP server can resolve external DNS (likely).
The use of a firewall is not a fix but it is one layer of defence - and could potentially be implemented quickly.
Detecting the Vulnerability
We now have the ability to detect vulnerable Log4j systems using the latest OpenVAS signatures (rolled out 14/12/21 1600 UTC). The excellent team at Greenbone Networks (OpenVAS/GVM feed maintainers) released new signatures for detecting the vulnerability. In our testing before deployment we confirmed successful detection against a number of vulnerable lab systems. We are monitoring the situation closely and will release any updates as they become available and more information comes to hand.
Vulnerability Scanning for Log4J
Vulnerability Scanners (including OpenVAS / Greenbone Vulnerability Manager / Nesssus etc) using remote only testing will catch the low-hanging fruit; the easily accessible and exploitable Internet-facing systems. We have tested the newly released signatures from Greenbone Networks in our lab and can confirm that they detected a vulnerable version of Apache Solr remotely.
Keep in mind that no vulnerability scanner is 100% accurate. With this vulnerability. in particular, it is unlikely that there will be a vulnerability detection released that can definitively say that the vulnerability is present with a remote only scan. The problem is the attack surface is huge and varied. The number of places the malicious string could find its way to an instance of Log4j is almost endless.
Knowing your Software (assets)
The ideal solution for detection is knowing the software you have running within your organisation (accurate asset register) and the ability to patch and update software as required.
Use Attack Surface Mapping Tools and Vulnerability Scanners to find the gaps in your organisations network knowledge. Get the answers to the following questions; what services are you running, and what sites are running on those services. Get immediate benefit in the triage process allowing you to identify which systems to remediate as a priority.
Detection using Canary Tokens
An interesting Honeypot option is available for quick testing of an application. A canary token is a service that will monitor for hits from a query and alert you to the successful hit.
The Canary Tokens are made available for Free by Thinkst Canary - a very well-respected company within the Infosec space.
How the Canary Token Works
The smart people at Thinkst Canary have released a log4j token that will alert you to a successfully triggered Log4j exploit vector.
Getting Started
It's an amazing service, you don't even need to create an account:
1. Create a token, entering your email address for the alert to be sent to.
2. Copy the Log4j trigger code and use it in any forms or input you wish to test within your organisation.
In the example below, we tested the token against an old version of Graylog running in our lab. Pasting the code snippet straight into the search triggered the canary token alerting me to the fact that the vulnerability was present.
The alert was immediate and came straight into my inbox.
After adding the -Dlog4j2.formatMsgNoLookups=true parameter to the elasticsearch jvm.options we tried the same ElasticSearch query and found that the token did not trigger. Confirming both, the vulnerability was present initially, and the fact that the vulnerability has now been remediated.
Wrapping Up
For many security / operations teams the next weeks (or months) will be a challenge. We wish you all the best and we will keep monitoring the situation updating this blog post as needed.
Further Resources
Youtube Internet Storm Center Update - Good information from the Sans ISC.
CERT (CH) - Swiss Government Computer Emergency Response Team
Apache Log4j Security Vulnerabilities - Apache Logging Services pages listing the security vulnerabilities fixed in released versions of Apache Log4j 2
NVD: National Vulnerability Database CVE-2021-44228 Detail