An initial step in attacking a web application is Recon, and part of that entails enumerating hidden directories and files. Brute forcing web directories and filenames on a web server can often reveal unprotected web applications, scripts, old configuration files, and many other interesting things that should not be available to the public.
It is even possible to brute force virtual hosts to find hidden vhosts such as development sites or admin portals.
Gobuster Installation
Written in the Go language, this tool enumerates hidden files along with the remote directories. Using the command line it is simple to install and run on Ubuntu 20.04.
For version 2 its as simple as:
$ sudo apt install gobuster
The Linux package may not be the latest version of Gobuster. Check Repology: the packaging hub, which shows the package of Gobuster is 2.0.1 (at the time of this article). The Github repository shows a newer version V3.1.0. https://github.com/OJ/gobuster.git
Under "Easy installation" on the github page the options to install are binary releases, a Go install, and Building from source. For this install lets play around with the Go install. Gobuster needs Go to be at least v1.16
Setting up a Go environment (optional)
Download the GO install from here: https://go.dev/dl/
change to the directory where Downloads normally arrive and do the following;
--> extract $ sudo tar xvzf go1.17.7.linux-amd64.tar.gz --> change permissions $ sudo chown -R root:root ./go --> move to local directory $ sudo mv -v go /usr/local
A local environment variable called $GOPATH needs to be set up. Since Go 1.8 this is not essential, though still recommended as some third party tools are still dependent on it.
Add the following to the .bash_profile Locate in home directory with ls -la .
export GOPATH=/usr/local/go export PATH=$PATH:/usr/local/go/bin
To check its all worked and the Go environment is set up:
$ go version go version go1.17.7 linux/amd64
Now with the Go environment confirmed. Its simply a matter of using the following command to install Gobuster.
$ go install github.com/OJ/gobuster/v3@latest
check Gobuster is installed with:
$ gobuster version 3.1.0
How to use Gobuster
Gobuster is now installed and ready to use. The rest of the tutorial is how to use Gobuster to brute force for files and directories.
Gobuster modes and flags
Gobuster has a variety of modes/commands to use as shown below. This tutorial focuses on 3: DIR, DNS, and VHOST.
To see a general list of commands use: gobuster -h Each of these modes then has its own set of flags available for different uses of the tool.
$ gobuster -h Usage: gobuster [command] Available commands: dir Uses directory/file enumeration mode dns Uses DNS subdomain enumeration mode fuzz Uses fuzzing mode help Help about any command s3 Uses aws bucket enumeration mode version shows the current version vhost Uses VHOST enumeration mode Flags: --delay duration Time each thread waits between requests (e.g. 1500ms) -h, --help help for gobuster --no-error Don't display errors -z, --no-progress Don't display progress -o, --output string Output file to write results to (defaults to stdout) -p, --pattern string File containing replacement patters -q, --quiet Don't print the banner and other noise -t, --threads int Number of concurrent threads (default 10) -v, --verbose Verbose output (errors) -w, --wordlist string Path to the wordlist
Wordlists
Gobuster needs wordlists. One of the essential flags for gobuster is -w . Wordlists can be obtained from various places. Depending on the individual setup, wordlists may be preinstalled or found within other packages, including wordlists from Dirb or Dirbuster. The ultimate source and "Pentesters friend" is SecLists - https://github.com/danielmiessler/SecLists which is a compilation of numerous lists held in one location.
Gobuster DIR command
The DIR mode is used for finding hidden directories and files.
To find additional flags available to use gobuster dir --help
$ gobuster dir --help
Uses directory/file enumeration mode Usage: gobuster dir [flags] Flags: -f, --add-slash Append / to each request -c, --cookies string Cookies to use for the requests -d, --discover-backup Upon finding a file search for backup files --exclude-length ints exclude the following content length (completely ignores the status). Supply multiple times to exclude multiple sizes. -e, --expanded Expanded mode, print full URLs -x, --extensions string File extension(s) to search for -r, --follow-redirect Follow redirects -H, --headers stringArray Specify HTTP headers, -H 'Header1: val1' -H 'Header2: val2' -h, --help help for dir --hide-length Hide the length of the body in the output -m, --method string Use the following HTTP method (default "GET") -n, --no-status Don't print status codes -k, --no-tls-validation Skip TLS certificate verification -P, --password string Password for Basic Auth --proxy string Proxy to use for requests [http(s)://host:port] --random-agent Use a random User-Agent string -s, --status-codes string Positive status codes (will be overwritten with status-codes-blacklist if set) -b, --status-codes-blacklist string Negative status codes (will override status-codes if set) (default "404") --timeout duration HTTP Timeout (default 10s) -u, --url string The target URL -a, --useragent string Set the User-Agent string (default "gobuster/3.1.0") -U, --username string Username for Basic Auth --wildcard Force continued operation when wildcard found Global Flags: --delay duration Time each thread waits between requests (e.g. 1500ms) --no-error Don't display errors -z, --no-progress Don't display progress -o, --output string Output file to write results to (defaults to stdout) -p, --pattern string File containing replacement patterns -q, --quiet Don't print the banner and other noise -t, --threads int Number of concurrent threads (default 10) -v, --verbose Verbose output (errors) -w, --wordlist string Path to the wordlist
Flags
The 2 flags required to run a basic scan are -u -w. This example uses common.txt from the SecList wordlists.
user@matrix:$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/common.txt
Example results
===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url: https://example.com
[+] Method: GET
[+] Threads: 10
[+] Wordlist: /wordlists/Discovery/Web-Content/common.txt
[+] Negative Status codes: 404
[+] User Agent: gobuster/3.1.0
[+] Timeout: 10s
===============================================================
2022/03/01 10:34:16 Starting gobuster in directory enumeration mode
===============================================================
/assets
/css
/download
Not too many results and was quite heavy on the system processess. Results depend on the wordlist selected. It is worth working out which one is best for the job. The length of time depends on how large the wordlist is. It can also be worth creating a wordlist specific to the job at hand using a variety of resources.
Threads
Gobuster is fast, with hundreds of requests being sent using the default 10 threads. This speeds can create problems with the system it is running on. It could be beneficial to drop this down to 4.
Additionally it can be helpful to use the flag --delay duration Time each thread waits between requests (e.g. 1500ms). For example --delay 1s in other words, if threads is set to 4 and --delay to 1s, this will send 4 requests per second.
$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/big.txt -t 4 --delay 1s -o results.txt
Results
Results are shown in the terminal, or use the -o option to output results to a file example -o results.txt
user@matrix:$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/directory-list-2.3-small.txt -t 4 --delay 1s -o results.txt
===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url: https://example.co.uk/
[+] Method: GET
[+] Threads: 4
[+] Delay: 1s
[+] Wordlist: /wordlists/Discovery/Web-Content/directory-list-2.3-small.txt
[+] Negative Status codes: 404
[+] User Agent: gobuster/3.1.0
[+] Timeout: 10s
===============================================================
2022/03/08 12:12:19 Starting gobuster in directory enumeration mode
===============================================================
/admin
/aux
===============================================================
2022/03/08 12:46:57 Finished
===============================================================
Took a while, but by filtering the results to an output file its easy to see and retain for future enumerating, what was located. A few more interesting results this time.
Other DIR flag examples
The results above show status codes. To exclude status codes use -n
user@matrix:$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/big.txt -n -t 4 --delay 1s -o results.txt
An example of another flag to use is the -x File extension(s) to search for. This is for the times when a search for specific file extension or extensions is specified. Such as, -x .php or other only is required.
user@matrix:$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/big.txt -x .php, .txt -t 4 --delay 1s -o results.txt
Continue enumerating
Continue to enumerate results to find as much information as possible. Run gobuster again with the results found and see what else appears. Keep digging to locate those hidden directories.
$ gobuster dir -u https://example.com/aux -w /wordlists/Discovery/Web-Content/big.txt -t 4 --delay 1s -o results.txt
Gobuster DNS command
Use the DNS command to discover subdomains with Gobuster. To see the options and flags available specifically for the DNS command use: gobuster dns --help
user@matrix:$ gobuster dns --help Uses DNS subdomain enumeration mode Usage: gobuster dns [flags] Flags: -d, --domain string The target domain -h, --help help for dns -r, --resolver string Use custom DNS server (format server.com or server.com:port) -c, --show-cname Show CNAME records (cannot be used with '-i' option) -i, --show-ips Show IP addresses --timeout duration DNS resolver timeout (default 1s) --wildcard Force continued operation when wildcard found Global Flags: --delay duration Time each thread waits between requests (e.g. 1500ms) --no-error Don't display errors -z, --no-progress Don't display progress -o, --output string Output file to write results to (defaults to stdout) -p, --pattern string File containing replacement patterns -q, --quiet Don't print the banner and other noise -t, --threads int Number of concurrent threads (default 10) -v, --verbose Verbose output (errors) -w, --wordlist string Path to the wordlist
DNS example
$ gobuster dns -q -r 8.8.8.8 -d example.com -w wordlists/Discovery/DNS/subdomains-top1million-5000.txt -t 4 --delay 1s -o results.txt"
Breaking this down.
dns mode
-q --quiet : Don't print the banner and other noise
-r --resolver string : Use custom DNS server (format server.com or server.com:port)
-d --domain string
-w --wordlist string : Path to the wordlist
-t --threads
--delay -- delay duration
-o --output string : Output file to write results to (defaults to stdout)
Using another of the Seclists wordlists /wordlists/Discovery/DNS/subdomains-top1million-5000.txt.
Results
In this case, as the flag -q for quiet mode was used, only the results are shown, the Gobuster banner and other information are removed.
Found: www.example.com Found: nagios.example.com Found: dev.example.com Found: auto.example.com
The same search without the flag -q obviously gives the same results - and includes the banner information.
===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Domain: example.com
[+] Threads: 4
[+] Delay: 1s
[+] Resolver: 8.8.8.8
[+] Timeout: 1s
[+] Wordlist: /home/wordlists/subdomains-top1million-5000.txt
===============================================================
2022/03/18 16:20:35 Starting gobuster in DNS enumeration mode
===============================================================
Found: www.example.com
Found: nagios.example.com
Found: dev.example.com
Found: auto.example.com
===============================================================
2022/03/18 16:20:37 Finished
===============================================================
Gobuster VHost command
The vhost command discovers Virtual host names on target web servers. Virtual hosting is a technique for hosting multiple domain names on a single server.
Exposing hostnames on a server may reveal supplementary web content belonging to the target. Vhost checks if the subdomains exist by visiting the formed URL and cross-checking the IP address.
To brute-force virtual hosts, use the same wordlists as for DNS brute-forcing subdomains.
Similar to brute forcing subdomains eg. url = example.com, vhost looks for dev.example.com or beta.example.com etc.
For options and flags available use gobuster vhost --help
user@matrix:$ gobuster vhost --help
Uses VHOST enumeration mode Usage: gobuster vhost [flags] Flags: -c, --cookies string Cookies to use for the requests -r, --follow-redirect Follow redirects -H, --headers stringArray Specify HTTP headers, -H 'Header1: val1' -H 'Header2: val2' -h, --help help for vhost -m, --method string Use the following HTTP method (default "GET") -k, --no-tls-validation Skip TLS certificate verification -P, --password string Password for Basic Auth --proxy string Proxy to use for requests [http(s)://host:port] --random-agent Use a random User-Agent string --timeout duration HTTP Timeout (default 10s) -u, --url string The target URL -a, --useragent string Set the User-Agent string (default "gobuster/3.1.0") -U, --username string Username for Basic Auth Global Flags: --delay duration Time each thread waits between requests (e.g. 1500ms) --no-error Don't display errors -z, --no-progress Don't display progress -o, --output string Output file to write results to (defaults to stdout) -p, --pattern string File containing replacement patterns -q, --quiet Don't print the banner and other noise -t, --threads int Number of concurrent threads (default 10) -v, --verbose Verbose output (errors) -w, --wordlist string Path to the wordlist
As shown above the Global flags are the same as for the all modes. Again, the 2 essential flags are the -u URL and -w wordlist. Not essential but useful -o output file and -t threads, -q for quiet mode to show the results only.
Vhost example
user@matrix:$ gobuster vhost -u https://example.com -t 50 -w /wordlists/Discovery/DNS/subdomains-top1million-5000.txt
Results
===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url: https://example.com
[+] Method: GET
[+] Threads: 4
[+] Wordlist: /wordlists/subdomains-top1million-5000.txt
[+] User Agent: gobuster/3.1.0
[+] Timeout: 10s
===============================================================
2022/03/22 10:21:38 Starting gobuster in VHOST enumeration mode
===============================================================
Found: auto.example.com (Status: 200) [Size: 162]
Found: beta.example.com (Status: 200) [Size: 162]
Found: apache.example.com (Status: 200) [Size: 162]
===============================================================
2022/03/22 10:21:39 Finished
===============================================================
Conclusion
Gobuster is a useful tool for recon and increasing the knowledge of the attack surface. Start with a smaller size wordlist and move to the larger ones as results will depend on the wordlist chosen. Keep enumerating. Don't stop at one search, it is surprising what is just sitting there waiting to be discovered.