SECURITY RESEARCH, TOOLS, TUTORIAL |

Gobuster tutorial

You would be surprised at what people leave unprotected on a web server. An initial step in attacking a web application is Recon, and part of that entails enumerating hidden directories and files. Brute forcing web directories and filenames on a web server can often reveal unprotected web applications, scripts, old configuration files, and many other interesting things that should not be available to the public.

It is even possible to brute force virtual hosts to find hidden vhosts such as development sites or admin portals.

Gobuster is an aggressive scan. Its noisy and is noticed. Only use against systems you have permissions to scan against

Gobuster Installation

Written in the Go language, this tool enumerates hidden files along with the remote directories. Using the command line it is simple to install and run on Ubuntu 20.04.

For version 2 its as simple as:

$ sudo apt install gobuster 

The Linux package may not be the latest version of Gobuster. Check Repology: the packaging hub, which shows the package of Gobuster is 2.0.1 (at the time of this article). The Github repository shows a newer version V3.1.0. https://github.com/OJ/gobuster.git

screenshot of information about gobuster

Under "Easy installation" on the github page the options to install are binary releases, a Go install, and Building from source. For this install lets play around with the Go install. Gobuster needs Go to be at least v1.16

Setting up a Go environment (optional)

Download the GO install from here: https://go.dev/dl/

change to the directory where Downloads normally arrive and do the following;

--> extract
$ sudo tar xvzf go1.17.7.linux-amd64.tar.gz 
--> change permissions
$ sudo chown -R root:root ./go
--> move to local directory
$ sudo mv -v go /usr/local

A local environment variable called $GOPATH needs to be set up. Since Go 1.8 this is not essential, though still recommended as some third party tools are still dependent on it.

Add the following to the .bash_profile Locate in home directory with ls -la .

export GOPATH=/usr/local/go
export PATH=$PATH:/usr/local/go/bin

To check its all worked and the Go environment is set up:

$ go version
go version go1.17.7 linux/amd64 

Now with the Go environment confirmed. Its simply a matter of using the following command to install Gobuster.

$ go install github.com/OJ/gobuster/v3@latest

check Gobuster is installed with:

$ gobuster version
3.1.0

How to use Gobuster

Gobuster is now installed and ready to use. The rest of the tutorial is how to use Gobuster to brute force for files and directories.

Gobuster modes and flags

Gobuster has a variety of modes/commands to use as shown below. This tutorial focuses on 3: DIR, DNS, and VHOST.

To see a general list of commands use: gobuster -h Each of these modes then has its own set of flags available for different uses of the tool.

 $ gobuster -h 
Usage:
  gobuster [command]

Available commands:
  dir         Uses directory/file enumeration mode
  dns         Uses DNS subdomain enumeration mode
  fuzz        Uses fuzzing mode
  help        Help about any command
  s3          Uses aws bucket enumeration mode
  version     shows the current version
  vhost       Uses VHOST enumeration mode

Flags: 
      --delay duration    Time each thread waits between requests (e.g. 1500ms)
  -h, --help              help for gobuster
      --no-error          Don't display errors
  -z, --no-progress       Don't display progress
  -o, --output string     Output file to write results to (defaults to stdout)
  -p, --pattern string    File containing replacement patters
  -q, --quiet             Don't print the banner and other noise
  -t, --threads int       Number of concurrent threads (default 10)
  -v, --verbose           Verbose output (errors)
  -w, --wordlist string   Path to the wordlist
  

Wordlists

Gobuster needs wordlists. One of the essential flags for gobuster is -w . Wordlists can be obtained from various places. Depending on the individual setup, wordlists may be preinstalled or found within other packages, including wordlists from Dirb or Dirbuster. The ultimate source and "Pentesters friend" is SecLists - https://github.com/danielmiessler/SecLists which is a compilation of numerous lists held in one location.

Gobuster DIR command

The DIR mode is used for finding hidden directories and files.

To find additional flags available to use gobuster dir --help

$ gobuster dir --help
Uses directory/file enumeration mode

Usage:
  gobuster dir [flags]

Flags:
  -f, --add-slash                       Append / to each request
  -c, --cookies string                  Cookies to use for the requests
  -d, --discover-backup                 Upon finding a file search for backup files
      --exclude-length ints             exclude the following content length (completely ignores the status). Supply multiple times to exclude multiple sizes.
  -e, --expanded                        Expanded mode, print full URLs
  -x, --extensions string               File extension(s) to search for
  -r, --follow-redirect                 Follow redirects
  -H, --headers stringArray             Specify HTTP headers, -H 'Header1: val1' -H 'Header2: val2'
  -h, --help                            help for dir
      --hide-length                     Hide the length of the body in the output
  -m, --method string                   Use the following HTTP method (default "GET")
  -n, --no-status                       Don't print status codes
  -k, --no-tls-validation               Skip TLS certificate verification
  -P, --password string                 Password for Basic Auth
      --proxy string                    Proxy to use for requests [http(s)://host:port]
      --random-agent                    Use a random User-Agent string
  -s, --status-codes string             Positive status codes (will be overwritten with status-codes-blacklist if set)
  -b, --status-codes-blacklist string   Negative status codes (will override status-codes if set) (default "404")
      --timeout duration                HTTP Timeout (default 10s)
  -u, --url string                      The target URL 
  -a, --useragent string                Set the User-Agent string (default "gobuster/3.1.0")
  -U, --username string                 Username for Basic Auth
      --wildcard                        Force continued operation when wildcard found

Global Flags:
      --delay duration    Time each thread waits between requests (e.g. 1500ms)
      --no-error          Don't display errors
  -z, --no-progress       Don't display progress
  -o, --output string     Output file to write results to (defaults to stdout)
  -p, --pattern string    File containing replacement patterns
  -q, --quiet             Don't print the banner and other noise
  -t, --threads int       Number of concurrent threads (default 10)
  -v, --verbose           Verbose output (errors)
  -w, --wordlist string   Path to the wordlist

Flags

The 2 flags required to run a basic scan are -u -w. This example uses common.txt from the SecList wordlists.

user@matrix:$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/common.txt  

Example results
===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     https://example.com
[+] Method:                  GET
[+] Threads:                 10
[+] Wordlist:                /wordlists/Discovery/Web-Content/common.txt
[+] Negative Status codes:   404
[+] User Agent:              gobuster/3.1.0
[+] Timeout:                 10s
===============================================================
2022/03/01 10:34:16 Starting gobuster in directory enumeration mode
===============================================================
/assets              
/css                  
/download             

Not too many results and was quite heavy on the system processess. Results depend on the wordlist selected. It is worth working out which one is best for the job. The length of time depends on how large the wordlist is. It can also be worth creating a wordlist specific to the job at hand using a variety of resources.

Threads

Gobuster is fast, with hundreds of requests being sent using the default 10 threads. This speeds can create problems with the system it is running on. It could be beneficial to drop this down to 4.

screenshot of Gobuster's Global flags highlighting -t Threads

Additionally it can be helpful to use the flag --delay duration Time each thread waits between requests (e.g. 1500ms). For example --delay 1s in other words, if threads is set to 4 and --delay to 1s, this will send 4 requests per second.

$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/big.txt -t 4 --delay 1s -o results.txt

Results

Results are shown in the terminal, or use the -o option to output results to a file example -o results.txt

user@matrix:$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/directory-list-2.3-small.txt -t 4 --delay 1s -o results.txt

===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     https://example.co.uk/
[+] Method:                  GET
[+] Threads:                 4
[+] Delay:                   1s
[+] Wordlist:                /wordlists/Discovery/Web-Content/directory-list-2.3-small.txt
[+] Negative Status codes:   404
[+] User Agent:              gobuster/3.1.0
[+] Timeout:                 10s
===============================================================
2022/03/08 12:12:19 Starting gobuster in directory enumeration mode
===============================================================
/admin
/aux
===============================================================
2022/03/08 12:46:57 Finished
===============================================================

Took a while, but by filtering the results to an output file its easy to see and retain for future enumerating, what was located. A few more interesting results this time.

Gobusters Dir results output
Other DIR flag examples

The results above show status codes. To exclude status codes use  -n 

user@matrix:$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/big.txt  -n  -t 4 --delay 1s -o results.txt

An example of another flag to use is the  -x  File extension(s) to search for. This is for the times when a search for specific file extension or extensions is specified. Such as, -x .php or other only is required.

user@matrix:$ gobuster dir -u https://example.com -w /wordlists/Discovery/Web-Content/big.txt  -x .php, .txt  -t 4 --delay 1s -o results.txt

Continue enumerating

Continue to enumerate results to find as much information as possible. Run gobuster again with the results found and see what else appears. Keep digging to locate those hidden directories.

Gobusters Dir results output
$ gobuster dir -u https://example.com/aux -w /wordlists/Discovery/Web-Content/big.txt -t 4 --delay 1s -o results.txt

Gobuster DNS command

Use the DNS command to discover subdomains with Gobuster. To see the options and flags available specifically for the DNS command use: gobuster dns --help

user@matrix:$ gobuster dns --help
Uses DNS subdomain enumeration mode

Usage:
  gobuster dns [flags]

Flags:
  -d, --domain string      The target domain
  -h, --help               help for dns
  -r, --resolver string    Use custom DNS server (format server.com or server.com:port)
  -c, --show-cname         Show CNAME records (cannot be used with '-i' option)
  -i, --show-ips           Show IP addresses
      --timeout duration   DNS resolver timeout (default 1s)
      --wildcard           Force continued operation when wildcard found

Global Flags:
      --delay duration    Time each thread waits between requests (e.g. 1500ms)
      --no-error          Don't display errors
  -z, --no-progress       Don't display progress
  -o, --output string     Output file to write results to (defaults to stdout)
  -p, --pattern string    File containing replacement patterns
  -q, --quiet             Don't print the banner and other noise
  -t, --threads int       Number of concurrent threads (default 10)
  -v, --verbose           Verbose output (errors)
  -w, --wordlist string   Path to the wordlist

DNS example

$ gobuster dns -q -r 8.8.8.8 -d example.com -w wordlists/Discovery/DNS/subdomains-top1million-5000.txt -t 4 --delay 1s -o results.txt"	 

Breaking this down.

dns mode
-q --quiet : Don't print the banner and other noise
-r --resolver string : Use custom DNS server (format server.com or server.com:port)
-d --domain string
-w --wordlist string : Path to the wordlist
-t --threads
--delay -- delay duration
-o --output string : Output file to write results to (defaults to stdout)

Using another of the Seclists wordlists /wordlists/Discovery/DNS/subdomains-top1million-5000.txt.

Results

In this case, as the flag -q for quiet mode was used, only the results are shown, the Gobuster banner and other information are removed.

Found: www.example.com
Found: nagios.example.com
Found: dev.example.com   
Found: auto.example.com                                

The same search without the flag -q obviously gives the same results - and includes the banner information.

===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Domain:     example.com
[+] Threads:    4
[+] Delay:      1s
[+] Resolver:   8.8.8.8
[+] Timeout:    1s
[+] Wordlist:   /home/wordlists/subdomains-top1million-5000.txt
===============================================================
2022/03/18 16:20:35 Starting gobuster in DNS enumeration mode
===============================================================

Found: www.example.com
Found: nagios.example.com
Found: dev.example.com   
Found: auto.example.com  
                              
===============================================================
2022/03/18 16:20:37 Finished
===============================================================

Gobuster VHost command

The vhost command discovers Virtual host names on target web servers. Virtual hosting is a technique for hosting multiple domain names on a single server.

Exposing hostnames on a server may reveal supplementary web content belonging to the target. Vhost checks if the subdomains exist by visiting the formed URL and cross-checking the IP address.

To brute-force virtual hosts, use the same wordlists as for DNS brute-forcing subdomains.

Similar to brute forcing subdomains eg. url = example.com, vhost looks for dev.example.com or beta.example.com etc.

For options and flags available use gobuster vhost --help

user@matrix:$ gobuster vhost --help
Uses VHOST enumeration mode

Usage:
  gobuster vhost [flags]

Flags:
  -c, --cookies string        Cookies to use for the requests
  -r, --follow-redirect       Follow redirects
  -H, --headers stringArray   Specify HTTP headers, -H 'Header1: val1' -H 'Header2: val2'
  -h, --help                  help for vhost
  -m, --method string         Use the following HTTP method (default "GET")
  -k, --no-tls-validation     Skip TLS certificate verification
  -P, --password string       Password for Basic Auth
      --proxy string          Proxy to use for requests [http(s)://host:port]
      --random-agent          Use a random User-Agent string
      --timeout duration      HTTP Timeout (default 10s)
  -u, --url string            The target URL
  -a, --useragent string      Set the User-Agent string (default "gobuster/3.1.0")
  -U, --username string       Username for Basic Auth

Global Flags:
      --delay duration    Time each thread waits between requests (e.g. 1500ms)
      --no-error          Don't display errors
  -z, --no-progress       Don't display progress
  -o, --output string     Output file to write results to (defaults to stdout)
  -p, --pattern string    File containing replacement patterns
  -q, --quiet             Don't print the banner and other noise
  -t, --threads int       Number of concurrent threads (default 10)
  -v, --verbose           Verbose output (errors)
  -w, --wordlist string   Path to the wordlist

As shown above the Global flags are the same as for the all modes. Again, the 2 essential flags are the -u URL and -w wordlist. Not essential but useful -o output file and -t threads, -q for quiet mode to show the results only.

Vhost example

user@matrix:$ gobuster vhost -u https://example.com -t 50 -w /wordlists/Discovery/DNS/subdomains-top1million-5000.txt 

Results

===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:          https://example.com
[+] Method:       GET
[+] Threads:      4
[+] Wordlist:     /wordlists/subdomains-top1million-5000.txt
[+] User Agent:   gobuster/3.1.0
[+] Timeout:      10s
===============================================================
2022/03/22 10:21:38 Starting gobuster in VHOST enumeration mode
===============================================================
Found: auto.example.com (Status: 200) [Size: 162]
Found: beta.example.com (Status: 200) [Size: 162]
Found: apache.example.com (Status: 200) [Size: 162]
                                                        
===============================================================
2022/03/22 10:21:39 Finished
===============================================================
To see Gobuster being used check out Ippsec walkthrough of HTB Toby released Apr 2022.

Conclusion

Gobuster is a useful tool for recon and increasing the knowledge of the attack surface. Start with a smaller size wordlist and move to the larger ones as results will depend on the wordlist chosen. Keep enumerating. Don't stop at one search, it is surprising what is just sitting there waiting to be discovered.