CTF

Proper Preparation Prevents Poor Performance

 · 19 mins read

TL;DR: Proper preparation prevents poor performance in all areas. This post looks at the 2020 Metasploit Community CTF and how failing to properly prepare guaranteed poor performance.

Background

Understanding how technology is used is important to understanding how to tell what it was used for. To dig into Apple Notes, I spent a lot of time using Apple Notes to understand the actions users normally took and how they appeared. To do better incident response, however, you need to know not just how users normally behave, but also how the adversaries normally behave. That should not be too surprising of a statement and hopefully you spend time considering how a distracted driver behaves, how someone commiting corporate espionage would likely hide their tracks, and how a malicious actor might probe your network. I enjoy capture the flag (CTF) events for just that reason.

This post is a brief write-up for the 2020 Metasploit Community CTF event. While many others will likely be walking through their successful solutions, I felt this year it would be more apropos to examine the biggest cause of my failure: lack of preparation. I want to examine the specific lessons I and others might want to take away to do better next year. The underlying principal is the same: proper planning prevents poor performance.

Assumptions

For this post, I’m assuming the reader is not a professional pen tester and is going the quick and easy route using Kali Linux. I am also assuming the reader desires to have some form of write-up after the fact and needs to know what they did. Finally, I am writing this in the context of the Metasploit CTFs, so while there are other good tools out there, I will be spending more time on the Metasploit Framework.

Your Hardware

When you participate in a CTF you’ll need to know how exactly you will connect to it. Are you using your daily driver computer and its usual OS? Are you using your daily driver with a LiveCD? Are you using a separate computer? More than one? These are all options which have positives and negatives, but you need to know the answer ahead of time, or you’ll have to make a decision without considering them. For security’s sake, it is generally not a good idea to use your normal computer in a CTF.

Lesson #1: Always Prepare Hardware And OS With Time To Test

If you have no other starting point, just set up a LiveCD with persistent storage. This lets you get in the game quickly, without messing up anything you actually need on your computer. If you have sufficient RAM on your daily driver, you might also consider VirtualBox. If you are so spoiled as to have a spare computer laying around, you might even install it. Whatever your choice, make sure you do this well before 1AM the day of the CTF to be better prepared than I was.

In the specific case of Metasploit’s Community CTF, they generally provide you with an SSH key to access a jump box already set up with Kali. Even if you have nothing else working, you could always use your favorite SSH client to connect.

Lesson #2: Always Be Flexible

However you assume you will be connecting to the CTF, you should assume that won’t work and have a backup. The backup might be another OS in case the rolling release of Kali decides not to install, or it might mean having two different Kali Live CDs on hand. I ran into issues where 2020.1 refused to install on the hardware I was planning on using (again, at 1AM the day of), but once I tried 2019.4 it worked out. I’d wished I had the LiveCD downloaded and ready.

Your Tools

Lesson #3: Find Your Word Lists

Directory listing of /usr/share/wordlists

No matter what CTF you are doing, at some point you are probably going to give up on strategy and simply try to brute force something (or maybe that is the best strategy). Whether that is guessing usernames and passwords, trying to find relevant files or folders on a webserver, or responding to a prompt on a random telnet port, you will want to know ahead of time what you are going to use. If you are using a default Kali installation and have no other starting point, you’ll want to look in /usr/share/wordlists/. Each folder in here is a symbolic link to the actual location of the specific tool’s wordlist, with one exception. rockyou.txt.gz is a large set of roughly 14 million potential passwords and is the usual fallback when nothing else works (not a starting point, because of the size).

If you are trying to brute force a user’s password, you probably want to check out some of the files under the /usr/share/wordlists/metasploit/ folder, especially /usr/share/wordlists/metasploit/common_roots.txt. It is really important to know the difference between a password list (one word per line) and a userpass list (two things, separated by a space on each line). The latter can be parsed by Metasploit to try each combination, whereas if you feed that to something wanting a password, it will try the entire line and fail.

If you need to look for words in a dictionary, those are kept in /usr/share/dict/. This was key in 2018’s 3 of Clubs, wherein you had to find the adjective that described the card and add “.png” to the end to find the image file. By pointing Dirbuster at /usr/share/dict/wordlist-probable.txt and letting it sit overnight, you’d come back to find that splendiferous.png was the 3 of Clubs.

If you need to chew through a website to find where a particular folder might be, you can look in /usr/share/wordlists/dirb/. It has some commone ones broken out into/usr/share/wordlists/dirb/small.txt and you can get progressively larger by looking at common.txt or big.txt. You will be more effective targeting the specific service you are looking at, but absent a starting point, use these.

Lesson #4: Set Up Metasploit’s Database

I can’t say enough how important it is not to just fire up msfconsole and start derping without making sure a database is connected. Rapid7 has the canonical step-by-step guide to do this properly and I would strongly recommend you follow it. If you start using msfconsole and you see “WARNING: No database support” you are not doing yourself any favors. This database is how Metasploit will keep track of a lot of the knowledge you gain (potentially offsetting other fails later on), such as which hosts have which ports and which vulnerabilities worked.

If you are running msfconsole for the first time, it should prompt you to create a database, but what if you see this message in the future? You likely rebooted, or stopped the database service, and if you can fix this by running msfdb init. Once you run that command, you should see “[+] Starting Database” and then when you run msfconsole you will again be connected to the database.

At the end of the CTF, do not lose this information. Before you tear everything down, run db_export [filename] to save off the database in case you want to review it elsewhere. If you wanted to import a previous year, you could do the reverse, with db_import [filename]. Importing my database from 2018 let me instantly get on ports 25 and 79 in 2020 and get the Two of Diamonds, given the user credentials were the same as 2018 and already in my database.

Lesson #5: Create a Metasploit Workspace

Since you shouldn’t keep all your CTF data in the same place, you’ll want to create a workspace to bundle all the information together. workspace -a [name] is the command to create a new workspace in msfconsole, so at the start of the 2020 CTF you might run workspace -a ctf_2020 to add a new workspace named “ctf_2020” and automatically switch to it. Until you switch workspaces (or restart msfconsole), you’ll be tracking everything you do in the ctf_2020 workspace. If you do restart msfconsole, you would use workspace ctf_2020 to check out that workspace, instead of the default.

Lesson #6: Improve Metasploit Logging

When writing up a CTF, I like to be able to review what I did. You can run the following in msfconsole to turn on more logging ensure you have a solid record of what you did:

set TimestampOutput true
set Prompt "%T %L"
set ConsoleLogging true
set SessionLogging true

You can automate this by saving those lines into your user’s .msf4 folder in a file named msfconsole.rc: ~/.msf4/msfconsole.rc. These logs will be found in the home directory of your user: ~/.msf4/logs/. Make sure you copy that entire directory out when you finish the CTF, especially if you are doing work from your jump box.

Lesson #7: Use A Metasploit Launch Script

Because I know I will inevitably forget to restart the metasploit database after a reboot, it can be really helpful to create a bash script to launch msfconsole after running everything else you need to. For a simple example, save this as ~/launch-msfconsole.sh and run it instead of msfconsole to restart the database if it stops:

msfdb init
msfconsole

Lesson #8: Put Everything Into a Standard Repo

Wordlists can be tailored to be more successful and Metasploit configs and launch scripts can be very helpful, but all have to be put together at the start if you use a new jump box. To get around that, you can take time before the CTF even begins to do all your prep and package it into a Git repo. That way, as soon as the CTF begins, you can check out the repo to your jump box and be ready with all the tools you might have wanted but didn’t have on hand. Do this today, be ready in a year.

Your Knowledge

Knowledge is power, so what do you have if you don’t keep track of your knowledge?

Lesson #9: Plan Your Folder Structure

As soon as you know which device you will be working from, you need to set up the right folder structure to enable success. I use something like the following:

~/
  |-> msf_ctfs
      |-> 2018
      |-> 2020 [one for each year you have data]
          |-> after_action.odt [larger write up, see below]
          |-> targets
              |-> 172.16.3.85 [one for each IP in scope]
                  |-> 80 [one for each port you'll be looking at]
                      |-> notes.txt [basic notes on this service as you find them]

The above directory structure can just as easily be imagined as Wiki pages, with each year having its targets listed on it and each target having the ports listed on it. The downside of a Wiki is you can’t just cat the output of a command onto it. The upside is if you are working on a team and it is exposed to the entire team, it allows for much better rich-text editing between many people than trying to share that LibreOffice document.

Lesson #10: Document From The Start

There are many ways you can document your findings, the key things you’ll want to consider is whether it is shareable between your team (if you are on one) and how structured/searchable it is. As I generally don’t participate in a team, I use LibreOffice’s Writer application to document each host within scope and the services running on each port. For Metasploit, as long as you’ve set things up correctly (see above), you’ll also be able to make use of its automatic documentation of the hosts, services, users, and vulnerabilities you have seen. In my case, as I started throwing my randomly scattered notes into one LibreOffice document late in the day on Saturday, I realized I had 3 or 4 ports I had scanned initially, but never returned to. These likely would have been much more promising leads earlier in the week.

Lesson #11: Know The Network

Both the 2018 and 2020 Metasploit Community CTFs have made good use of Docker to show one host but have 4 or 5 actually be represented. As you scan up the ports on each device, you have to be putting together a picture of that network, even if it is completely within the one host. I’d recommend draw.io for this task. I’d also recommend starting as soon as you get the information of what your jump box is and what is within scope of the CTF so you don’t accidentally send rounds in the wrong direction. If past CTFs are an indicator, you’ll need to figure out which ports go to the same docker image to fully exploit it. In 2020, for example, ports 25 and 79 were both pointing to the same ancient BSD image as was used in 2018, whereas ports 564 and 5900 pointed to a Plan 9 device. Below is from 2018 because I did not follow my own advice in 2020.

2018 Network Diagram

Lesson #12: Be Ready To Research

What I’ve loved about the Metasploit CTF is that it is not there to test your ability to use the search function on msfconsole. From the ancient BSD version, to Plan 9, to giving us RISC-V assembly, Rapid 7 expects the player to be challenged and not have all the answers. There really isn’t a specific tip here other than to be ready to look up recent talks at security conferences and go in with an open mind and Duck Duck Go bookmarked. Definitely keep tabs on what challenges were in the previous iteration (read the write-ups) and assume there might be some variations on those themes.

Your Execution

Lesson #13: Prepare For Success

Go into the challenges assuming you will have some success and consider “What next?” If you are able to get a shell in msfconsole and can’t quite seem to copy that flag out of the environment, make sure you know the “help” command. It’ll tell you, helpfully, that there is also a “download” command. Depending on the shell, you’ll see lots of other commands that might also be relevant.

Let’s also assume you gain access and want to return a bit easier than you did before, you should consider an SSH key. Creating a new one for each CTF ensures you are not using something that may give others access to what you set up, or leaving your own key laying around. After you create a new SSH key, you can add it to the devices you access by adding the value of the public key (for example, id_rsa.pub) to the appropriate user’s ~/.ssh/authorized_keys file. If the device is running SSH and you can see the appropriate port, this is a very useful technique and you should create the key well before the CTF begins.

Lesson #14: Perform Full Recon

As soon as you can, kick off all of your recon tools so that you have the data needed to make smart decisions about where to spend your time. You can have nmap produce XML output that Metasploit will nicely ingest using db_import, if you use the -oX switch: nmap -A [target_ip] -p0-65535 -oX [target_ip].nmap.xml. Yes, this will potentially take some time. You may want to also kick off some simpler scans to look for obvious starting points while the larger ones complete, such as nmap [target_ip] -oX [target_ip].simple_nmap.xml However, having a full picture of what you are up against will let you play to your strengths.

In this case, as soon as I saw that both port 25 and port 79 were the same services as last year and that it was my favorite challenge from last year, I jumped into the Two of Diamonds again and was able to capture that flag fairly quickly.

Lesson #15: Brute Force Early

This may seem obvious, but in my flail I didn’t start looking at the challenges I expected to be brute forced until way late in the CTF. If you believe you’ll need to brute force something, similar to the 3 of Clubs in 2018, get that set up and start it early. I like to identify those challanges, plan for how I would solve them, then kick off those solutions while I sleep each night. You’ll want to review the wordlist section above to figure out which batch of data you’ll be trying.

Lesson #16: Understand SSH Tunnels

You will almost certainly need to use SSH tunnels during the CTF. For Metasploit CTFs, you will likely be connecting from your computer through the jump box on Amazon to the target network. That means you either will use your computer to probe the network (needing two tunnels, one to connect through the jump box to the target service and one for the shell to call back to through the jump box), or use the jump box to probe the network directly (only needing tunnels should you want to, for example, look at websites hosted in the target network on your computer).

The command you want to run for a forward tunnel is: ssh -L [local port]:[target IP]:[target port] user@[tunnel host]. As long as you leave the window open, your tunnel will be up. For example, taking the network diagram from 2018 above, if I wanted to view the webserver running on port 8080 from my computer, tunnelled through the jump box, the command would be (note this is using a key to authenticate, vice password): ssh -i [ssh_key] -L 8081:172.16.1.213:8080 ec2-user@18.213.3.xx. Then to connect to the website, I would use a browser to open 127.0.0.1:8081. Note: 8081 was chosen as the local port simply to show where each different port went in the command, this could easily be 8080 as well.

While I did most of my network probing from the jump box directly, because so many of this year’s challenges had a web component, I ended up with a lot of tunnels. My second desktop window was just a ton of terminal tabs, each one with one specific port open. That way I didn’t have to remove one to set up another as I moved between challenges, I could just leave them all up.

Checklist For Next Year

Prior to Start

  1. Bin up all my usual tools into a git repo
    1. msfconsole.rc
    2. launch-msfconsole.sh
  2. Determine which computer I’m using to connect
  3. Install an appropriate OS and make sure it works head of time (bleeding edge version may not be smart)
    1. Make a backup LiveCD with persistent storage in case that dies
  4. Set up tools as necessary
    1. Check out git repo
    2. gunzip /usr/share/wordlists/rockyou.txt.gz
  5. Create folder structure
  6. Copy over previous year CTF folders for reference
  7. Create SSH key to authenticate, if needed

Upon Start

  1. Connect to jump box
  2. Set up msfconsole (if you aren’t just tunneling through)
  3. Set up tunnels (as needed)
  4. Make sure msfdb is running
  5. Kick off quick nmap scans against target range
  6. Kick off full nmap scans against target host(s)
  7. Ingest results into msfconsole (or you can run them directly in msfconsole if you don’t want to keep the output in flat file)
  8. Eyeball which challenges seem most interesting
  9. Get after it, updating documentation as relevant

At the End

  1. Copy data from jump box
    1. Any cards, screenshots, etc
    2. ~/.msf4/loot/*
    3. The SSH key you created (if necessary)
    4. ~/.john/* (you’ll find a lot of tools have relevant logs to look for)
  2. Copy logs from jump box
    1. ~/.bash_history
    2. ~/.msf4/history
    3. ~/.msf4/logs/*.log
    4. ~/.msf4/logs/sessions/*
    5. ~/.msf4/history
  3. Export msfconsole database and copy off
    1. db_export ~/ctf_[year].db

Conclusion

I did better this year than in the 2018 Community CTF, which was my main goal. I also learned a lot, which was my secondary goal. However, the vast majority of what I learned was simply how to set myself up for success, or rather how not to set myself up for failure, next time. I hope these lessons are valuable to you as well and that next time you won’t fail to plan your hardware, software, tools, and knowledge.