Researching Automated Pentesting with AI Agents

 (source)

It sounds great right? Ask an AI Agent to pentest your website, and ask another AI Agent to go fix the issues the first one found.

My Early Experimentation 

I was curious about the current capabilities of AI Agents, so I setup an intentionally vulnerable API from OWASP called "crAPI" or completely ridiculous API on my local network. To avoid incurring high API costs, I preferred to use local models with Ollama, running on my gaming PC which should be more powerful than my laptop. I started out with the llama3:8b model because it was recommended and wasn't so big that my PC couldn't handle it.

There are many AI Agent frameworks built for this purpose, but so far I've spend the most time with strix. I prompted it to perform a penetration test on the crAPI, but I quickly ran into an "ethical guardrail loop", here is an example:

 

A Story About Password Auditing

Hi everyone, sorry I haven't blogged in awhile. Life gets busy and with the recent boom of generative AI I wasn't sure about sharing original content like this that will inevitably be scraped and used to train models. Anyway, on to the story.

I enjoy learning from platforms like TryHackMe and HackTheBox. One technique I learned was how to obtain Windows password hashes and crack them using rainbow tables and Hashcat permutation rules. I thought this was really cool and it got me curious, how good or bad were the passwords in production? I got permission from key stakeholders, grabbed the password hashes, and ran Hashcat to crack them. Immediately my screen was flooded with hundreds of very weak very easily guessable passwords. I was shocked, was this real? I tested a couple and confirmed they were indeed real passwords.

 

 

How was this possible? We had a password policy with complexity and minimum length requirements. I began digging deeper and discovered that it was easy to meet the complexity requirements with a weak password. If you have four minutes to spare I recommend this very relevant bit by comedian Michael McIntyre

 

 

Let's take a closer look at a fairly common password policy that:

1. Requires an uppercase letter
2. Requires a number
3. Requires a special character
4. Requires a minimum length of eight characters 
5. Must be changed every 90 days
   

 

Password123!

Welcome2020!

Ihatepasswords1!

 

These are all acceptable passwords according to the password policy. Even worse, with rule number five forcing passwords to be changed on a regular basis, I saw that people were lazy and will simply increment a digit so "Ihatepasswords1!" becomes "Ihatepasswords2!" or "Welcome2020!" becomes "Welcome2021!".

 

I felt a sense of regret having looked under this rock to discover such a huge problem, but I soon started working on a solution. I started by improving the password policy. If you read up on NIST Special Publication 800-63B it says some things like:

1. Elimination of complexity requirements: NIST advises against enforcing arbitrary complexity rules (e.g., requiring a mix of upper-case letters, symbols, and numbers). Instead, passwords should be checked against known compromised credentials and prohibited password lists.
2. Elimination of regular password resets: NIST recommends not forcing users to change their passwords on a regular basis unless there is evidence of compromise. Regular resets often lead to weaker password choices due to user frustration.
3. Preference for longer passwords: NIST encourages using longer passwords or passphrases, as they provide better security than shorter, complex passwords. Longer passwords are harder to crack and easier for users to remember.
   

So here's my plan:

1. Remove the complexity requirements
2. Increase the minimum length requirement and educate users about passphrases
   
3. Remove the requirement to force a password change on a regular interval, it should only be necessary when an external event requires it
4. Create a custom list of banned keywords that are industry, brand, and country specific
   
5. Install a password filter to check the password being set against a list of known compromised passwords (something like 20 billion and growing with every breach made public)
   
6. Setup a password auditing tool to run on a weekly basis. This is important because the list of "known breached passwords" is always growing and it gives us an opportunity to see if a password is in use that was recently added to a breach list. The filter will only check it at the time the password is set.
   

 

I am happy to say that after some time, the project was a success. It took a lot of time and effort. I had to coordinate with many disparate teams, learn new technologies, educate thousands of non-technical people, implement new standards, force a lot of password resets (and probably upset a few people), participate in dozens of meetings, write documentation, coordinate schedules, and work with vendors. What started as a curiosity took me down a road I was not prepared for, but am glad that I did because I learned a lot and felt like I made a significant contribution to improving security.

An Introduction To External Attack Surface Management

In the past year or two I have been learning a lot about External Attack Surface Management and I wanted to share some of my knowledge. As a short introduction, what is an Attack Surface? Imagine you lived in a house with lots of valuables and you were in a bad neighborhood. You would probably lock your doors and windows at all times, and have additional layers of security like a barbed wire fence and an alarm system. In this scenario, the attack surface would be any point of entry a burglar could use to gain access to your valuables. A clever burglar would find a weakness in the fence, and test all of the doors and windows to find the easiest way inside.

To verify that you are secure, you could test your defenses by trying to break in yourself, or hire a professional to do it for you on a regular basis. What if you had thousands of windows and doors opening and closing regularly, and you were in a neighborhood where all of your neighbors are burglars? This is closer to the reality faced in the internet world.

A house with many doors. Source

 

An insecure computer plugged directly into the internet with no firewall can be compromised in seconds. So how do you monitor your attack surface? The first step is to inventory everything that is internet facing. Find every IP Address, Root Domain, and Subdomain you own. If you have access to your DNS registrar you can obtain a source of truth list of every configured DNS record. If you don't, you can use the same techniques used by Bug Bounty hunters and use subdomain enumeration tools like Amass, subfinder, and massdns. This inventory of assets will likely change over time, so it would be a good idea to run automation that updates the inventory as new assets are deployed and old ones are decommissioned.

The second step is to do a use your inventory list and run a simple port scan with a tool like nmap, masscan, or naabu to see what ports and services are open to the public internet. For the best results this port scan should be done from a dedicated server. This is like testing to see if your doors and windows are unlocked and what's behind them. The port scan should return some results with fingerprint data that gives you an idea what may be behind each open port.

If it's an SSH, FTP, or Telnet service, try to connect with a client to see if it lets you try to login with a username and password. You can also often see the version of the service running and do an internet search for known vulnerabilities.

If it's a web server, you can try to load it in your browser, or use a tool like httpx to see what information is available. Try to connect using HTTP and HTTPS, and if you can connect using HTTPS take a look at the SSL Certificate to learn more about the service. There are some great command line tools for parsing SSL Certificate data such as SSL Checker and some more mentioned by Jason Haddix in his recent Tool Time video discussing SSL Certificate Parsers. Explore the web server in depth to gain an understanding of what it's used for. Review the HTML and Javascript source code. Does it have any login forms where a username and password could be used? Does it reveal any information that shouldn't be public? Does it have any known vulnerabilities that could be exploited? Can you do Content Discovery to find hidden content?

For other open ports you will need to do some research to better understand what service is listening and what risk that could present by remaining publicly exposed.

As a general rule of thumb, you want to reduce your attack surface to improve your security posture, or in other words, the less doors and windows you have to lock, the less likely you are to lose your valuables. When you find an open port and service, you should ask yourself if you really want that to remain public, or if it should be hidden from public view. If you want to hide it from public view you will need to identify where it is hosted, and change the firewall settings to only allow access from specific management IPs, or close the port entirely. If it's an old and unmaintained asset, consider decommissioning it to save money and reduce your attack surface.

If you discover new assets are regularly deployed in an insecure state, identify who is deploying those assets and get to know more about their needs and processes. They may not know that the new servers they deployed all had a web server with an admin login page exposed on port 8000. Work with them to improve their processes so they are less likely to expose a risk to the environment in the future.

The steps here can all be automated, and depending on your budget, you can build a solution yourself or you can pay a third party to do it for you. Here is a nice list of free and premium solutions.

This is a high level introduction to the world of External Attack Surface Management, or as it is also known, simply "recon". I hope someone has found this information useful, have a great weekend everyone!

Bug Bounty: Behind The Scenes

As a Bug Bounty hunter it can be frustrating to submit a report and have to wait for a long time to get a response, have your report closed as duplicate/out of scope, or receive a reply from the triage team requesting a better explanation of the impact and to improve your PoC (proof of concept). Early in my career I was a Bug Bounty Triager and I wanted to write this to provide some insights on what happens behind the scenes when a Bug Bounty report is submitted.

Before a Bug Bounty program is created, many things need to be in place. The security team has to:

• Be very familiar with what public facing assets the company has
• Be very familiar with the development and infrastructure teams
• Know who to contact depending on what issues are reported
• Have the skills to understand the severity and verify the report is valid
• Have the customer service skills to interface with the hacker in a friendly manner
• Have support from senior leadership

Without these fundamentals, a Bug Bounty program is unlikely to be successful. To put these in another perspective, when I received a new report, these are the questions I would ask myself:

• Is the reported issue in scope for the program?
  
• Can I validate the issue is accurate with the provided PoC?
• What is the impact to the business if this issue were exploited by a Black Hat Hacker?
  
• Who do I contact internally to get the issue fixed?
• How does that team like new tasks to be added to their backlog?
• How do I explain and demonstrate the risk so they understand it?
  
• How will they prioritize the issue with their other tasks?
• Are there compensating controls that can be used to mitigate the risk?
• Could this issue exist in other areas?
  

Assuming the report is valid and in scope, I would normally create a ticket for another team to make a change and fix the issue. Depending on the workload of that team and priority of their other tasks, it could take them weeks or months to complete the fix. During this time I would often receive new messages from the Bug Hunter asking for an update, and I would do my best to be friendly and provide realistic expectations. Every Bug Hunter was unique, and some were understanding, while others would get frustrated and threaten to make the issue public if they weren't paid in a timely manner. This is part of the reason why many Bug Bounty programs are invite-only and curated so only respectable Bug Hunters participate in the program.

It wasn't uncommon for service owners to contact the security team and ask about unusual activity. Sometimes the scanners that Bug Hunters use can negatively impact services. If we were able to identify which Bug Hunter is causing the negative impact we would contact them and ask them to rate limit their scans, or simply ban their IP to protect the service. When reviewing logs it is not always easy to tell the difference between a Bug Hunter and a malicious hacker. This is why sometimes programs request that a unique user-agent is used, so it is easy to attribute scan activity with a specific Bug Hunter.

When the report involves a Remote Code Execution vulnerability, or the PII of customers/employees, it is typically treated as a P1/Critical issue. This is when the security team kicks into high gear and performs Incident Response. To me, conducting Incident Response was like being a "Digital Firefighter". Adrenaline kicks in and you have to try to think clearly and contain the issue as quickly as possible. Leadership must be informed, on-call people sometimes need to be paged, logs need to be reviewed, a timeline must be created, and other priorities take a back seat until the issue is contained. In this context, containment means that the risk is mitigated and can no longer be exploited.

Incident Response: Digital Firefighting

In June of 2021 Lesley Carhart posted an excellent photo of her heart stress during an incident:

Source: https://twitter.com/hacks4pancakes/status/1407165004424507401

Once an issue is fixed, I would message the Bug Hunter to ask them to verify the issue is fixed. Once everyone agrees it is resolved we would reward the Bug Hunter. The reward amount was often a discussion point within the team and many factors were considered. Primarily: "What is the impact to the business if this issue were exploited by a Black Hat Hacker?".

The work I did as a Bug Bounty Triager early in my career taught me a lot about Application Security, Penetration Testing, Incident Response, and Vulnerability Management. It inspired me to dive deeper into these topics and eventually learn how to do some Bug Hunting of my own. In May of 2021 I received my first paid bounty for two findings I submitted through HackerOne.

I am happy to see that Bug Bounty programs are being created by so many companies out there. It is a great opportunity for companies to improve their security, and for ethical hackers to get paid for doing what they love.

If you are interested in becoming a Bug Hunter, or just like to learn how hackers do their thing, I highly recommend these learning resources:

HackerOne CTF https://ctf.hacker101.com/

PortSwigger Web Security Academy https://portswigger.net/web-security

HackTheBox https://www.hackthebox.com/

TryHackMe https://tryhackme.com/

PentesterLab https://www.pentesterlab.com/

Many of these learning sites have built-in guides or you can find high quality guidance on Youtube

Advice For My Younger Self

Looking back at the beginning of my career in InfoSec, I made a lot of dumb mistakes. I would ask questions that could easily be Googled, I would skip over the fundamentals and go right to the advanced stuff, and I rarely read the manual (aka "RTFM"). Sometimes I find myself doing these things now, but at least I recognize it and work to improve.

Here is some advice I would give to my younger self that I hope someone finds useful:

Computers, networks, and software is complex and no one knows everything. Don't treat your senior team members as if they have all of the answers. When you encounter a problem, spend some time trying to solve it on your own. When that fails, read the manual, and when that fails, Google it. You might be surprised to learn that senior professionals Google everything.

Remember the human. Everyone makes mistakes, everyone has bad days, and everyone forgets things. Don't treat people like robots that should be perfect. This also applies to yourself, leave room for failure and learning.

It's not you versus me, it's you and me versus the problem. It's easy to encounter a problem and present it to someone else to solve, but it is more important that you treat people as if they are on the same team and collaborate on a solution.

Some conversations should not be done through text (email or instant message). It is possible that someone reads your email in a different tone of voice than you intended. When possible, converse through a face to face medium and use email to share supporting information.

As you learn things, take notes as if you will forget everything tomorrow. I like to write documentation because it helps me confirm what I've learned and share it with colleagues, as well as my future self when I forget some of the details.

Don't let perfect be the enemy of good enough for now. It's easy to get stuck trying to perfect something, when sometimes all you need is something that gets the job done for now. It is easier to improve something that already exists.

Don't be overwhelmed by how much you have to learn, take it day by day and follow what interests you. This image is just a sample of the subjects within Cybersecurity, each one can be a specialty on it's own. It is important to keep learning and challenging yourself, but you don't have to know everything. Once you develop a specialty it is relatively easy to move between others. 

Security has many specialties

Learn and respect the fundamentals. On the technical side of Cybersecurity you should know some basics about networking, systems, and coding. You don't have to be an expert at all three, or even know how to code, but the more you learn about these the better off you will be in your career.

Remember "CIA": Confidentiality, Integrity, and Availability. This is another fundamental concept that is at the core of all Cybersecurity careers. Our job is to understand and protect all three. Sometimes that can be as easy as verifying the data is backed up, or that an encrypted protocol is being used for transport.

Try, fail, and repeat until successful. Failure is part of the learning process, and I remember giving up after failing once. Get back on that horse as many times as it takes!

Certifications can be helpful to give you some structured learning, but they alone will not make you successful. You can use certs to confirm your knowledge, improve your resume, or to help you pivot into a new specialty. Everyone learns different and sometimes it's better to learn on your own.

Immerse yourself in the hacker culture and connect with the community. Listen to Darknet Diaries, read /r/netsec, watch Hackers, follow security professionals on Twitter, Setup a Feedly account to monitor RSS feeds for blogs and news, read popular books (fiction and non-fiction), go to meetups, join a discord, try HackTheBox, and share your code or ideas to give back to the community.

Understand the difference between knowledge and experience. It takes time to obtain experience, you cannot rush it. Enjoy the journey!

I think that's enough for now, I'll leave you with some advice from Professor Feynman:

• Study hard.

• What others think of you is none of your business.

• It's OK not to have all the answers.

• Experiment, Fail, Learn and Repeat.

• Knowledge comes from experience.

• Imagination is important.

• Do what interests you the most.

• Stay curious.

Recent Accomplishments

Not long after my last blog post I was hired as a professional penetration tester for the first time in my IT Security career. This had been a dream of mine for some time, so I was full of excitement and motivation to learn everything I could. I completed the eLearnSecurity Junior Penetration Tester (EJPT) certification in May and immediately started working on my next cert, the Certified Red Team Professional (CRTP) from PentesterAcademy. I quickly learned that although the description says it is for beginners, it actually felt quite advanced for me in the beginning. I even took a Udemy course called Advanced Scripting & Tool Making using Windows PowerShell to improve my comfort level with PowerShell. The CRTP course focuses on attacking and defending Windows Active Directory. Some of the things you will learn as a student:

• Active Directory Architecture
• Kerberos
  
• Active Directory Enumeration
• Local Privilege Escalation
• Domain Privilege Escalation
• Domain Persistence and Dominance
• Cross Trust Attacks
• Forest Persistence and Dominance
• Defensive Monitoring
• Bypassing Defenses
• Deception Techniques
• PowerShell
• Mimikatz
• BloodHound
• PowerView
• Custom Obfuscation
  

Just yesterday I submitted my final exam report, and hours later I received confirmation that I passed! For me this is the result of four months of effort and hundreds of hours of practice. In addition to the CRTP I also completed these Windows Active Directory challenges:

Practice Pentest Report - HackTheBox - Postman

I have been studying for my OSCP certification recently and purchased VIP access to HackTheBox.eu. This site is an excellent resource for penetration testers of all levels. For those that aren't aware, HackTheBox is a penetration testing lab with live machines to practice your hacking skills against. In the beginning you are only given an IP address and have to figure out how to gain access to the "flags" which you can then submit to the leaderboard for points. So far I have only completed a couple of the "easy" rated boxes, such as one called "Postman". After completing Postman I took the opportunity to practice writing my first "penetration testing report" for a fake company. Normally after a penetration tester is hired, they deliver a report to the business that hired them to review the discovered issues and recommended follow-up actions. Here is my Postman "Pentest Report":

Introduction

A “penetration test” was requested by “HackTheBox” for their soon-to-be-deployed “Postman” service. The goal of this test is to verify that security is up to par with their expectations before being released to production. The test will be done in black box format, without access to any code or prior knowledge of the system.

Scope and Duration

The scope of this test is limited to the “Postman” server located at 10.10.10.160. The duration for the test is one week starting from the 18th of November 2019. The client has requested we avoid using any "DoS" exploits to avoid unnecessary crashes. If root level access is obtained, the testing can be stopped immediately and the test results reported to the client for remediation.

Technical Summary

Initial port scanning revealed several exposed services, such as HTTP, SSH, Redis, and Webmin.

Initial Port Scan Results
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Discovered open port 6379/tcp on 10.10.10.160
Discovered open port 80/tcp on 10.10.10.160
Discovered open port 10000/udp on 10.10.10.160
Discovered open port 10000/tcp on 10.10.10.160
Discovered open port 22/tcp on 10.10.10.160
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Port Scan Results With Banners
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
22/tcp    open  ssh     OpenSSH 7.6p1 Ubuntu 4ubuntu0.3 (Ubuntu Linux; protocol 2.0)
|_banner: SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
80/tcp    open  http    Apache httpd 2.4.29 ((Ubuntu))
|_http-server-header: Apache/2.4.29 (Ubuntu)
6379/tcp  open  redis   Redis key-value store 4.0.9
10000/tcp open  http    MiniServ 1.910 (Webmin httpd)
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Particularly interesting is the Redis service, which is an in-memory database application. Internet research reveals a known security flaw detailed by the developer here: http://antirez.com/news/96. While attempting to reproduce this we discovered that command line access to redis was easily obtained and commands can be executed.

We continue to follow the developers instructions and create a text file containing our public SSH key with newline padding. Once that is created we need to:
1. Change the dbfilename to authorized_keys
2. Change the config directory to .ssh
3. Insert the contents of our text file into a “set” within the database
   1) "Redis Sets are an unordered collection of unique strings. Unique means sets does not allow repetition of data in a key."
5. Save
6. Test to see if we can SSH in

We can see that this attack was successful, and we now have a shell on the system as the redis user. This user only has limited access:

While searching the system for files containing sensitive information, we discovered a file called “id_rsa.bak” in the /opt/ directory. The owner is shown as the “Matt” user and the contents reveal it is a private key.

Internet research reveals an article ( https://medium.com/@canarerobertjohn/basic-pentesting-2-325cafc19fcc ) with instructions for cracking a private key such as this to reveal the passphrase. The steps are as follows:

1. Convert the key to a format supported by JohnTheRipper
   1) "John the Ripper is a fast password cracker, currently available for many flavors of Unix, macOS, Windows, DOS, BeOS, and OpenVMS"
3. Run JohnTheRipper on the converted key using a wordlist

Sorry for the small text...

Within seconds we have revealed the passphrase, and using this information we want to see if it's possible to escalate our privileges to the Matt user.

The privilege escalation was successful, we are now logged in as Matt and have more access, but not to everything.

Additional testing reveals that the credentials for “Matt” also work for the webmin panel:

Earlier banner information revealed Webmin as being version 1.910, and searching for exploits reveals one:

Now we load up MetaSploit with this exploit and configure all of the options:

After running the exploit we have successfully obtained root level access to the system.

Remediation Recommendation

This server was able to be fully compromised due to a series of weaknesses that were easily exploited. We recommend:

1. Closing ports to services that do not need to be publicly exposed
2. Removing banner information that can reveal software versions
3. Training your users not to leave sensitive information on the server in a directory readable to all users
4. Enforcing stronger password requirements
5. Patching all software to the latest version

Improving Security With Kindness

A couple years ago I was on a flight next to an older woman who had never flown before. She had no idea how to use the tablet in front of her, or even how to buckle her seat belt. She asked me for help and I think at this point some people would sigh and dread sitting next to her for the rest of the flight. However I saw this as an opportunity to help someone who was technically illiterate, and probably nervous about her first flight. After I helped her buckle up I showed her how to plug her headphones into the tablet and start a movie, adjust the volume, play a game, or some music if she wanted. She was a sweet grandma and told me all about her grandkids and her life in small town America. Throughout the flight she would ask me questions and I took my time to explain everything as simply as possible. After the flight while waiting in line one of the passengers who sat behind me tapped me on the shoulder and asked how I could deal with someone like that, and said I have incredible patience. I said it was no problem and that I was just happy to help.

The way I see it, I am fortunate to have grown up around computers and be very comfortable with technology. However, not everyone is as fortunate or naturally inclined to learn how to use tech. I think it's even safe to say there are a lot of people out there that are confused or even afraid of tech. These are the people who still cling to old software, POST IN ALL CAPS, and always seem to get viruses. This is where I say that it is our job, as the technically literate, to exercise our patience and kindness to help someone who is struggling with something we figured out years ago.

Early in my IT career I worked at a Helpdesk where we were each assigned trouble tickets to work on. These tickets were often things like installing a printer, migrating someone's email to a new device, or replacing a broken hard drive. My colleagues would often cherry pick the tickets from users who were easy to work with, and avoid the "problem users". I had no problem taking on the "problem users" tickets, seeing them as opportunities to help someone in need. Many of these users were experts in their fields, and very well educated, but when it came to technology they were frustrated, confused, and angry. I often thought to myself that those are normal emotions felt by all of us when learning something new, or when something "just doesn't work", and that these same people would gladly help me if I were frustrated with something they were experts at, so why don't I share the same courtesy?

Free Cookies!
(Image Credit: Wikimedia)

I would approach each of these users with some friendly small talk, try to get the backstory on what's going wrong, empathize with their frustration, and start working on solving the problem while doing my best to explain what's going on. I would often recommend they take notes to reference later when they run into the same problem again, and come up with small tricks like using a Eudora skin on Thunderbird to make them more comfortable with a new interface. Over time these users encountered fewer problems, submitted less tickets, and some even developed a new appreciation for technology and started learning more on their own. At the end of the day this meant less work for my team, much happier users, and sometimes free cookies for me :).

More recently in my career I was building a Python script that had to be used by some of my technical and non-technical colleagues. While testing my first version of the script with my technical colleagues they told me how great it was and thanked me for building it. However when testing the same script with my non-technical colleagues, they expressed concerns about the difficulty of use and risk of making a big mistake. In my original design I had not considered this perspective, but it made sense when brought to my attention. I improved the script to make it more simple to use, added confirmation prompts where necessary, and expanded the documentation. I went back to my non-technical colleague and had them test it, they were thrilled at how easy it was to use, how hard it was to make a mistake with the confirmation prompts, and appreciated having documentation to reference.

Anyone working in IT Security can tell you that the weakest link is always the human. Phishing emails work because they trick people into installing malware. Hackers breach databases because people make mistakes in configuring the systems that are supposed to protect it. People use old vulnerable software because it's what they are used to. When these things happen we need to:

1. Avoid shaming, that doesn't help the situation
2. Be sympathetic, everyone makes mistakes
3. Be patient, not everyone is on your level of comfort, knowledge, and experience
4. Work with them to understand the problem from their perspective
5. Educate without condescending
6. Support

I honestly believe that if these steps are taken, hackers will have a harder time doing bad things, and the internet will be a safer place for everyone.

What is a SOC?

In a recent conversation I said that I have SOC experience, and then I had to explain what a SOC is. So what is a SOC and what do they do?

In many large companies a SOC ("Security Operations Center") is a team of people that are like firefighters for security threats. Their job is to detect and respond to incidents (aka "Incident Response"), and ideally to contain or mitigate the threat as quickly as possible. Detection of threats can be done through a SIEM (Security Information and Event Management), IDS (Intrusion Detection System), or simply via word of mouth. It is important that the SOC is easy to contact in the event that someone notices something strange they can report it to the SOC for a closer look.

It is a common misunderstanding that a SOC is able to prevent a company from being hacked. In reality no one is hack proof, and any attacker with enough time, energy, motivation, and money can succeed. The job of a SOC and the Information Security team is to reduce the company's risk level and to "raise the bar" for attackers so that a compromise is less likely.

How do they raise the bar for attackers? They build defenses in layers so that if any one of them fails, another is there to back it up. For example, phishing is a common method of attack hackers use every day. Because of this, user awareness training is vital to the security of any organization. In the event that a phishing email succeeds in compromising a computer, ideally that computer will have an isolated network so that the hacker cannot use that computer as a launching point to jump to other computers.

When a SOC analyst isn't reacting to incidents, they can be proactive by doing things like verifying systems are being patched regularly, running internal and external vulnerability scans to identify and resolve issues before attackers can find them, building and revising a threat model for the companies most sensitive assets, and constantly improving the detection tools they rely on.

Another key piece is Threat Intelligence (aka "intel"). Intel is a report of Tactics Techniques and Procedures ("TTPs") used by attackers that have been known to target your industry or even your company specifically. If you don't know what kind of attackers are targeting you, and how they do it, you can't effectively mitigate that threat.

Me (right) working in a SOC

What does a SOC look like? The image here features me and my colleagues working in a SOC. The screens on the wall feature many different dashboards that provide useful metrics on everything we deemed important. For example we had a dashboard that showed us which user accounts had the most failed login attempts. This allowed us to see what might be a brute force attack, but was often just someone's script using an old password. Another dashboard would show us recent alerts from our IDS and AV systems. so we could easily identify and follow up on potential issues.

The qualities of a good team

We're all part of a team in some capacity, and we all depend on each other for success. In my professional experience the better the team, the more successful the outcome of that team, and the happier the individuals on that team are. A good team can be hard to come by, and in my opinion should consist of these qualities:

• Teammates should feel like friends that hang out during work hours, friendly banter is common and sometimes they even become friends outside of work.
• They support each other in their goals. Example: You want to be better at "X"? Cool I can help with that, have you read these resources or tried these things?
• Knowledge is shared freely and without judgement. This way everyone can benefit from each other's strengths and weaknesses, and the "fear of asking a stupid question" isn't an issue.
• Honest feedback is shared between everyone (openly or in private depending on the content). This is essential to improving yourself and each other. Being the recipient of feedback shouldn't make you defensive, take it as an opportunity to improve and thank them for the feedback.
• Have each other's backs. If you hear someone complaining about a colleague, maybe you could suggest they share that feedback with them directly? How can they improve if they aren't aware of any issues?
• Trust should be default. This means assuming the best until confirmed otherwise.
• They openly collaborate and don't just work in silos.
• Conflicts are resolved in a professional manner.
• From the perspective of outsiders, the team should "speak with one voice". None of this "but I talked to Joe and he said something else".

For me many of these qualities define a good or bad team, manager, or colleague. As you read this, do you think you are part of a good team?

Hackers: Who are they?

Hollywood Hacking
(image credit: inkmedia.eu)

In a recent conversation with my mom she was concerned that I had recently gone to a hacker meetup. She thought all hackers were criminals, and I don't blame her for thinking that after reading about them in the news and seeing them in movies. It's usually a guy in a hoodie, typing fast, with complicated text scrolling by (check out hackertyper!). He has an aura of mystery and probably some malicious intent. I went on to explain the white hat / black hat concept. For those of you that aren't aware, in old cowboy movies the bad guy often wore a black hat, and the good guy wore a white hat. These are terms that we've adopted to distinguish between the good and bad hackers of today's world.

The tools and techniques used by white hats and black hats are the same, the big difference is in the intent. I wanted to review a couple different types of hackers and the things that motivate them:

Penetration Tester - A white hat hacker with good intentions that works to identify flaws and get them fixed. Sometimes they are part of an in-house security team, or a third party hired to test software or networks. Pen tests are essential to any modern security program, because I guarantee that black hats are looking for vulnerabilities to exploit, so you should hope a white hat finds them first. Their actions are limited by the scope of a test, and the time the testing is scheduled for.

Bug Bounty Researcher - A white hat hacker, usually with good intentions, that also works to identify flaws and report them to companies in exchange for money, swag, or "kudos" (street cred). These are often individuals with a pen testing background that do it on the side for extra income, or if they are really good they can do it full time. Some consider it fun and like to practice their hacking skills on real systems where permission is given via a bug bounty program's scope and rules (Example: Google's bug bounty rules). Some of them are not seen in the best light, see "Grey Hat" below.

Red Team - A term that is used to describe a team of white hat hackers, usually well intentioned, with the goal of simulating a real attack. They use Tactics, Techniques, and Procedures (TTPs) similar to those used by real world attackers. Some of these techniques could include making phone calls to social engineer someone into giving up useful information, wearing janitors clothes to sneak into a building to insert an infected USB drive, or launching an attack on a holiday when the defenders aren't around.

Blue Team - A term used to describe the defensive white hat hackers working as Incident Responders in a Security Operations Center (SOC). These are the white knights of the security world, doing everything they can to secure the environment they protect, although they are at a significant disadvantage having to secure potentially thousands of computers at all times, where an attacker only has to find one entry point. They will often use a Security Information and Event Management (SIEM) to help them monitor the health of their network, computers, and overall infrastructure. This could mean monitoring for failed login attempts that indicate a brute force attempt, looking for port scans that might indicate the first stages of an attack, or just reviewing recent malware alerts for uncontained threats. Additionally they will work with the people in the company to educate them on threats like phishing emails and unfamiliar USB drives, and notify them of current events and recommend defensive actions they can take to protect themselves.

Advanced Persistent Threat (APT) - APT's are teams of black hat hackers, often state sponsored, with goals like espionage for political or business gain. Historically they have been behind attacks such as the Stuxnet worm built to damage Iran's nuclear program, the 2014 hack on Sony Pictures, and the 2013 Target breach where malware was installed on point of sale devices to steal credit card information. I recently found a Google Doc with a nice summary of known APT's and their TTPs. They have names like "Fancy Bear", "Unit 121", and "Hurricane Panda". I would classify them as having malicious intent, but with a laser focused goal in mind.

North Korea's Hacker Farm - Bloomberg recently wrote a great article on North Korea's Hacker Farm where hackers are worked up to 15 hours a day and are required to make money for the government by any means necessary. This can mean hacking online gambling sites allowing them to cheat, compromising ad servers to deliver malware to distribute cryptocurrency miners, or distributing ransomware at hospitals and demanding a payout to return their data. These guys are black hats, but I can't help feeling sorry for their situation.

Grey Hat Hackers - Grey hat hackers live in the moral gray area. Some of the previously mentioned hackers can be considered grey hats. I've met bug bounty hunters that used aggressive tactics that border on blackmail to negotiate for a bigger payday for a bug they reported. While the intent was to fix an issue to secure the company, the tactics were a little dirty. One time Mark Zuckerberg's Facebook page was hacked to prove a security flaw, this is against the concept of responsible disclosure and was considered a "grey hat action".

As you can see, the world of hackers is a diverse group of individuals with various motives and techniques, and I find it endlessly fascinating! :)

What happens when you type google.com into your browser and hit enter?

This is a question I heard often while interviewing engineer candidates, and it sounds so simple that most people would say something like: "It loads Google, duh!". However, to a good engineer, this question can take over an hour to answer. Computers are extremely complex, this is not a surprise to anyone. It's easy to take it for granted that it all works so seamlessly every time we sit down and load a website like Google. I wanted to challenge myself and try to answer the question from memory as best I can (mistakes and all!), and possibly re-visit the question in the future as I learn more.

1. Hitting enter on the keyboard completes a circuit that sends a signal to initiate loading of the website into the browser.
2. The operating system has to allocate memory for the incoming data to be stored locally and displayed by the browser (an application, itself running in RAM).
3. All of the code is broken down into 1's and 0's for the processor to compute the data. There are a lot of registers, caches, and a specific instruction set, which is like a language used only by that type of processor.
4. Computing uses power and generates heat, which is dissipated by heatsinks and fans in most modern desktops and laptops, or by passive cooling in mobile devices.
5. The computer's NIC (Network Interface Card) communicates with the local router/gateway and switch to establish a stream of data. A TCP packet is assembled with a header specifying where it wants to go, with a return address for where the data should return, and the payload carrying the data.
6. The router changes the recipient address to itself, this happens for every hop of the network between the client and server. This is like a mailman carrying a letter to the post office, and intentionally labeling the letter with that post office's address so the next recipient of the letter knows where to send it back.
7. The communication happens over copper CAT5 cables with 8 smaller cables that are called "Twisted Pair", with four of them being RX (receive) and four of them being TX (transmit). These are all twisted slightly differently so that there isn't any interference. The long distance communication happens over fiber optic cables at the speed of light. I once heard that a single fiber optic cable can handle enough bandwidth for the entire world to call one place. The limitation is the hardware on both sides, which is always improving.
8. Computers don't know how to load websites by name without relying on DNS (Domain Name Service) to resolve it to an IP address. So a request is sent to your local DNS server asking "What is the IP Address for Google.com?". DNS will likely have this in it's cache and can answer very quickly, otherwise it has to ask the parent DNS server for the address.
9. Once it has the IP Address, the browser will utilize the network stack of the operating system to open a socket on the high end of the port spectrum (probably 50000-65000 for TCP IPV4) and initiate a TCP (Transfer Control Protocol) connection with Google's web server with a SYN, SYN-ACK, ACK (Synchronize, Acknowledge Synchronize, Acknowledge).
10. The TCP protocol is designed so that received data is always verified, and any missing data is re-transmitted.
11. The initial connection will load on port 80 (HTTP), but the web server will automatically redirect it to port 443 (HTTPS).
12. The SSL/TLS handshake begins with a "CLIENT HELLO". It asks the server which ciphers it supports and the client will choose the most secure cipher it can support. Additionally, the client will examine the server's SSL Certificate for validity and authenticity, by asking the issuing CA (Certificate Authority): "Is this really Google?".
13. CAs are the issuers of SSL Certificates. It is their job to verify the people requesting a certificate are really who they say they are. With EV (Extended Validation) certificates, the CA will take it a step further and verify their phone number and address.
14. Once a session key is negotiated using some amazing math (basically magic to me at this point) with prime numbers, encrypted data can be sent to and from the server where only the client and server can decrypt the data. The client uses his private key and Google's public key to encrypt the data, which can only be decrypted by the server with his private key and the client's public key. A man in the middle cannot easily decrypt and read the data without a TLS proxy or having already compromised one of the endpoints. Ideally it will be using a cipher with FS (Forward Secrecy), which adds another layer of encryption to each request. This means that even if the original key is identified, all of the data cannot be decrypted with it.
15. The browser communicates with the web server via the HTTP protocol by sending a GET request. The server replies with data the browser can use to display the website.
16. The HTML/XML/Javascript is rendered into a human readable format that we all see every day.
17. Some additional data is exchanged for analytics, advertisements, and session cookies
18. The information is displayed on a monitor that has hundreds of pixels per inch, at it's most basic level a pixel is a combination of Red, Green, and Blue that is mixed differently to display different colors.

This is not a perfect picture of everything that happens under the hood. You would get a different answer from every engineer you asked, and their answers would highlight their areas of expertise. I think about this question all of the time, and strive to learn more every day. The most amazing thing about this process is that it happens in milliseconds, billions of times every day, with reliability that we've learned to depend on.

Webcam privacy - Should you hide behind tape?

I was recently researching webcam options for some video conferencing and was sad to see that a built-in privacy cover doesn't appear to exist in any new cameras. I used to have a Logitech C600 which had it built-in.

Logitech C600 with built-in privacy cover

It was a simple camera that had my favorite privacy feature. While searching Newegg and Amazon I found that about 3/4 of all webcams are made by Logitech, and while scrolling through pages and pages of new cameras, none had this feature. It's easy to assume this is some high level NSA conspiracy to allow for easier spying... but without evidence to support that it's just some tinfoil hat thinking.

I was recently at an AppSec conference and given a webcam privacy cover that I thought was really cool. Once I had it I started noticing everyone with something similar, or just a piece of tape covering their cameras. Mark Zuckerberg and James Comey even cover their cameras.

Admittedly Zuck and Comey likely have more serious targets on their back and thus their paranoia is probably justified. But my question is this: If a hacker already has enough access to your computer to spy on you, shouldn't your camera be the least of your concerns? I'd be more concerned about my passwords being keylogged, or a man-in-the-middle attack that exposes all of my traffic that's normally encrypted in transit.

Sliding privacy cover

Back to the tinfoil hat thinking, if you Google for webcam covers you'll find lots of companies offering cheap solutions to give people peace of mind. If I were a popular webcam manufacturer like Logitech, wouldn't I take notice that this is a feature that customers want, and build devices like the C600 with it built-in? They could use it as a marketing technique to sell to the privacy concerned masses. I recently Tweeted to Logitech and asked them to share my feedback with their product development team. I don't think I'll single handedly change their minds, but it does make you wonder why they're ignoring such an in-demand feature that they used to offer.

How to: prevent breaches with proactive security

I haven't posted recently due to a move to Germany from Los Angeles, I'm still settling in but I really like it here already. During my move I read a lot about the Equifax breach that impacted over 100 million people's personal information, exposing them to identity theft and whatever else bad guys think of using their information for. This breach had a lot of similarities to other breaches I'd read about over the years, such as Anthem, Target, OPEC, and many more. Hackers find vulnerabilities in their systems and exploit them to exfiltrate sensitive information, and they only need to find one hole to exploit, whereas those responsible for plugging the holes have the never-ending challenge of finding and plugging all of the holes before they are found and exploited.

Blue teaming is hard! (image credit: Geoff Pryor)

I can speak from personal experience from my time as an InfoSec professional working on the "blue team" (defensive hackers), trying to prevent the "red team" (offensive hackers) from accessing sensitive information. I wanted to share a high level overview of what basic things should be done to try to prevent a big breach.

First tip on that list is accepting that you cannot prevent all hacks. An attacker with enough time, motivation, and resources will eventually succeed. These are also known as APT's or Advanced Persistent Threats. The advice on how to deal with APTs can fill many books and is beyond the scope of this blog post.

Second tip is to build a regular patching process, usually weekly. All software has vulnerabilities, and it is essential that all software your company uses is patched to the most recent version.

Third tip is to educate everyone in your company on basic threats like phishing emails, plugging in unfamiliar USB drives, and installing unapproved software (assuming they have install rights to begin with). Specialized education should be given to developers in your organization to make sure they are writing secure code and testing it before it goes into production.

Fourth tip is to scan all of your assets for vulnerabilities, weekly if possible, and send automated reports to the system owners responsible for addressing those vulnerabilities. Ideally you will have regular contact with these system owners and can review the reports together to validate they have a process in place for addressing the high severity vulnerabilities and understand the risk level they present to the organization. It is useful to keep track of trending vulnerability data to help monitor which teams are on top of their patching, and who needs to pick up the slack.

Fifth tip is to segment your network as much as possible to prevent lateral movement. If someone does click a phishing email and their system is compromised, your network should limit that compromised system's access to even more sensitive systems that might have databases like those leaked by hackers in some of the previously mentioned breaches.

Sixth tip is to consider adopting a bug bounty program, and/or hire third party penetration testers to test your defenses. Learn from the issues they find and adapt your environment to become more secure. Reward your bug bounty hackers well and word will spread that you value their efforts and before long you'll have a small army of whitehat hackers competing to find and report issues. It's better to pay a whitehat $10,000 for reporting an SQL injection vulnerability in a sensitive database, than to have a blackhat find and exploit it so you end up on another breach list. Related to this is to have a reliable contact method for reporting security issues, usually a security@ email address that goes to your InfoSec team.

Seventh tip is to have a reliable inventory of assets on your network, and a regularly updated contacts list for those assets and for all business units. I can't tell you how many times I've found a vulnerability in an asset on the network, but I can't easily identify who's responsible for that system.

I'd consider this a short list of low hanging fruit, and not at all comprehensive. Security is hard, but usually you just have to make some effort to not be so easy to hack that any low skilled attacker can compromise you in minutes. I hope this list helps someone out there, and if you have questions feel free to reach out to me on Twitter.

OpSec And Why It Matters To You

OpSec, or Operational Security, is the practice of keeping information secure through awareness of ones daily actions. We've all learned it at one time, imagine planning a surprise party and getting everyone involved to keep it a secret. Some (mostly funny) examples of OpSec failures:

• Hacker brags about taking down an airport website, leading to his arrest.
• People post photos of their debit cards and this Twitter account shames them with retweets.
• Bank robber shares photos of him with large amounts of cash, leads to arrest.

What I really want you to take away from this is a little voice in your head that asks "How can this be used against me?" before posting a photo or walking away from your computer unlocked in a public area.

Bringing Sanity Back To Password Standards Thanks To NIST

I'm sure we've all at one time been frustrated with having to change our password every 90 days, or adhere to an annoyingly complex set of password rules like this:

There's good news though! NIST recently released a new set of password standards that adds a healthy dose of logic to our existing password rules. The full paper can be read here, but it is not a short read so I'd like to summarize it for you:

1. No unnecessary changing of passwords. This means unless there's a breach you shouldn't have to update your password.
2. Remove password rules entirely and instead focus on password length. Length is the best way to increase entropy and encourage the use of a passphrase as opposed to a password. Passphrases can also be easier to remember, compare these two: "J32@oukB" vs "Steve pickle dolphin 5 sunglasses!"
3. Remove password hints. Unfortunately users will sometimes put their password in the hint, or make the hint very easily guessable.
4. Compare the password with a list of "known bad" passwords. This could be a list of the top 1000 most used passwords, or even a collection of millions of weak passwords discovered from data breaches.
5. Allow all ASCII and UNICODE characters, even emojis!
6. Minimum length should be 8 characters, with a maximum of 64. The practice of preventing someone from using a password longer than 16 characters needs to stop!
7. Passwords should be hashed and salted. In the event of a data breach, we should be taking measures to make it difficult for attackers to crack the stolen password hashes.

HTTPS Interception Can Weaken Security

The US-CERT[1] recently published an alert detailing how HTTPS interception can weaken security. This is a topic that was recently discussed in Security Now episode 599[2]where Steve Gibson[3] talks about an awesome research paper[4] (PDF) on the impact of HTTPS interception.

The research paper goes into detail about how they were able to analyze eight billion TLS connections to identify which of those had their connections intercepted. Their results show that about 10% of all US based connections had their TLS broken at some point. This could be from a corporate TLS proxy, a host based Anti-Virus solution, or possibly a malicious actor using a MITM attack.

A TLS proxy acts like a person at the post office opening all of your mail, inspecting it, re-sealing it, and shipping it to the original destination. The problem is when re-sealing it and shipping it off, they aren't always using the same level of security that you started with, leaving you vulnerable to all sorts of attacks.

This is an important topic to be aware of and I'm personally researching how we can raise visibility to problematic TLS proxies, and alert end users when their "assumed secure" connections aren't actually as secure as they should be. I'm hoping to find a browser extension that can detect this, or maybe I need to build a solution myself :).

Credits:
[1] US-CERT
[2] Security Now Podcast
[3] Steve Gibson of GRC
[4] "The Security Impact of HTTPS Interception" by Zakir Durumeric, Zane Ma, Drew Springall, Richard Barnes, and Nick Sulli

Wrestling with a TLS proxy

Some large corporate networks have a TLS proxy to break SSL and inspect traffic for malicious activity. For this to work all of the endpoints on that network need to have the right certs installed so that SSL connections continue to work.

I recently ran into an issue caused by a TLS proxy where my Kali Linux VM (running via Parallels on my Macbook Pro with OSX 10.11.6) was unable to make SSL connections because the TLS handshake was failing due to the invalid chain of trust. So I began to poke around to find and install the correct certs to fix SSL on my VM. I found three certs from a local CA in the OSX keychain, exported those in PEM format and dropped them into my Kali VM via SCP after enabling SSH. Copy/paste would have worked here too except I haven't finished installing the Parallels tools due to a dependency issue I'm still working on solving.

I had to convert the three PEM files into CRT format via openssl like this:

openssl x509 -in certificate.pem -inform PEM -out certificate.crt

Then I copied them to the /usr/local/share/ca-certificates/ folder and ran the "update-ca-certificates" command to import the certs. To test it out I ran an openssl s_client command to Wikipedia:

openssl s_client -showcerts -connect en.wikipedia.org:443

This command initiates a TLS connection and displays the certificates used. I discovered what appeared to be an intermediate cert for the TLS proxy so I copied that and pasted it into a new file, copied it to the ca-certificates folder and ran update-ca-certificates again. Now when I ran openssl showcerts it appeared to be working, but my browser was still encountering cert errors. I suspected that Firefox ESR uses it's own cert store so I went ahead and imported all four certs into Firefox too.

Success! :)

Pro tips for basic security online

I wanted to share some basic security advice I give to family, friends, and my users at work.

1. Keep everything up to date!

The internet is full of bad guys doing bad things. When a new vulnerability in Windows is discovered, Microsoft releases a patch, but that patch isn't any good unless you install it. I know it can be annoying to stop what you're doing and update your computer or phone, but it is essential to staying safe in a modern world where we're surrounded by technology.

2. Maintain a healthy level of suspicion.

• Don't implicitly trust anyone online.
• Be careful what links you click. Especially in email, but also on Skype, Facebook, and anywhere else links can be shared.
• Don't install any software or apps that aren't from a trusted source.
• Even if they appear to be from a trusted source, review reputation information, and verify the site/store that you're downloading it from. Very recently an app called "Meitu" became popular and everyone was installing it and disregarding the permissions it requested (just about all of them!). The app turned out not to be malicious, but users who installed it have already sent their data to the Meitu servers... will they use that data for good?

3. Use strong passwords, enable two factor authentication everywhere possible, and avoid sharing your password across multiple websites.

• Your passwords should have length, complexity, and be unique. This isn't as hard as you think it is. XKCD has a popular comic talking about password strength. A password like "Hunter1" includes a capital and a number, but it's very short and easy to guess. A password like "correct Horse52 battery staple!" is very long, has a capital letter, numbers, several symbols (spaces are considered symbols), and is easy to remember.
• Two factor authentication is when you are asked to enter a code (usually sent to your phone via SMS or by opening an app that displays the code for 30 seconds) after you've entered your username and password. It is especially important because if a bad guy steals your username and password, he still needs your phone to finish logging in. Most popular sites offer 2FA in some form, I recommend checking out Two Factor Auth.org.
• When you register on a website and use the same password you used somewhere else, you're trusting them to store that password securely for as long as that website exists. In 2016 alone there were hundreds of millions of accounts that were hacked. I highly recommend you go to Have I Been Pwned? and sign up with all of your email addresses. It's a site run by Troy Hunt, who collects data breaches and sends out notifications when your information has been found in one.
• Use a password manager like LastPass, KeePass, or 1Password to securely store your passwords. Make sure you use a unique master password, in addition to 2FA. Ideally you would also utilize the "generate random password" feature so that all of your passwords look like this: "JzeKO7OXwN!GyHN&68Ue". You can just copy/paste or autofill the password, and only have to memorize your master password.

There is a lot more to security than what I've listed here, but this is a good start :)