Greetings and thanks for stopping by! It is with some seriously mixed emotions that I bring this blog post to you, as this post is the culmination of a failed business and nearly two years of heart-felt labor.
tl;dr I tried starting a company entitled Web Sight, worked on it alone for two years, was unable to get the company off the ground, and have decided to open source all of the software that I wrote. The software can be found in three repositories:
- Front-end Single-page Application – this is the front-end for the API
- Back-end Distributed System – this is the back-end including the API, the task nodes, and all the stuff that does the heavy lifting
- Dockerized Deployment – this is all of the software packaged up into a simple-to-use Docker deployment
If you’re looking for a bit more context on what Web Sight is and what you can do with it, read on! Also if you’re going to be at Black Hat this year come check Web Sight out at my Arsenal talk at 10:00AM on Thursday 07/27/2017!
What is Web Sight?
Web Sight is a software platform that enables red and blue teams to automate the information gathering processes required by their day-to-day jobs. At present, Web Sight performs the following activities:
- Domain name enumeration
- DNS record enumeration
- Network scanning (large scale)
- Network service fingerprinting
- SSL support enumeration
- SSL certificate inspection
- Application-layer inspection for supported application protocols (currently only HTTP)
These activities are entirely automated, and require only the following information as scanning “seeds”:
- Network ranges
- Domain names
For web applications that are discovered across an organization’s domain names and network ranges, the following activities are conducted:
- Virtual host enumeration
- User agent enumeration
- Screen shotting
The goal of automating this information gathering process is to provide users with the situational awareness that proper security strategizing (both offensively and defensively) requires.
Why is Web Sight?
As I’ve stated in blog posts before, I really enjoy breaking into things and networks in particular are my jam. I’ve worked as a security consultant for a few years and have been on quite a few network penetration tests and red team exercises. I typically spent the first half of these engagements gathering information and finding the low-hanging fruit. From my perspective, the only thing I needed to do to get in was to look harder than anyone had looked before.
And so, time and time again, this approach paid off. I found things that everyone else had missed – IP addresses, applications, domain names, network services, etc – and those things that had escaped everyone else’s attention were commonly vulnerable. I was so successful with this approach that I started being put on gigs that were longer in duration and that had more focus on helping organizations defend themselves after I had gotten into them 10 ways from Sunday.
Once I found myself in the position of having to counsel these organizations on how to make it so that I couldn’t get in again, things got a lot harder. I had found the stuff that they had previously missed, which in turn meant that all of their vulnerability scanners, IDS+IPS+DLP systems, endpoint monitoring, and the like was missing the stuff that mattered. Furthermore, I was now in charge of enumerating what the organizations owned that had fallen out of management.
At first I tried doing the same thing I had always done – running all of the various tools I use and collating all of that information into one semi-cohesive understanding of the environment. It worked to an extent, but the immediate problem I ran into was that by the time I was done gathering this information, half of it was stale. Manually solving this problem was not an option.
And so I automated everything that I did. From network scans to subdomain enumeration to DNS record lookups and web application screen shots, I wanted a platform that I could hand a few “seeds” of information and have it provide me with the enumerated attack surface of the targeted organization. While there are still many, many things that should be integrated into Web Sight, as it stands now it can perform some serious introspection into enterprises at scale.
What Technologies Does Web Sight Use?
Web Sight makes use of a diverse number of technologies:
- PostgreSQL – Database server
- Python Celery – Distributed task management
- RabbitMQ – Distributed message queue
- Django REST Framework – REST API
- Scrapy – Web scraping
- PhantomJS – Web application screen shotting
- Docker – Dockerized deployment
- Elasticsearch – Storage of collected data
- Angular 2 – Front-end application
- Redis – IPC and task history tracking
How Do I Use Web Sight?
I’d recommend checking out the Dockerized deployment repository and getting Web Sight set up locally (the documentation in the repository should be enough to get things rolling). Given how complex the software is, the Docker deployment option is by far the least painful. Once it’s up and running navigate to the front-end application and get started!
Once you log in you’ll be directed to part of the UI where you can add organizations. An organization can have a number of network ranges and domain names associated with it, which are in turn used as seeds to begin gathering information about the organization’s attack surface. To add a new organization, use the controls highlighted below:
Once you’ve added an organization, you can configure the organization by clicking on the following button:
From here, you can add domain names and network ranges to the organization. Note that individual domain names and networks can be toggled off and on via the included toggle switch:
Once you’ve added all of the relevant networks and domain names, you can start a scan by clicking on the following button:
This kicks off the scan! Once the scan has completed, you can browse the collected data by viewing the organization via the following button:
This will bring you to a page where you can choose between SSL certificates and web applications:
The contents of the web applications page are shown below:
The contents of the SSL certificate page are shown below:
All of the web applications and SSL certificates can be drilled down into via the following button on the relevant list page:
An example of SSL certificate details is shown below:
All of the data found in the list pages can also be exported to Excel and CSV files via the following button:
This button pops up an export modal where you can choose what data to export:
The list pages also have analytics and full text search. Clicking on any of the pie charts will apply filters based on the clicked value, and the search bar will similarly restrict results:
The UI currently only displays data for SSL certificates and web applications, whereas the API also has data about domain names and IP addresses. This other data will hopefully by present in the UI soon!
Where Is The Documentation?
Unfortunately I don’t have a Wiki set up yet, and the only documentation I have to offer is the documentation that is present in the README files in the software repositories. That being said, I’d love to get some feedback around what sort of documentation people require, and if you have any questions come submit a ticket and I’ll see to it that I respond ASAP.
Also, if you’re going to be at Black Hat this year then come check my Arsenal talk on Thursday 07/27/2017 at 10:00AM to see what Web Sight can do in the flesh!