How To Write A Vulnerability Scanner And Earn On Exploits

In this article, we will share some story that was not supposed to be for a public view but we decided to share it with you guys as we thought that could impact on society and make some impact.

So we have written a vulnerability scanner and will try to explain how to make money on it. In the end, you will find some part of the scanner code and specification.

Who and what hacks to

In the 1970s, a long time ago, the Internet looked like this:

The main threat to the internet was presented by researchers and enthusiasts who were the first to reach the documentation and networks of large corporations.

Since then, the Internet has changed a lot. Attacks on networks, services, and web applications have become commonplace. Every day, developers and researchers find dozens of vulnerabilities. In parallel, tens of thousands of sites are hacked.

This is clearly seen on the attack map of Kaspersky Lab. It displays IDS, vulnerability scans and botnet attacks. The map looks like such a beautiful picture and there are millions of attacks a day only in Russia.

Cyber-cross involved not amateurs, but organized groups of dozens of people. They do it out of interest. This is a business. They are not guys in black hoods writing “Black C0d” at night in front of the monitor. This is a socially active people working in offices around the world.

If you work\worked in a telecom, bank, provider or you are a developer in a large company, and especially if you are a security person, then you know that security in a company with more than 10-15 sites is trash.

Let’s try to improve the situation. They say that security is a process.
And a lot of processes.

Briefly list them:

• Threat management
• Vulnerability Management
• Risk management
• Patch management
• Identity & Acess Management
• Security alert
• Security Provider Management
• Other

The listed species in the article will not be considered, hurray.
There are many of them, and the benefits vary according to the size of the organization, the types of assets and their criticality.

We will talk about a process that will benefit even a small business.
From an online store with hundreds of thousands of rubles in revenue to a huge corporation with dozens of data centers.

Vulnerability Detection

In order to understand what are the ways to hack us, let’s look at ourselves through the eyes of an attacker.

Security analysis consists of several stages. For each stage, we will give a basic set of utilities that you can use.

Gathering information about which IP, domains, subnets our purpose has. Instead of knocking on the front door, it’s better to find less protected entry points:

dnsrecon, <span class="hljs-type">Fierce</span>, <span class="hljs-type">Censys</span>, <span class="hljs-built_in">reverse</span>-<span class="hljs-type">PTR</span>-lookup.

Analysis of services for the presence of known vulnerabilities (CVE). For them, public exploits are common:

Vulners, exploit-<span class="hljs-built_in">DB</span>, NIST, CVEdetails

Analysis of services for weak passwords:

potator, hydra

Analysis of web applications for vulnerabilities:

<span class="hljs-symbol">Zap</span>, <span class="hljs-keyword">Burp, </span>w3af, Arachni

Good

Okay, cool. We have a way and tools to check out 1 website.
A check + analysis of one service will take several days to understand thoroughly – a week or more.

Not Good

The problem is that we may have more than one service. And for example / 20 subnet.
4096 addresses. Each may have 65535 services. Not cool.

Yes, there are automated solutions. Qualys, Acunetix, Nessus, there are also domestic vendors. Analysis of what they are good and bad, I suggest leaving for another article.

They did not solve my problems. Decided to fix it. He quit his job and about a year he and his friends went to work out.

How to write a vulnerability scanner

Let’s start with the requirements for the system we wanted to receive:

• Conducts exploration and finds target domains and ip
• Monitors the network perimeter. Sees when a new address appears in the range or a subdomain appears on the protected site.
• Constantly checks addresses sees when network ports open and close
• Finds vulnerabilities and known banner/CPE exploits
• Picks up weak passwords
• Finds application and OS configuration errors
• Finds vulnerabilities in CMS and their plugins.
• Interactively finds XSS, SQLi, XXE, RCE, LFI, RFI and those from OWASP to 10 that are amenable to automated search
• It does this more than once, but constantly re-checks my resources and reacts if new vulnerabilities arise.

Users are owners of large network ranges, that is, those who have 10 or more web applications.
It is necessary to provide daily parallel scanning for web vulnerabilities, weak passwords, configuration errors, and to show exit exploits for hundreds of ip and websites.

To do this, we use a horizontally scalable system. You can add new nodes and new scanner types on the fly. Now the scanner uses 7 nodes and is divided into 2 interacting locations in Germany and the USA. If interested, we will write about it in another article.

• Understood that writing from scratch such a system makes little sense. It required tremendous experience and understanding of various platforms, its own database of network fingerprints, its own CVE database and exploits for them, and a whole huge system dedicated to analyzing the security of web applications.

Where the product license allows it, we decided to use open-source developments. There are components, closed source, but free for commercial use.

So, we have about 20 different applications, which are the components necessary to cover the L3-L7 levels for automated Pentest.

Scanner Interface

Each script, PoC program or binary accepts different parameters to start scanning. Not very comfortable. I wanted to get a unified format to run any possible scanner.

Thought that you need to know any possible scanner to identify the target. Made a table.

Type of check	Data Input
Integration with exploit databases and CVE	Vendor: Product: Version (CPE)
PoC exploit for service	IP, port, CVE
Brute Force Attack	IP, port, application layer protocol
Scanner CMS	Domain, port
Web vulnerability scanner	Domain or IP, port
PoC web vulnerabilities	URL
Port scanner	IP

After the initial verification of nmap and a plugin that accesses CVEdetails for each host, we get a set of such objects. We call them Metadata.

JSON representation:

{
"cves": [],
"service": "mysql",
"protocol": "tcp",
"target": "example.com",
"time": "1513941789",
"product": "mysql",
"vendor": "mysql",
"version": "5.1.63-community",
"port": 3306,
"uri": ""
}

For each port on Target, we get a set of such metadata. Now we need to understand which of our scanners want to work with such a goal.

All scanners have a parent – ScanManager, which makes it elegant:

<span class="hljs-title"><span class="hljs-function">product</span></span><span class="hljs-params"><span class="hljs-function">(scanners, metadata_as_list)</span></span>

Takes all the services and for each of them checks all possible scanners.
Each scanner provides a function that determines whether it wants to work with such a Target.

Let’s talk about what is a scanner.

Considered three options:

• Implement each scanner as a microservice that supports our RPC
• Implement each scanner as a service and use a common message bus
• Third option

But we are very lazy and realized that in the case of the first two options, in addition to the scanners, we also need some kind of infrastructure for deployment, writing RPC, catching bugs of this RPC. Plus we wanted to write everything in Python.

• We tried several solutions. We looked at similar projects to manage scanners, such as Yandex Molly and Minion from Mozilla. We looked at how the work of w3af, Zap. Burp has a very interesting plugin architecture.

We decided that each scanner should be implemented as a serializable function on python. This will allow them to be stored directly in rabbit-mq, quickly delivered to different nodes, atomically executed. We can add new scanners without the need to build and distribute applications. In fact, we received a service for distributed and asynchronous execution of functions on python.

Scanner Format

The scanner launched on our platform must be an object inherited from the base class Scanner.

Any scanner must accept a Metadata object as input and return Vulnerability objects.

class Scanner (object):
name = "scanner_base"
vuln_type = "default_vuln_type"
user_options = {}
Vulnerability_body_fields_to_web_interface = []

def __init __ (self, opts, target, metadata):
self.metadata = metadata
self.opts = opts
self.target = target

@staticmetod
def circuit (Metadata):
'' '
The logic of the scanner.
Accepts objects of type Metadata as input.
The result of the work must be instances of the CVE class.
'' '
return [Vulnerability (), Vulnerability ()]

def check_start_condition (self):
'' '
Checking the parameters that must comply with the Target to run the scanner
True if the scanner should start. In another case, False.
'' '
return true

class ScannerError (Exception):
def __init __ (self, value):
self.value = value
def __str __ (self):
return repr (self.value)

Vulnerability Format

The next task we faced was to create a unified format in which we can store the output of any scanner. Whether it is a web processor, directory brut force or PoC exploit for Memcached.

class Vulnerability (object):
'' 'All vulnerabilities should be cited to this type.
body stores scanner-specific fields.
'' '
def __init __ (self, target, port, scanner, text, VulnerabilityTypes, Score, protocol,
UpdateDate = None, scan_date = None, Complexity = None, Access = None, CWEID = None, Authentication = None,
Integ = None, PublishDate = None, Conf = None, ofExploits = 0, Avail = None,
CVEID = None, references = None, GainedAccessLevel = None, false_positive = False,
fixed = None, body = None):
scan_date = scan_date if scan_date is not None else calendar.timegm (gmtime ())
references = references or []
body = body or {}

self.name = self.construct_cve_name (VulnerabilityTypes, protocol, target, port, credentials, uri, params)

self.data = {
"target": target,
"port": int (port),
"Scanner": scanner,
"Scan_date": scan_date,
"Name": name,
"UpdateDate": UpdateDate,
"VulnerabilityTypes": VulnerabilityTypes,
"Complexity": complexity,
"text": text,
"Access": Access,
"CWEID": CWEID,
"Hash": sha1 (self.name.encode ('utf-8')). Hexdigest (),
"Authentication": Authentication,
"Integ": Integ,
"PublishDate": PublishDate,
"Conf": Conf,
"ofExploits": ofExploits,
"Score": Score,
"Avail": avail,
"CVEID": CVEID,
"References": references,
"GainedAccessLevel": GainedAccessLevel,
"FalsePositive": false_positive,
"Fixed": fixed,
"Body": body
}

@staticmethod
def construct_cve_name (VulnerabilityTypes, protocol, target, port, credentials, uri, params):
# vuln_type: host: port: protocol: credentials: uri: params
return "{}: {}: {}: {}: {}: {}: {}". format (
VulnerabilityTypes,
target,
port,
protocol,
credentials,
uri,
params)

def __str __ (self):
return self.data

def __repr __ (self):
return self.name

We decided not to fence our formats, but to use the industry standard – the CVE format. We will enrich CVE with our vulnerability metadata for easy storage and retrieval. To store scanner-specific data, add a body field. Which keys from the body should be displayed in the web interface is determined by the Scanner.

We tried to give the authors of scanners a maximum of freedom in implementation.
And we invite you to participate in the development.

How to make money on exploits

We want people who are interested in security, authors of utilities and scripts, researchers, to be able to absolutely legally monetize the results of their work.

You can sell you exploit on black, grey, white markets. If you interested you can try to sell it to ZERODIUM which pays premium bounties and rewards to security researches.