Skip to content

Commit

Permalink
v0.2 release
Browse files Browse the repository at this point in the history
  • Loading branch information
jonathanwalker committed Jan 11, 2023
1 parent ffd58eb commit bf6b6b6
Show file tree
Hide file tree
Showing 7 changed files with 278 additions and 6 deletions.
24 changes: 18 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,19 @@

<img src="assets/logo.png" width="400" height="300" align="right">

Nuclear Pond is used to leverage [Nuclei](https://github.com/projectdiscovery/nuclei) in the cloud with unremarkable speed, flexibility, and perform internet wide scans for far less than a cup of coffee.
Nuclear Pond is used to leverage [Nuclei](https://github.com/projectdiscovery/nuclei) in the cloud with unremarkable speed, flexibility, and [perform internet wide scans for far less than a cup of coffee](https://devsecopsdocs.com/blog/nuclear-pond/).

It leverages [AWS Lambda](https://aws.amazon.com/lambda/) as a backend to invoke Nuclei scans in parallel, choice of storing json findings in s3 to query with [AWS Athena](https://aws.amazon.com/athena/), and is easily one of the cheapest ways you can execute scans in the cloud.

## Features

- Output results to your terminal, as json, or to an S3 data lake
- Output results to your terminal, as json, or to an S3
- Specify threads and parallel invocations in any desired number of batches
- Specify any Nuclei arguments just like you would locally
- Specify a single host or from a file
- Run the http server to take scans from the API
- Run the http server to the status of the scans
- Query findings through Athena for searching

## Usage

Expand All @@ -26,9 +29,14 @@ To install Nuclear Pond, you need to configure the backend [terraform module](ht
$ go install github.com/DevSecOpsDocs/nuclearpond@latest
```

### Backend Configuration
## Environment Variables

You can either pass in your backend with flags or through environment variables. You can use `-f` or `--function-name` to specify your Lambda function and `-r` or `--region` to the specified region. The environment variables are `AWS_REGION` and `AWS_LAMBDA_FUNCTION_NAME`.
You can either pass in your backend with flags or through environment variables. You can use `-f` or `--function-name` to specify your Lambda function and `-r` or `--region` to the specified region. Below are environment variables you can use.

- `AWS_LAMBDA_FUNCTION_NAME` is the name of your lambda function to execute the scans on
- `AWS_REGION` is the region your resources are deployed
- `NUCLEARPOND_API_KEY` is the API key for authenticating to the API
- `AWS_DYNAMODB_TABLE` is the dynamodb table to store API scan states

### Command line flags

Expand All @@ -43,17 +51,21 @@ Usage:

Flags:
-a, --args string nuclei arguments as base64 encoded string
-b, --batch-size int batch size to run nuclei in parallel (default 1)
-b, --batch-size int batch size for number of targets per execution (default 1)
-f, --function-name string AWS Lambda function name
-h, --help help for run
-o, --output string output type to save nuclei results(s3, cmd, or json) (default "cmd")
-r, --region string AWS region to run nuclei
-s, --silent silent command line output
-t, --target string individual target to specify
-l, --targets string list of targets in a file
-c, --threads int number of threads to run nuclei in parallel (default 1)
-c, --threads int number of threads to run lambda functions, default is 1 which will be slow (default 1)
```

## Custom Templates

The terraform module uploads a zip file with templates downloaded directly from nuclei-templates repository but it can be customized by providing any url to your templates. This can be through downloading your repositories zip file and retrieving the url with the key as a variable. To reference these instead of the latest, also additionally for performance benefits, you can reference these by adding the flag `-t /opt/nuclei-templates-9.3.4/dns`. The directory `nuclei-templates-9.3.4` may change depending on what the folder name is within the zip file and in our case it includes the release.

## Retrieving Findings

If you have specified `s3` as the output, your findings will be located in S3. The fastest way to get at them is to do so with Athena. Assuming you setup the terraform-module as your backend, all you need to do is query them directly through athena. You may have to configure query results if you have not done so already.
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ go 1.19
require (
github.com/aws/aws-sdk-go v1.44.171
github.com/common-nighthawk/go-figure v0.0.0-20210622060536-734e95fb86be
github.com/google/uuid v1.3.0
github.com/spf13/cobra v1.6.1
)

Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ github.com/common-nighthawk/go-figure v0.0.0-20210622060536-734e95fb86be/go.mod
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/inconshreveable/mousetrap v1.0.1 h1:U3uMjPSQEBMNp1lFxmllqCPM6P5u/Xq7Pgzkat/bFNc=
github.com/inconshreveable/mousetrap v1.0.1/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
Expand Down
18 changes: 18 additions & 0 deletions pkg/cmd/cmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (

"github.com/DevSecOpsDocs/nuclearpond/pkg/core"
"github.com/DevSecOpsDocs/nuclearpond/pkg/helpers"
"github.com/DevSecOpsDocs/nuclearpond/pkg/server"

"github.com/common-nighthawk/go-figure"
"github.com/spf13/cobra"
Expand Down Expand Up @@ -75,6 +76,22 @@ var runCmd = &cobra.Command{
},
}

var startServer = &cobra.Command{
Use: "serve",
Short: "Launch API to launch run tasks to the nuclei runner.",
Long: "Executes nuclei through an API through asynchronous lambda functions",
Run: func(cmd *cobra.Command, args []string) {
// Print banner
fmt.Println(asciiBanner)
fmt.Println(" devsecopsdocs.com")
fmt.Println()
// Start server
log.Println("Running nuclear pond http server on port 8080")
log.Println("http://localhost:8080")
server.HandleRequests()
},
}

func init() {
// Mark flags as required
runCmd.MarkFlagRequired("args")
Expand Down Expand Up @@ -121,6 +138,7 @@ func Execute() error {

rootCmd.HasHelpSubCommands()
rootCmd.AddCommand(runCmd)
rootCmd.AddCommand(startServer)

return rootCmd.Execute()
}
35 changes: 35 additions & 0 deletions pkg/server/keys.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
package server

import (
"crypto/rand"
"encoding/hex"
"log"
"net/http"
"os"
)

func generateAPIKey() string {
// if NUCLEARPOND_API_KEY is set, use that
if os.Getenv("NUCLEARPOND_API_KEY") != "" {
return os.Getenv("NUCLEARPOND_API_KEY")
}
// otherwise, generate a random API key
b := make([]byte, 16)
_, err := rand.Read(b)
if err != nil {
log.Fatal(err)
}
apiKey := hex.EncodeToString(b)
log.Println("Generated API key:", apiKey)
os.Setenv("NUCLEARPOND_API_KEY", apiKey)
return apiKey
}

func checkAPIKey(r *http.Request) bool {
apiKey := generateAPIKey()
key := r.Header.Get("X-NuclearPond-API-Key")
if key != apiKey {
return false
}
return true
}
110 changes: 110 additions & 0 deletions pkg/server/scanner.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
package server

import (
"encoding/base64"
"fmt"
"log"
"os"
"strings"
"time"

"github.com/DevSecOpsDocs/nuclearpond/pkg/core"
"github.com/DevSecOpsDocs/nuclearpond/pkg/helpers"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
)

func backgroundScan(scanInput Request, scanId string) {
targets := helpers.RemoveEmpty(scanInput.Targets)
batches := helpers.SplitSlice(targets, scanInput.Batches)
output := scanInput.Output
threads := scanInput.Threads
NucleiArgs := base64.StdEncoding.EncodeToString([]byte(scanInput.Args))
silent := true

// Fail if AWS_LAMBDA_FUNCTION_NAME and AWS_REGION are not set
functionName := os.Getenv("AWS_LAMBDA_FUNCTION_NAME")
regionName := os.Getenv("AWS_REGION")
dynamodbTable := os.Getenv("AWS_DYNAMODB_TABLE")
if functionName == "" || regionName == "" || dynamodbTable == "" {
log.Fatal("AWS_LAMBDA_FUNCTION_NAME is not set")
}

// Convert scanId to a valid DynamoDB key
requestId := strings.ReplaceAll(scanId, "-", "")

log.Println("Initiating scan with the id of ", scanId, "with", len(targets), "targets")
storeScanState(requestId, "running")
core.ExecuteScans(batches, output, functionName, NucleiArgs, threads, silent)
storeScanState(requestId, "completed")
log.Println("Scan", scanId, "completed")
}

func storeScanState(requestId string, status string) error {
log.Println("Stored scan state in Dynamodb", requestId, "as", status)

sess, err := session.NewSession(&aws.Config{
Region: aws.String(os.Getenv("AWS_REGION")),
})
if err != nil {
return err
}
// Create DynamoDB client
svc := dynamodb.New(sess)
// Prepare the item to be put into the DynamoDB table
item := &dynamodb.PutItemInput{
TableName: aws.String(os.Getenv("AWS_DYNAMODB_TABLE")),
Item: map[string]*dynamodb.AttributeValue{
"scan_id": {
S: aws.String(requestId),
},
"status": {
S: aws.String(status),
},
"timestamp": {
N: aws.String(fmt.Sprintf("%d", time.Now().Unix())),
},
"ttl": {
N: aws.String(fmt.Sprintf("%d", time.Now().Add(time.Duration(30*time.Minute)).Unix())),
},
},
}
// Store the item in DynamoDB
_, err = svc.PutItem(item)
if err != nil {
log.Println("Failed to store scan state in Dynamodb:", err)
return err
}

return nil
}

// function to retrieve the scan state from DynamoDB
func getScanState(requestId string) (string, error) {
log.Println("Retrieving scan state from Dynamodb", requestId)

sess, err := session.NewSession(&aws.Config{
Region: aws.String(os.Getenv("AWS_REGION")),
})
if err != nil {
return "failed", err
}
// Create DynamoDB client
svc := dynamodb.New(sess)
// Prepare the item to be put into the DynamoDB table
item := &dynamodb.GetItemInput{
TableName: aws.String(os.Getenv("AWS_DYNAMODB_TABLE")),
Key: map[string]*dynamodb.AttributeValue{
"scan_id": {
S: aws.String(requestId),
},
},
}
// Store the item in DynamoDB
result, err := svc.GetItem(item)
if err != nil {
return "failed", err
}
return *result.Item["status"].S, nil
}
94 changes: 94 additions & 0 deletions pkg/server/server.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
package server

import (
"encoding/json"
"fmt"
"log"
"net/http"
"strings"

"github.com/google/uuid"
)

type Request struct {
Targets []string `json:"Targets"`
Batches int `json:"Batches"`
Threads int `json:"Threads"`
Args string `json:"Args"`
Output string `json:"Output"`
}

// Index
func indexHandler(w http.ResponseWriter, r *http.Request) {
// check if the key matches the generated API key
if !checkAPIKey(r) {
http.Error(w, "Invalid API key", http.StatusUnauthorized)
return
}
fmt.Fprintf(w, "Welcome to the API")
}

// Health check
func healthHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "ok")
}

func scanStatusHandler(w http.ResponseWriter, r *http.Request) {
// check if the key matches the generated API key
if !checkAPIKey(r) {
http.Error(w, "Invalid API key", http.StatusUnauthorized)
return
}

path := r.URL.Path
parts := strings.Split(path, "/")
if len(parts) < 3 {
http.Error(w, "Invalid path", http.StatusBadRequest)
return
}
scanId := parts[2]

log.Println("Getting scan state for", scanId)

log.Println("Getting scan state for", scanId)

state, err := getScanState(scanId)
if err != nil {
http.Error(w, "Error getting scan state: "+err.Error(), http.StatusInternalServerError)
return
}
json.NewEncoder(w).Encode(map[string]string{"status": state})
}

// Scan
func scanHandler(w http.ResponseWriter, r *http.Request) {
// check if the key matches the generated API key
if !checkAPIKey(r) {
http.Error(w, "Invalid API key", http.StatusUnauthorized)
return
}

var req Request
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "Error decoding JSON: "+err.Error(), http.StatusBadRequest)
return
}

log.Println("Received request", req)

scanId := uuid.New().String()
go backgroundScan(req, scanId)
json.NewEncoder(w).Encode(map[string]string{"RequestId": scanId})
}

func HandleRequests() {
// generate API key
generateAPIKey()

http.HandleFunc("/", indexHandler)
http.HandleFunc("/health-check", healthHandler)
http.HandleFunc("/scan", scanHandler)
http.HandleFunc("/scan/", scanStatusHandler)

http.ListenAndServe(":8080", nil)
}

0 comments on commit bf6b6b6

Please sign in to comment.