Coalfire NPK – Installation, Configuration, and Usage

I setup Coalfire NPK recently for some password cracking and wanted to share the process and my thoughts.

Coalfire NPK – Introduction

First, if you’ve never heard of NPK, then you can check out the release announcement here.

Additionally, the GitHub repository is worth checking out.

NPK is supposed to serve as a server-less, distributed hashcat platform, and I’d say that is fairly true.

I needed to crack some hashes for work, and I figured that this would be more cost effective than trying to buy our own cracking rig.

Prerequisites

First, I created a local copy of the repository.

doyler@mbp:~/tools# git clone https://github.com/Coalfire-Research/npk
Cloning into 'npk'...
remote: Enumerating objects: 90, done.
remote: Counting objects: 100% (90/90), done.
remote: Compressing objects: 100% (65/65), done.
remote: Total 761 (delta 33), reused 47 (delta 22), pack-reused 671
Receiving objects: 100% (761/761), 5.48 MiB | 17.77 MiB/s, done.
Resolving deltas: 100% (173/173), done.

Next, I installed jsonnet and jq, per the instructions.

doyler@mbp:~/tools/npk# pip install jsonnet
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting jsonnet
  Downloading https://files.pythonhosted.org/packages/33/b8/a8588d4010f13716a324f55d23999259bad9db2320f4fe919a66b2f651f3/jsonnet-0.15.0.tar.gz (255kB)
     |�-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-��-�| 256kB 2.5MB/s 
Building wheels for collected packages: jsonnet
  Building wheel for jsonnet (setup.py) ... done
  Stored in directory: /Users/doyler/Library/Caches/pip/wheels/57/63/2e/da89cfe1ba08550bd7262d5d9c027edc313980c3b85b3b0a38
Successfully built jsonnet
Installing collected packages: jsonnet
Successfully installed jsonnet-0.15.0

... <snip> ...

doyler@mbp:~/tools/npk# brew install jq
==> Installing dependencies for jq: oniguruma
==> Installing jq dependency: oniguruma
==> Downloading https://homebrew.bintray.com/bottles/oniguruma-6.9.4.mojave.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/ab/ab2bb92e40e17569c54dda0ed3b3a0fc6f98be761107fba918754af75817d
######################################################################## 100.0%
==> Pouring oniguruma-6.9.4.mojave.bottle.tar.gz
 /usr/local/Cellar/oniguruma/6.9.4: 17 files, 1.3MB
==> Installing jq
==> Downloading https://homebrew.bintray.com/bottles/jq-1.6.mojave.bottle.1.tar.gz
==> Downloading from https://akamai.bintray.com/71/71f0e76c5b22e5088426c971d5e795fe67abee7af6c2c4ae0cf4c0eb98ed2
######################################################################## 100.0%
==> Pouring jq-1.6.mojave.bottle.1.tar.gz
 /usr/local/Cellar/jq/1.6: 18 files, 1MB

I also needed to install the AWS CLI, which was simple enough.

Finally, I installed Terraform, and I was ready to go.

Coalfire NPK – Installation (and Hiccups)

With the prerequisites installed, I changed the npk-settings.json file.

doyler@mbp:~/tools/npk/terraform# cp npk-settings.json.sample npk-settings.json
doyler@mbp:~/tools/npk/terraform# vi npk-settings.json

Next, I setup my AWS credential file. For a good example, I recommend the following guide.

doyler@mbp:~/tools/npk/terraform# mkdir ~/.aws
doyler@mbp:~/tools/npk/terraform# vi ~/.aws/credentials

I verified my credential file and access with the ‘iam get-user’ AWS CLI command.

doyler@mbp:~/tools/npk/terraform# aws --profile NPKuser iam get-user
{
    "User": {
        "Path": "/",
        "UserName": "NPKuser",
        "UserId": "A...XJ",
        "Arn": "arn:aws:iam::794951671079:user/NPKuser",
        "CreateDate": "2020-02-19T18:38:15+00:00"
    }
}

With everything in place, I ran the deploy script. Unfortunately, I ran into some issues with attribute types.

doyler@mbp:~/tools/npk/terraform# ./deploy.sh 
[*] Preparing to deploy NPK.
[*] Getting availabilityzones from AWS
[*] - us-east-1
[*] - us-east-2
[*] - us-west-1
[*] - us-west-2
[*] Checking service-linked roles for EC2 spot fleets

An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name AmazonEC2SpotFleetRole cannot be found.

...

Error: Incorrect attribute value type

  on routetable.tf.json line 39, in resource.aws_route_table.us-west-2:
  39:             "route": {
  40:                "cidr_block": "0.0.0.0/0",
  41:                "gateway_id": "${aws_internet_gateway.us-west-2.id}"
  42:             },

Inappropriate value for attribute "route": set of object required.

After looking through the NPK issues, this was due to my Terraform version.

doyler@mbp:~/tools/npk/terraform# terraform --version
Terraform v0.12.20

I removed the latest version, and installed v0.11.0.

doyler@mbp:~/tools/npk/terraform# rm /usr/local/bin/terraform 
doyler@mbp:~/tools/npk/terraform# wget https://releases.hashicorp.com/terraform/0.11.0/terraform_0.11.0_darwin_amd64.zip
--2020-02-19 13:44:27--  https://releases.hashicorp.com/terraform/0.11.0/terraform_0.11.0_darwin_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.129.183, 151.101.193.183, 151.101.1.183, ...
Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.129.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 15753806 (15M) [application/zip]
Saving to: 'terraform_0.11.0_darwin_amd64.zip'

terraform_0.11.0_darwin_amd 100%[===========================================>]  15.02M  28.8MB/s    in 0.5s    

2020-02-19 13:44:28 (28.8 MB/s) - 'terraform_0.11.0_darwin_amd64.zip' saved [15753806/15753806]

doyler@mbp:~/tools/npk/terraform# unzip terraform_0.11.0_darwin_amd64.zip 
Archive:  terraform_0.11.0_darwin_amd64.zip
  inflating: terraform               
doyler@mbp:~/tools/npk/terraform# rm terraform_0.11.0_darwin_amd64.zip 
doyler@mbp:~/tools/npk/terraform# mv terraform /usr/local/bin/
doyler@mbp:~/tools/npk/terraform# terraform --version
Terraform v0.11.0

This time, when I ran the deploy script, everything finished!

doyler@mbp:~/tools/npk/terraform# ./deploy.sh 
[*] Preparing to deploy NPK.
[*] Getting availabilityzones from AWS
[*] - us-east-1
[*] - us-east-2
[*] - us-west-1
[*] - us-west-2
[*] Checking service-linked roles for EC2 spot fleets

...

Apply complete! Resources: 0 added, 2 changed, 0 destroyed.

Outputs:

admin_create_user_command = aws --region us-west-2 --profile NPKuser cognito-idp admin-create-user --user-pool-id us-west-2_KF3XeeGEl --username redacted --user-attributes '[{"Name": "email", "Value": "redacted"}, {"Name": "email_verified", "Value": "true"}]' --temporary-password redacted
admin_password = redacted
cloudfront_url = d319eumg867kv9.cloudfront.net
s3_static_site_sync_command = aws --profile NPKuser s3 --region us-west-2 sync /Users/doyler/tools/npk/terraform/../site-content/ s3://npk-site-content-20200219184518947700000007

I also had to subscribe to the Amazon Linux AMI with NVIDIA TESLA GPU Driver
, but that was simple enough.

When I went to the Cloudfront URL, I saw the NPK login screen.

First Cracking Attempt

When I logged in with the temporary password, I saw NPK dashboard.

I setup a new campaign to try and crack the descrypt hashes for my engagement.

When I tried to start this campaign, I received some blank error messages, and nothing seemed to start.

Coalfire NPK – Debugging

When I looked at the campaign status, I saw an error about the service-linked role.

Error: Error requesting spot fleet: AuthFailure.ServiceLinkedRoleCreationNotPermitted: The provided credentials do not have permission to create the service-linked role for EC2 Spot Fleet.

I found some documentation that indicated that my roles and permissions were incorrect.

First, I tried to manually create the JSON with the AWSServiceRoleForEC2Spot role.

{
            "Sid": "8",
            "Effect": "Allow",
            "Action": [
                "iam:CreateServiceLinkedRole",
                "iam:PutRolePolicy"
            ],
            "Resource": "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot",
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "spot.amazonaws.com"
                }
            }
        }

I changed the npk_lambda_api_handler_policy and npk_lambda_api_handler_role, as I wasn’t sure which of these the error was coming from.

Unfortunately, this was still still failing, after updating npk_fleet_role_policy as well.

I created a GitHub issue, to try and get some help during my debugging process.

First, it looked like my return code was 254 instead of 255, so I edited the deploy scripts for this case.

$ aws --profile NPKuser iam get-role --role-name AmazonEC2SpotFleetRole

An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name AmazonEC2SpotFleetRole cannot be found.
$ echo $?
254

I ran the destroy command after changing the scripts, to ensure that everything was properly removed.

doyler@mbp:~/tools/npk/terraform# terraform destroy
There are warnings related to your configuration. If no errors occurred,
Terraform will continue despite these warnings. It is a good idea to resolve
these warnings in the near future.

Warnings:

  * aws_cognito_user_pool.npk: "admin_create_user_config.0.unused_account_validity_days": [DEPRECATED] Use password_policy.temporary_password_validity_days instead

...

aws_s3_bucket.static_site: Destruction complete after 3m49s

Destroy complete! Resources: 127 destroyed.

When I ran the deploy script again, it looked like everything was again good to go.

doyler@mbp:~/tools/npk/terraform# ./deploy.sh 
[*] Preparing to deploy NPK.
[*] Getting availabilityzones from AWS
[*] - us-east-1
[*] - us-east-2
[*] - us-west-1
[*] - us-west-2
[*] Checking service-linked roles for EC2 spot fleets

...

Apply complete! Resources: 0 added, 2 changed, 0 destroyed.

Outputs: (redacted)

Unfortunately, I ended up with a ton of different AWS errors, due to destroy not removing everything properly.

aws_cognito_user_pool.npk: Modifications complete after 6s (ID: us-west-2_I8pd6eQTI)

Error: Error applying plan:

4 error(s) occurred:

* aws_key_pair.us-west-1: 1 error(s) occurred:

* aws_key_pair.us-west-1: Error import KeyPair: InvalidKeyPair.Duplicate: The keypair 'npk-key' already exists.
        status code: 400, request id: 864f0d74-f4e9-4d5b-bd0d-cda3bca377b0
* aws_dynamodb_table.settings: 1 error(s) occurred:

* aws_dynamodb_table.settings: error creating DynamoDB Table: ResourceInUseException: Table already exists: Settings
* aws_subnet.us-west-2a: 1 error(s) occurred:

* aws_subnet.us-west-2a: Error creating subnet: InvalidSubnet.Conflict: The CIDR '10.202.1.0/24' conflicts with another subnet
        status code: 400, request id: 4d5f16da-1aa4-4dbf-81de-ac6d404c74e5
* aws_dynamodb_table.campaigns: 1 error(s) occurred:

* aws_dynamodb_table.campaigns: error creating DynamoDB Table: ResourceInUseException: Table already exists: Campaigns

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Once I manually removed EVERYTHING, the deployment succeeded, and I no longer received these errors.

Second Attempt

With everything stood up again, I was able to successfully start a new campaign.

Going to the campaign page, it showed the price increasing and the status as “RUNNING”.

Unfortunately, this campaign ended quickly, and I did not crack the hash. When I took a look at the potfile, it seemed like hashcat wasn’t really running.

Credentials loaded
[ '--quiet',
  '-O',
  '--remove',
  '--potfile-path=/potfiles/i-0715e8ac226a8f026.potfile',
  '-o',
  '/potfiles/cracked_hashes-i-0715e8ac226a8f026.txt',
  '-w',
  '4',
  '-m',
  1500,
  '-a',
  0,
  '-r',
  '/root/npk-rules/NSAKEY.v2.dive.rule.txt',
  '/root/hashes.txt',
  '/root/npk-wordlist/rockyou.txt' ]
Found status report in output
nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported


Caught error: TypeError: Cannot read property 'split' of undefined


Died with code 255 and signal 0

Dying words:

After looking at more GitHub issues, I realized that my hash file was improperly formatted.

After fixing my hash file, everything was up and running properly.

Note that I was still receiving errors upon starting a campaign, but these would go away immediately and not negatively affect anything.

Finally, the actual statistics while a campaign are running are neat, and cool enough to throw up on a dashboard or something similar.

Coalfire NPK – Conclusion

While I was able to finally get everything working, I do not think that I will move forward with NPK.

Everything was a bit difficult to setup, and it doesn’t give any option for raw hashcat interaction.

I was hoping to be able to add brute-force options, or more easily upload different rules/dictionaries.

Finally, here is me trying to setup a mask manually, which obviously didn’t work.

If you have any suggestions for cloud cracking platforms or configuration, then please let me know!