Vulnerability History Project

Cite This Data

---

To cite the writing on vulnerabilityhistory.org, use:

To cite data from a specific project, use:

• Chromium
• HTTPD
• Tomcat
• Struts
• Django

Table of Contents

---

• Getting Set Up
  • Normal Setup
  • With Docker
    • Common Gotchas
    • Testing
    • Running Commands
    • Reference
• Finding Useful Commands
• Rebuild the Database
• Reload the Production Data
• Testing VHP
• Useful Commands
• Helpful Dev Tools
• The CWE csv File
• IntelliJ Setup

Getting Set Up

---

Normal Setup

Normally

• Install Ruby 3.2.x
• Install NodeJS (16.x)
• Install Yarn
• Install PostgreSQL (latest - pgAdmin is also recommended)
• In PostgreSQL, create a user called "vhp" and "vhp_test". Set the permissions to "Can Log In". Create a random password and set that password for the vhp user. Put that password into secrets.yml (see next bullet). Create two databases: "vhp" and "vhp_test", and set the owners for both of those databases as vhp
• Create a config/secrets.yml based on config/secrets.default.yml. Follow the comments in that file.
• After that, we'll need to install all of our dependencies. From the root of the repo on the command line run:

  yarn install
  bundle install

Then, let's set up our database and load our data:

  rails db:schema:load
  rails data:clear data:all
  rails server

That second one can take a while. Once the server is up, go to http://localhost:3000 to see the site.

If you are working on the frontend, it's useful to run Webpacker's dev server:

  yarn watch

This speeds up rebuilding the frontend JS, CSS, etc. If you want to do a full build of webpack, do:

  yarn build

Windows Considerations

• Use the Ruby from RubyInstaller.org. Get the installer with DevKit. These are compilers that will build C/C++ code that some gems depend on
• The installer does take some time (a few minutes).
• When done, make sure the Run ridk install is checked.
  • This will take additional time.
  • The "base installation" is fine ("if unsure press enter")
• If you had a previous version of Ruby installed
  • Make sure you check your PATH environment variable that the correct Ruby is on your path, W
  • We recommend removing old Rubies (unless you have reason to switch between the two)
  • When you get a new Ruby on Windows, you'll need to reinstall all your gems. Use bundle install

With Docker

Building the project with Docker is meant to be an easy setup so there aren't any OS-specific issues throughout the dev process. That being said, this way of developing still needs a lot of testing to make sure it's good to go, and I wouldn't really advise trying this unless you have some time to debug too.

Common Gotchas

• Windows Users

  If you are attempting this on Windows, it's been been noted that the way the OS interprets the Dockerfile will cause docker-compose to complain about a file not existing. Because of this, you should run the dos2unix command on the file before trying to run it. This article explains better

• Directories

  The Docker convention for container naming is foldername_imagename. That means if we don't change our folder name from the repo and are building the app container, it will be named vulnerability-history_app, and other containers will look for this name. This naming is set up by default in docker-compose.yml, however if you have renamed your folder, you will want to change the references from vulnerability-history to whatever your folder name is (webpacker and test are two containers that will need this done). If you changed the folder to vhp, you would update the references to vhp_app.

Getting Started

You'll need docker and docker-compose on your machine to start. You can either do this by means of Docker Desktop or through a package manager, though the former may be better for a visual aid - plus it comes with docker-compose.

Once this is done, go to the directory and type docker-compose build. You should see things happening on the terminal! This command builds the image we need to run the VHP container.

➜  vulnerability-history git:(system_tests_docker) ✗ docker-compose build
...
Successfully built 988dcf0b62ce
Successfully tagged vulnerability-history_app:latest

Once this command finishes up, it's time to pull from docker hub. Run docker-compose up in the same container. This pulls all other necessary images and starts them up.

Your next destination will be to localhost:2345, where our pgAdmin container is exposed. Here, you'll give the login credentials supplied in your docker-compose.yml file to log in to both pgAdmin and the server itself. In the future, these may be refactored into local environment variables, but for now, you can edit them in the file itself under the environment section of pgadmin and postgres.

Once logged in, you need to create two databases, vhp and vhp_test, and their respective users in the server, both with passwords supplied in your secrets.yml file, and the permission Can Log In.

The last thing to do is kill the containers with ^C and run two commands to build the database:

  docker-compose run --rm app rails db:schema:load
  docker-compose run --rm app rails data:clear data:all

These will spin up a temporary container to build the database. Once these processes are done, do one more docker-compose up. This time if you visit localhost:3000, there should be a VHP site waiting for you!

Now I'll go over each of the containers as they appear in docker-compose.yml.

• redis

  Redis helps with the server and db - don't put too much thought into it.

• postgres

  This is the container for our database. It uses the latest version released on dockerhub, which may cause issues in the future.

• pgadmin

  I like using pgAdmin to have a better visual representation of the database running. The instructions for setting up the database are the same as listed in the normal configuration. I set it to run on port 2345.

• app

  This is the main rails server running on port 3000.

• webpacker

These will spin up a temporary container to build the database. Once these processes are done, do one more docker-compose up. This time if you visit localhost:3000, there should be a VHP site waiting for you!

Now I'll go over each of the containers as they appear in docker-compose.yml.

• redis

  Redis helps with the server and db - don't put too much thought into it.

• postgres

  This is the container for our database. It uses the latest version released on dockerhub, which may cause issues in the future.

• pgadmin

  I like using pgAdmin to have a better visual representation of the database running. The instructions for setting up the database are the same as listed in the normal configuration. I set it to run on port 2345.

• app

  This is the main rails server running on port 3000.

• webpacker

  You can comment this one out if you want, but this is better for web development, so I prefer to leave it in. This runs ruby bin/webpack-dev-server and communicates with the VHP container. The only downside is that it adds some time to the startup process.

• selenium

  This is only useful for system testing, and you can comment this service out if you won't be doing any of that. More about system tests are in the Testing section.

• test

  The testing container does not continuously run after startup. It only runs when the user wants to run tests, which will be covered in the testing section.

Testing

Regular tests can be run by running docker-compose run --rm test <command>

This tells docker-compose to start up a test container that removes itself whenever done testing. The test command can be run just like normal after that.

System testing requires one extra tool. Any VNC client will do, but I use VNC Viewer. The system tests will run fine, however to actually see the browser (if you care), you'll need to be able to get into the VNC server running on the selenium container. The VNC server itself runs on port 5900, so you'll need to point the client to 0.0.0.0:5900. Now you can observe the browser during the system tests.

Running Commands

You've already seen an example of how to run commands inside a container. The preferred way of doing this is by using docker-compose run --rm <container_name> <command>. This spins up the container of your choice to run any command you supply it. It's like a one-time terminal command. The --rm flag is not required, but it's useful for not keeping around any containers that would be otherwise using memory.

If you prefer a more interactive shell, you can always connect to the container itself, either through Docker Desktop, or by connecting through your terminal.

Reference

This was pretty tricky to get working, if you care to know how to do this here are some articles that really helped me out.

Rails Development with Docker

Running a Rails App with Webpacker and Docker

Testing Rails Apps in Docker Containers

Using Rails 5.1 System Tests with Docker

How to run a Rails 6 application using Docker Compose

Finding Useful Commands

Having trouble installing pg gem?

As of January 2023, we had some new issues with installing the pg gem on Windows. Fortunately, the error message has recently been updated to direct you in what to do:

• Note: you'll need to run as administrator this command: ridk exec sh -c "pacman -S ${MINGW_PACKAGE_PREFIX}-postgresql", then do gem install pg
• Alternatively, you can do gem install pg -- --with-pg-config=<path to pg_config> (don't forget those middle --). We've less success with this though.
• For posterity: ged/ruby-pg#365 was helpful

Be sure to check out the Ruby pg README and their issues for anything that might come up: https://github.com/ged/ruby-pg

Powershell not letting you run scripts?

This gives a good explanation: https://tecadmin.net/powershell-running-scripts-is-disabled-system/

One solution to that is to do your administrative installs using CMD instead of Powershell.

Finding useful commands

To find out useful commands for working with data:

rails -T

Rebuild the Database

---

If db/schema.rb has changed recently, you'll need to rebuild the database:

rails db:schema:load

Reload the Production Data

---

To load the production data from Chromium and HTTPD:

rails data:all

Testing out merging? This is helpful:

rails data:dev_all

This will set the environment variable VHP_DATA_BRANCH to dev, which means that vulnerabilities and vhp-mining will be downloaded using their dev branch.

Testing VHP

---

System Tests

System tests are actively being worked on and always improving. That being said, it is worth explaining how we do system tests at the moment.

As of Rails 5.1, system tests are a part of the framework itself using Capybara for testing. These system tests are meant to simulate user actions and navigation through the site to make sure everything performs at it should. These tests can be found in the test/system folder.

Because of some behavior issues with AJAX on our tests, we perform them against the running server as opposed to one spun up by Capybara. This means that to run system tests on your own, there should be a local server running for Capybara to find.

Useful Commands

---

  rails server
  rails console

Helpful Dev Tools

---

Here's a list of our favorite tools for developing this:

• Atom or VS Code
• Git Extensions for Windows Git clients
• JSONView to format JSON responses in your browser

The CWE csv File

---

To download a copy of the CWE csv file, go to https://cwe.mitre.org/data/downloads.html and download the latest Research Concepts csv listed under 'Navigate CWE'.

Place the csv file in lib/taggers/resources/cwes.csv

Note: this is incomplete, and some CWEs are not included in this that we need. Running a data rebuild will tell you which ones we're missing. In that case, get the info manually and put them into the manual_cwes.csv file.

On line 677 of the file, near the end of the line, change the double quotes around "<script" to single quotes 'script'

IntelliJ Setup

---

In case you didn't know, if you are a student with an .edu email you can get the professional version of all JetBrains IDEs. You can read more about this here

With this in mind, I will be proceeding with project setup instructions for the IntelliJ Ultimate Edition. With the correct plugins installed, this IDE can function the same as all of JetBrains other language specific IDEs, such as PyCharm, WebStorm, and RubyMine.

Plugins

The first step to running VHP within IntelliJ is installing a Ruby plugin. The plugin menu can be found at Preferences -> Plugins. Within the Marketplace tab, search for Ruby by JetBrains s.r.o. Once this plugin is installed, you may or may not be required to restart your IDE.

Project Structure

With the Ruby plugin installed, you can now configure your project. Something to be aware of is that IntelliJ might have autogenerated an SDK, Module, and Run Configuration. It is entirely possible that these function correctly and you do not need to follow the steps outlined here. In my case, they were not autogenerated as I initially worked on VHP within IntelliJ. However, once I got IntelliJ Ultimate they were autogenerated. So keep this in mind!

SDK

• Navigate to File -> Project Structure. You'll first need to add your Ruby version as an SDK. This can be done by selecting SDKs under Platform Settings. If you don't already see a Ruby 3.0+ version listed within the sdk list, you'll have to add it.
• Select the + icon in the middle column and select Add Ruby SDK.... You should see a Ruby version popup at the top of the list with its location on your computer, something similar to /usr/local/opt/ruby/bin/ruby. If this does not pop up, you'll have to navigate and find the specific Ruby 3.0+ version on your system by selecting New local...
• Once the SDK has been added, ensure that the Language level row has a value above 3.0. This corresponds to what version of Ruby you are running VHP with.
• With the SDK added, you can now configure the project to use it. While still in Project Structure navigate to the Project tab under Project Settings. Once here, simply select the newly added Ruby sdk from the Project SDK: dropdown menu. Once you correctly identify the Ruby 3.x SDK, select Apply on the bottom right and then OK to see your changes take effect.
• IntelliJ may take a few minutes to index the newly added SDK so allow it to do so.

Modules

• While still in File -> Project Structure, navigate to the Modules tab under Project Settings. Select the + icon in the middle column and select New Module from the dropdown menu.
• Select Ruby in the left column and then in the Module SDK option choose the Project SDK option if it is present. This should be the Ruby version you set up in the previous step. If it is not present then select the Ruby version that corresponds with this project, most likely some SDK option with the name ruby-3.x
• Select Next in the bottom right. Name the Module whatever you prefer and then select Finish
• Now that you're back in the Modules tab, you should now see the Module you just created in the left most column. Select the newly created Module and then select the Ruby SDK and Gems tab in the middle column. Ensure that the correct 3.x version of Ruby is currently selected. You should see every gem that is in Gemfile.lock in the right most column.
• If this all looks correctly, select Apply and OK to save this Module.

Run Configuration

Rails Server

• Navigate to Run -> Edit Configurations... and select the + icon in the upper left and select Rails
• The following information is mirrored off of what occurs when you run rails server within the terminal. Enter the following:
  • IP address: 127.0.0.1
  • Port: 3000
  • Ruby SDK -> Choose SDK from module: this should be the module we created earlier
• Select Apply and OK. In my process of building these instructions, I have run into 2 of the following scenarios when I attempt to run this configuration.
  • It runs fine and the VHP application is at localhost:3000
  • It does NOT run and some error similar to Rails server launcher wasn't found in project is found. This error is what happened to me the first time I made this configuration. I did find a fix for it, but it seems hacky and takes 20-60 minutes to finish. If you do find a better solution than what follows, please make a PR and update these instructions
    • You will need to remove the autogenerated .idea folder within this project, while IntelliJ is CLOSED. Since this directory starts with a ., it might not show up within File Explorer for Windows or Finder for Mac. A unix command for deleting everything within the .idea directory is rm -rf .idea. However, MAKE SURE you are in the VHP directory. This command will recursively delete everything within the .idea folder for this project and essentially force IntelliJ to re-index everything within VHP.
    • Once .idea is deleted, open IntelliJ again. This is the long part as IntelliJ, at least on my system, will take a considerable amount of time to re-index everything. Once it is finished, the run configuration should now work!

Rake

• Navigate to Run -> Edit Configurations... and select the + icon in the upper left and select Rake
• The only thing that needs to be changed, unless the SDK is not properly configured, is the Task Name: field located at the top of the Configuration tab.
• To see all the valid inputs you can put here, enter rake --tasks within a Terminal window. This should display a list of commands, with their purpose, similar to rake data:all # Clean rebuild of ALL data: Chromium, HTTPD / Build all *-vulnerabilities data
• Once you select the command you want to execute, put the data after rake into the Task Name: field. In this scenario, the inputted data would be data:all
• Select Apply and OK. This allows you to run and use the beautiful IntelliJ debugger with any of the tasks listed. Make sure to set a breakpoint!

Updating the OpenAPI Spec

We're using the OpenAPI specification for defining our RESTFul API. This allows us to be put in various registries, as generate some nice docs for us, and some other potential good things.

Our specification is in public/openapi.yml, making it publicly available. So don't put any secrets in there!

We're using Redocly tools to generate our docs and do linting on our spec.

To install Redocly globally, run npm install --global @redocly/cli

• yarn openapi, which runs redocly preview-docs public/openapi.yml that generates a preview of the Redocly OpenAPI documentation.
• redocly lint will check our openapi.yml file and find problems.
• redocly lint --generate-ignore-file if we're happy with all the warnings and errors, we can generate an ignore file if we want

Useful tools

OpenAPI is complicated. These guides and tools will help:

• Rails Guides for Testing, specifically integration testing
• Swagger Definition Objects Generator was a huge time saver because it infers the existing API response schema for you
• Minivore is the gem we use to use our integration tests to test against our OpenAPI spec
• Swagger.io
• inspector.swagger.io can be somewhat useful

The Rails Doctrine

Recommended reading: https://rubyonrails.org/doctrine. In particular these two sections are important to VHP:

• Optimize for Programmer Happiness
• Convention Over Configuration
• Omakase