Dev: 0. Requirements

This page describes the requirements for various phases of Asami since its inception. Most phases were well-defined in their timeframe and feature set, while Phase 3 was a maintenance phase where new requirements were brought in as needed.

Asami was initially built as support for the Naga rules engine, which was designed to integrate with any graph database, via protocols. Naga's original adapter was for Datomic. However, because this was a commercial product it was deemed unsuitable for the needs of the organization (ThreatGrid at Cisco Systems, which later became a part of SecureX).

Phase 1 (2016)

The initial requirements for Asami were:

• Open Source
• Source in Clojure
• In-memory storage
• A Graph Database with a subset of Datomic operations:
  • Query language using Clojure data structures
  • Pattern-based triple selection, with variable binding
  • Join operations (inner join)
  • Filtering (removal of rows, based on a condition)
  • Projection (removal of columns)
  • Query planner

This system was deployed as a web service using Clojure Ring and Jetty.

Phase 2 (early 2017)

After the initial success of Asami, new requirements were developed:

• Source in ClojureScript
• Importing arbitrary JSON

This system was integrated into the ThreatGrid dashboard to manage the application's data model.

Phase 3 - Maintenance and Expansion (2017-2019)

Now under regular use, Asami expanded to include:

• Transaction support
• Multigraph support (multiple edges between nodes)
• Aggregate queries
• New query operations: MINUS, BINDING, AND, OR, Transitive closures, Path discovery
• Improved Query planner
• Improved security
• DELETE operation
• Command Line Interface
• Arbitrary nodes for direct JSON representation (e.g. strings)

Phase 4 (2020)

A new team struggled to work with Asami due to its unique API. This led to adopting an API that was more familiar:

• Datomic API
• New data structures to encapsulate state:
  • Connections
  • Databases
  • Datoms
• Speculative transactions
• Searchable transaction history
• Upsert and Update operations (changing a field on an object)

Phase 5 (2020-2021)

While Asami continued to operate in production, this phase was started in parallel to phase 4:

• Storage abstraction layer
• Implementation of Storage using JVM NIO
  • Index selection
  • Indexing component implementation: blocks, trees, flat files
  • Significant expansion of testing
  • Object Serialization/deserialization
  • Reimplementation of in-memory operations (e.g. path discovery) in more general terms
• Expansion and testing of Connection API