10x-100x Compilation speed boost

The core macro transformation engine has been re-written from the ground up and micro-optimizations applied wherever possible. This has resulted in dramatic compilation speed improvements, some several orders of magnitude!

Macro materialization of molecules earlier produced a lot of code that has now been moved out to static methods. An absolute minimal amount of code is now generated, thus minimizing the job of the macros and the compiler. As an example, the Seattle tests file sometimes took up to 70 seconds to compile and now average around 4 seconds. Some long molecules with close to 22 attributes almost never finished compiling but now take about 2 seconds to compile! This is good news since users of Molecule can therefore now freely create as large molecules as they please without any speed penalty.

5 optimized getter groups

Type casting of returned data from Datomic was earlier not completely optimized. Taking advice from
Haoyi's "Benchmarking Scala Collections"
Molecule now also returns super fast mutable pre-allocated Arrays of typed data for large data sets.

Json has also been thoroughly optimized to build as fast as possible directly from raw Datomic data.

So Molecule now offers 5 optimized getter groups:

• get - Default List of typed tuples for convenient access to smaller data sets.
• getArray - Array of typed tuples for fast retrieval and traversing of large data sets.
• getIterable - Iterable of typed tuples for lazy evaluation of data
• getJson - Json formatted result data
• getRaw - Raw untyped data from Datomic

Each getter group comes with all time-related variations:

• get
• getAsOf(t)
• getSince(t)
• getWith(txData)
• getHistory (only implemented for List getter)

Scala docs and semantic updates

Major overhaul of Molecule:

Thorough Scala docs API documentation

All relevant public interfaces have been documented in the new Scala docs. Shortcuts to Scala docs sub packages and documents are also directly available via "API docs" in the menu on the Molecule website.

To aid constructing molecules in your code, all attributes defined now also have Scala docs automatically defined by the sbt-molecule plugin upon compilation.

Input molecules correctly implemented

Input molecules are now semantically correctly implemented and thoroughly tested (1 input, 2 inputs, 3 inputs).

Interfaces updated and streamlined

The standard getters return Lists of tuples of type-casted tuples.

• get
• getAsOf(t)
• getSince(t)
• getHistory
• getWith(txData)

If large data sets are expected, an Iterable of tuples of lazily type-cased tuples can be retrieved instead.
Data is type-casted on each call to next on the iterator.

• getIterable
• getIterableAsOf(t)
• getIterableSince(t)
• getIterableHistory
• getIterableWith(txData)

If typed data is not required we can get the raw untyped java collections of Lists of objects.

• getRaw
• getRawAsOf(t)
• getRawSince(t)
• getRawHistory
• getRawWith(txData)

Breaking changes

The whole directory layout of the Molecule library has been re-arranged and optimized, including many interfaces. So you might have to change some method names if you have used earlier versions of Molecule.

Bug fixes

v0.13.2

0.13.2 various bug-fixes

Native Json output

We can now get data in json format directly from the database by calling getJson on a molecule. So instead of converting tuples of data to json with some 3rd party library we can call getJson and pass the json data string directly to an Angular table for instance.

Internally, Molecule builds the json string in a StringBuffer directly from the raw data coming from Datomic
(with regards to types being quoted or not). This should make it the fastest way of supplying json data when needed.

Flat data

Normal "flat" molecules creates json with a row on each line in the output:

Person.name.age.getJson ===
  """[
    |{"name": "Fred", "age": 38},
    |{"name": "Lisa", "age": 35}
    |]""".stripMargin

Composite data

Composite data has potential field name clashes so each sub part of the composite is rendered
as a separate json object tied together in an array for each row:

m(Person.name.age ~ Category.name.importance).getJson ===
  """[
    |[{"name": "Fred", "age": 38}, {"name": "Marketing", "importance": 6}],
    |[{"name": "Lisa", "age": 35}, {"name": "Management", "importance": 7}]
    |]""".stripMargin

Note how a field name appears in each sub object. Since the molecule is defined in client code it is presumed that
the semantics of eventual duplicate field names are also handled by client code.

Nested data

Nested date is rendered as a json array with json objects for each nested row:

(Invoice.no.customer.InvoiceLines * InvoiceLine.item.qty.amount).getJson ===
  """[
    |{"no": 1, "customer": "Johnson", "invoiceLines": [
    |   {"item": "apples", "qty": 10, "amount": 12.0},
    |   {"item": "oranges", "qty": 7, "amount": 3.5}]},
    |{"no": 2, "customer": "Benson", "invoiceLines": [
    |   {"item": "bananas", "qty": 3, "amount": 3.0},
    |   {"item": "oranges", "qty": 1, "amount": 0.5}]}
    |]""".stripMargin

Optional values in save-molecules

Often, form submissions have some optional field values. Molecule now allow us to save molecules with both mandatory and optional attributes.

We could for instance have aName, optionalLikes and anAge values from a form submission that we want to save. We can now apply those values directly to a mandatory name attribute, an optional likes$ attribute ($ appended makes it optional) and a mandatory age attribute of a save-molecule and then save it:

Person
  .name(aName)
  .likes$(optionalLikes)
  .age(anAge)
  .save

We can also, as before, insert the data using an "insert-molecule" as a template:

Person.name.likes$.age.insert(
  aName, optionalLikes, anAge
)

It can be a matter of taste if you want to save or insert - but now you can choose :-)

Time API

Datomic has some extremely powerful time functionality that Molecule now makes available in an intuitive way:

Ad-hoc time queries

Ad-hoc time queries against our database can now be made with the following time-aware getter methods on a molecule:

Person.name.age.getAsOf(t) === ... // Persons as of a point in time `t`

Person.name.age.getSince(t) === ... // Persons added after a point in time `t`

Person(fredId).age.getHistory === ... // Current and previous ages of Fred in the db

t can be a transaction entity id (Long), a transaction number (Long) or a java.util.Date.

If we want to test the outcome of some transaction without affecting the production db we can apply transaction data to the getWith(txData) method:

Person.name.age.getWith(
  // Testing adding some transactional data to the current db
  Person.name("John").age(42).saveTx, // Save transaction data
  Person.name.age.insertTx(
    List(("Lisa", 34), ("Pete", 55)) // Insert transaction data
  ),
  Person(fredId).age(28).updateTx, // Update transaction data
  someEntityId.retractTx // Retraction transaction data
) === ... // Persons from db including transactions tested 

By adding the Tx suffix to the standard molecule commands (save, insert, update or retract)
we can get the transactional data that those operations would normally transact directly. Here, <command>Tx methods on transaction molecules just return the transaction data so that we can apply it to the getWith(txData) method. This make it convenient for us to ask speculative questions like "what would I get if I did those transactions".

For more complex test scenarios we can now use a test database:

Test db

All molecules expect an implicit connection object to be in scope. If we then set a temporary test database on such conn object we can subsequentially freely perform tests against this temporary database as though it was a "branch" (think git).

When the connection/db goes out of scope it is simply garbage collected automatically by the JVM. At any point we can also explicitly go back to continuing using our live production db.

To make a few tests with a "branch" of our live db we can now do like this:

// Current state
Person(fredId).name.age.get.head === ("Fred", 27)

// Create "branch" of our production db as it is right now
conn.testDbAsOfNow  

// Perform multiple operations on test db
Person(fredId).name("Frederik").update
Person(fredId).age(28).update

// Verify expected outcome of operations
Person(fredId).name.age.get.head === ("Frederik", 28)

// Then go back to production state
conn.useLiveDb

// Production state is unchanged!
Person(fredId).name.age.get.head === ("Fred", 27)

Test db with domain classes

When molecules are used inside domain classes we want to test the domain operations also without affecting the state of our production database. And also ideally without having to create mockups of our domain objects. This is now possible by setting a temporary test database on the implicit connection object that all molecules expect to be present in their scope - which includes the molecules inside domain classes.

When we test against a temporary database, Molecule internally uses the with function of Datomic to apply transaction data to a "branch" of the database that is simply garbage collected when it goes out of scope!

To make a few tests on a domain object that have molecule calls internally we can now do like this:

// Some domain object that we want to test
val domainObj = MyDomainClass(params..) // having molecule transactions inside...
domainObj.myState === "some state"

// Create "branch" of our production db as it is right now
conn.testDbAsOfNow  

// Test some domain object operations
domainObj.doThis
domainObj.doThat

// Verify expected outcome of operations
domainObj.myState === "some expected changed state"

// Then go back to production state
conn.useLiveDb

// Production state is unchanged!
domainObj.myState == "some state"

Since internal domain methods will in turn call other domain methods that also expects an implicit conn object then the same test db is even propragated recursively inside the chain of domain operations.

Multiple time views

We can apply the above approach with several time views of our database:

conn.testDbAsOfNow
conn.testDbAsOf(t)
conn.testDbSince(t)
conn.testWith(txData)

This make it possible to run arbitrarily complex test scenarios directly against our production data at any point in time without having to do any manual setup or tear-down of mock domain/database objects!

Obsolete `maybe` interfaces removed

We only need the notation for optional values attr$:

val nameAndOptionalAge: List[(String, Option[Int])] = Person.name.age$.get

(maybe was using the entity api, whereas attr$ uses pull in queries)

Update to Scala 2.12.1

Updating to using SBT 0.13.13 and Scala 2.12.1

v0.10.0 Entity selection retrieval and manipulation

Molecule now allows retrieving attribute values of selected entities:

val List(e1, e2, e3) = Ns.int.insert(1, 2, 3).eids

// Use selected entity ids to access attributes of those entities
Ns(e1, e2).int.get === List(1, 2)

// Or use a variable with a collection of entity ids
val e23 = Seq(e2, e3)
Ns(e23).int.get === List(2, 3)

Likewise we can update attribute values of selected entities (group editing):

val List(a, b, c) = Ns.str.int insert List(("a", 1), ("b", 2), ("c", 3)) eids

// Apply value to attribute of multiple entities
Ns(a, b).int(4).update
Ns.str.int.get.sorted === List(("a", 4), ("b", 4), ("c", 3))

See more examples here and here.

Datomic encourages multi-step queries where you find some entities ids with one query and then pass those ids on as input to the following query. Since we don't have the cost of round-trips to a database server, this is a powerful technique that Molecule now supports with ease.

v0.9.0 Bidirectional references

Major upgrade of Molecule introducing Bidirectional references.

Normal Datomic references are unidirectional. If we add a friend reference from Ann to Ben

Person.name("Ann").Friends.name("Ben").save

Then we can naturally query to get friends of Ann

Person.name_("Ann").Friends.name.get === List("Ben")

But what if we want to find friends of Ben? This will give us nothing:

Person.name_("Ben").Friends.name.get === List()

Instead we would have to think backwards to get the back reference

 Person.name.Friends.name_("Ben").get === List("Ann")

If we want to traverse deeper into a friendship graph we would have to query both forward and backward for each step in the graph which would quickly become a pain. With Molecules new bidirectional references we can uniformly query from both ends:

 Person.name_("Ann").Friends.name.get === List("Ben")
 Person.name_("Ben").Friends.name.get === List("Ann")

Please see Bidirectional refs for more information and the Gremlin graph examples.

• This release also adds support for BigInts and BigDecimals. Only bytes is not supported now due to the limited
  capabilities this type has in Datomic.
• Input molecules can now also include nested data structures.