Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Archive states & prune refactor #513

Merged
merged 36 commits into from
Jun 1, 2018
Merged
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
601e905
refactored JournalPruneDataSource to be independent of the block and …
AlexandraRoatis May 11, 2018
e92d842
replacing synchronized methods with read-write locks
AlexandraRoatis May 14, 2018
0fb29b9
disabling locking on the state database since the JournalPruneDS is l…
AlexandraRoatis May 14, 2018
8aa1133
moving the JournalPruneDS test class in the correct folder
AlexandraRoatis May 14, 2018
ca0c44b
unit tests for conformance to interface
AlexandraRoatis May 14, 2018
d10df63
concurrency tests for JournalPruneDS
AlexandraRoatis May 14, 2018
3205e07
unit tests for pruning functionality
AlexandraRoatis May 14, 2018
0f78cc8
removed dependency on modDbImpl
AlexandraRoatis May 14, 2018
527f382
applying review comments
AlexandraRoatis May 17, 2018
606b1e3
bugfix: incorrect check for presence in the data source
AlexandraRoatis May 22, 2018
d7c1f58
new configuration class and interface for pruning
AlexandraRoatis May 15, 2018
e319850
using new detailed pruning configuration
AlexandraRoatis May 15, 2018
cf6e9d5
bugfix: method parameters should be generic
AlexandraRoatis May 15, 2018
e40b699
new data source class that deletes from the current db only keys not …
AlexandraRoatis May 18, 2018
471614e
refactored journal prune class to take a data store as argument; meth…
AlexandraRoatis May 18, 2018
9c5b552
building the journal prune on top of the data source with archive
AlexandraRoatis May 18, 2018
8e487b5
archiving the world state at runtime
AlexandraRoatis May 18, 2018
631da9e
bugfix: incorrect variable update
AlexandraRoatis May 18, 2018
2e89e3f
bugfix: reseting the root back to its initial value after recovery
AlexandraRoatis May 23, 2018
f8242db
optimizing the sync BACKWARD and FORWARD behavior
AlexandraRoatis May 23, 2018
cf03e54
fixing config file
AlexandraRoatis May 23, 2018
47da914
bugfix: printing the size of the trie states should only consider the…
AlexandraRoatis May 24, 2018
b42dd50
additional stop condition for sync FORWARD mode
AlexandraRoatis May 24, 2018
27a7a52
new pruning configurations and default values
AlexandraRoatis May 24, 2018
ada9502
correcting broken test cases
AlexandraRoatis May 25, 2018
0e4880a
extracting the buildGenesis functionality to a utils class
AlexandraRoatis May 25, 2018
188a483
state pruning as CLI options
AlexandraRoatis May 25, 2018
d87ac87
skipping full batch when in FORWARD mode and last block satisfies req…
AlexandraRoatis May 25, 2018
d415e4a
checking for prune restrictions before importing blocks
AlexandraRoatis May 25, 2018
eeb8627
moved the prune restriction to block filtering
AlexandraRoatis May 28, 2018
8a1160b
bugfix: checking for null keys, values and batches
AlexandraRoatis May 28, 2018
51edfc4
backward sync step as constant
AlexandraRoatis May 29, 2018
9e2d789
extracting repeated code into methods in TaskImportBlocks
AlexandraRoatis May 29, 2018
bff2e1c
prune parameters as constants
AlexandraRoatis May 29, 2018
e0ca459
bugfix: reducing the heap space requirements for the state recovery
AlexandraRoatis May 30, 2018
ab6d339
Merge branch 'master-pre-merge' into db-archive-states
AlexandraRoatis Jun 1, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions modAionBase/src/org/aion/base/db/IKeyValueStore.java
Original file line number Diff line number Diff line change
@@ -147,4 +147,13 @@ public interface IKeyValueStore<K, V> extends AutoCloseable {
* @throws IllegalArgumentException if the collection contains a null key
*/
void deleteBatch(Collection<K> keys);

/**
* Checks that the data store connection is open. Throws a {@link RuntimeException} if the data
* store connection is closed.
*
* @implNote Always do this check after acquiring a lock on the class/data. Otherwise it might
* produce inconsistent results due to lack of synchronization.
*/
void check();
}
12 changes: 3 additions & 9 deletions modDbImpl/src/org/aion/db/generic/DatabaseWithCache.java
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/*******************************************************************************
/* ******************************************************************************
* Copyright (c) 2017-2018 Aion foundation.
*
* This file is part of the aion network project.
@@ -163,14 +163,8 @@ public CacheStats getStats() {
return this.loadingCache.stats();
}

/**
* Checks that the database connection is open.
* Throws a {@link RuntimeException} if the database connection is closed.
*
* @implNote Always do this check after acquiring a lock on the class/data.
* Otherwise it might produce inconsistent results due to lack of synchronization.
*/
private void check() {
@Override
public void check() {
if (!database.isOpen()) {
throw new RuntimeException("Database is not opened: " + this);
}
13 changes: 13 additions & 0 deletions modDbImpl/src/org/aion/db/generic/LockedDatabase.java
Original file line number Diff line number Diff line change
@@ -366,6 +366,19 @@ public void deleteBatch(Collection<byte[]> keys) {
}
}

@Override
public void check() {
// acquire read lock
lock.readLock().lock();

try {
database.check();
} finally {
// releasing read lock
lock.readLock().unlock();
}
}

@Override
public void drop() {
// acquire write lock
9 changes: 9 additions & 0 deletions modDbImpl/src/org/aion/db/generic/TimedDatabase.java
Original file line number Diff line number Diff line change
@@ -252,6 +252,15 @@ public void deleteBatch(Collection<byte[]> keys) {
LOG.debug(database.toString() + " deleteBatch(" + keys.size() + ") in " + (t2 - t1) + " ns.");
}

@Override
public void check() {
long t1 = System.nanoTime();
database.check();
long t2 = System.nanoTime();

LOG.debug(database.toString() + " check() in " + (t2 - t1) + " ns.");
}

@Override
public void drop() {
long t1 = System.nanoTime();
74 changes: 36 additions & 38 deletions modDbImpl/src/org/aion/db/impl/AbstractDB.java
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/*******************************************************************************
/* ******************************************************************************
* Copyright (c) 2017-2018 Aion foundation.
*
* This file is part of the aion network project.
@@ -34,18 +34,16 @@
******************************************************************************/
package org.aion.db.impl;

import org.aion.base.db.IByteArrayKeyValueDatabase;
import org.aion.base.util.ByteArrayWrapper;
import org.aion.log.AionLoggerFactory;
import org.aion.log.LogEnum;
import org.h2.store.fs.FileUtils;
import org.slf4j.Logger;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.*;
import java.util.stream.Stream;
import org.aion.base.db.IByteArrayKeyValueDatabase;
import org.aion.base.util.ByteArrayWrapper;
import org.aion.log.AionLoggerFactory;
import org.aion.log.LogEnum;
import org.slf4j.Logger;

/**
* Common functionality for database implementations.
@@ -70,7 +68,8 @@ protected AbstractDB(String name) {
this.name = name;
}

protected AbstractDB(String name, String path, boolean enableDbCache, boolean enableDbCompression) {
protected AbstractDB(
String name, String path, boolean enableDbCache, boolean enableDbCompression) {
this(name);

Objects.requireNonNull(path, "The database path cannot be null.");
@@ -81,14 +80,22 @@ protected AbstractDB(String name, String path, boolean enableDbCache, boolean en
}

protected String propertiesInfo() {
return "<name=" + name + ",autocommit=ON,cache=" + (enableDbCache ? "ON" : "OFF") + //
",compression=" + (enableDbCompression ? "ON" : "OFF") + ">"; //
return "<name="
+ name
+ ",autocommit=ON,cache="
+ (enableDbCache ? "ON" : "OFF")
+ //
",compression="
+ (enableDbCompression ? "ON" : "OFF")
+ ">"; //
}

@Override
public boolean commit() {
// not implemented since we always commit the changes to the database for this implementation
throw new UnsupportedOperationException("Only automatic commits are supported by " + this.toString());
// not implemented since we always commit the changes to the database for this
// implementation
throw new UnsupportedOperationException(
"Only automatic commits are supported by " + this.toString());
}

@Override
@@ -119,22 +126,16 @@ public Optional<String> getPath() {
return Optional.ofNullable(this.path);
}

/**
* Checks that the database connection is open.
* Throws a {@link RuntimeException} if the database connection is closed.
*
* @implNote Always do this check after acquiring a lock on the class/data.
* Otherwise it might produce inconsistent results due to lack of synchronization.
*/
protected void check() {
@Override
public void check() {
if (!isOpen()) {
throw new RuntimeException("Database is not opened: " + this);
}
}

/**
* Checks that the given key is not null.
* Throws a {@link IllegalArgumentException} if the key is null.
* Checks that the given key is not null. Throws a {@link IllegalArgumentException} if the key
* is null.
*/
public static void check(byte[] k) {
if (k == null) {
@@ -143,8 +144,8 @@ public static void check(byte[] k) {
}

/**
* Checks that the given collection of keys does not contain null values.
* Throws a {@link IllegalArgumentException} if a null key is present.
* Checks that the given collection of keys does not contain null values. Throws a {@link
* IllegalArgumentException} if a null key is present.
*/
public static void check(Collection<byte[]> keys) {
if (keys.contains(null)) {
@@ -170,23 +171,21 @@ public boolean isPersistent() {
}

/**
* For testing the lock functionality of public methods.
* Helps ensure that locks are released after normal or exceptional execution.
* For testing the lock functionality of public methods. Helps ensure that locks are released
* after normal or exceptional execution.
*
* @return {@code true} when the resource is locked,
* {@code false} otherwise
* @return {@code true} when the resource is locked, {@code false} otherwise
*/
@Override
public boolean isLocked() {
return false;
}

/**
* Functionality for directly interacting with the heap cache.
*/
/** Functionality for directly interacting with the heap cache. */
public abstract boolean commitCache(Map<ByteArrayWrapper, byte[]> cache);

// IKeyValueStore functionality ------------------------------------------------------------------------------------
// IKeyValueStore functionality
// ------------------------------------------------------------------------------------

@Override
public Optional<byte[]> get(byte[] k) {
@@ -200,12 +199,11 @@ public Optional<byte[]> get(byte[] k) {
}

/**
* Database specific get functionality, without locking required. Locking is applied in {@link #get(byte[])}.
* Database specific get functionality, without locking required. Locking is applied in {@link
* #get(byte[])}.
*
* @param k
* the key for which the method must return the associated value
* @param k the key for which the method must return the associated value
* @return the value stored in the database for the give key.
*/
protected abstract byte[] getInternal(byte[] k);

}
}
6 changes: 6 additions & 0 deletions modMcf/src/org/aion/mcf/ds/ArchivedDataSource.java
Original file line number Diff line number Diff line change
@@ -117,6 +117,12 @@ public void deleteBatch(Collection<byte[]> keys) {
commitBatch();
}

@Override
public void check() {
data.check();
archive.check();
}

@Override
public void close() {
data.close();
16 changes: 10 additions & 6 deletions modMcf/src/org/aion/mcf/ds/XorDataSource.java
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/*******************************************************************************
/* ******************************************************************************
* Copyright (c) 2017-2018 Aion foundation.
*
* This file is part of the aion network project.
@@ -34,13 +34,11 @@
******************************************************************************/
package org.aion.mcf.ds;

import org.aion.base.db.IByteArrayKeyValueDatabase;
import java.util.*;
import org.aion.base.db.IByteArrayKeyValueStore;
import org.aion.base.util.ByteArrayWrapper;
import org.aion.base.util.ByteUtil;

import java.util.*;

public class XorDataSource implements IByteArrayKeyValueStore {
IByteArrayKeyValueStore source;
byte[] subKey;
@@ -89,8 +87,9 @@ public void putBatch(Map<byte[], byte[]> rows) {
}

public void updateBatch(Map<ByteArrayWrapper, byte[]> rows, boolean erasure) {
//not supported
throw new UnsupportedOperationException("ByteArrayWrapper map not supported in XorDataSource.updateBatch yet");
// not supported
throw new UnsupportedOperationException(
"ByteArrayWrapper map not supported in XorDataSource.updateBatch yet");
}

@Override
@@ -104,6 +103,11 @@ public void deleteBatch(Collection<byte[]> keys) {

}

@Override
public void check() {
source.check();
}

@Override
public boolean isEmpty() {
// TODO Auto-generated method stub
33 changes: 13 additions & 20 deletions modMcf/src/org/aion/mcf/trie/JournalPruneDataSource.java
Original file line number Diff line number Diff line change
@@ -38,7 +38,6 @@
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import org.aion.base.db.IByteArrayKeyValueDatabase;
import org.aion.base.db.IByteArrayKeyValueStore;
import org.aion.base.util.ByteArrayWrapper;
import org.aion.log.AionLoggerFactory;
@@ -85,13 +84,13 @@ public String toString() {

Map<ByteArrayWrapper, Ref> refCount = new HashMap<>();

private IByteArrayKeyValueDatabase src;
private IByteArrayKeyValueStore src;
// block hash => updates
private LinkedHashMap<ByteArrayWrapper, Updates> blockUpdates = new LinkedHashMap<>();
private Updates currentUpdates = new Updates();
private AtomicBoolean enabled = new AtomicBoolean(true);

public JournalPruneDataSource(IByteArrayKeyValueDatabase src) {
public JournalPruneDataSource(IByteArrayKeyValueStore src) {
this.src = src;
}

@@ -119,7 +118,7 @@ public void put(byte[] key, byte[] value) {
src.put(key, value);

} else {
checkOpen();
check();

// Value does not exist, so we delete from current updates
currentUpdates.deletedKeys.add(keyW);
@@ -129,7 +128,7 @@ public void put(byte[] key, byte[] value) {
if (value != null) {
src.put(key, value);
} else {
checkOpen();
check();
}
}
} catch (Exception e) {
@@ -152,7 +151,7 @@ public void delete(byte[] key) {
lock.writeLock().lock();

try {
checkOpen();
check();

currentUpdates.deletedKeys.add(ByteArrayWrapper.wrap(key));
// delete is delayed
@@ -350,6 +349,8 @@ public void close() {

try {
src.close();
} catch (Exception e) {
LOG.error("Could not close source due to ", e);
} finally {
lock.writeLock().unlock();
}
@@ -375,7 +376,7 @@ public void deleteBatch(Collection<byte[]> keys) {
lock.writeLock().lock();

try {
checkOpen();
check();

// deletes are delayed
keys.forEach(key -> currentUpdates.deletedKeys.add(ByteArrayWrapper.wrap(key)));
@@ -397,7 +398,7 @@ public boolean isEmpty() {
try {
// the delayed deletes are not considered by this check until applied to the db
if (!currentUpdates.insertedKeys.isEmpty()) {
checkOpen();
check();
return false;
} else {
return src.isEmpty();
@@ -410,21 +411,13 @@ public boolean isEmpty() {
}
}

public IByteArrayKeyValueDatabase getSrc() {
public IByteArrayKeyValueStore getSrc() {
return src;
}

/**
* Checks that the data store connection is open. Throws a {@link RuntimeException} if the data
* store connection is closed.
*
* @implNote Always do this check after acquiring a lock on the class/data. Otherwise it might
* produce inconsistent results due to lack of synchronization.
*/
protected void checkOpen() {
if (!src.isOpen()) {
throw new RuntimeException("Data store is not opened: " + src);
}
@Override
public void check() {
src.check();
}

/**