Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Archive states & prune refactor #513

Merged
merged 36 commits into from
Jun 1, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
601e905
refactored JournalPruneDataSource to be independent of the block and …
AlexandraRoatis May 11, 2018
e92d842
replacing synchronized methods with read-write locks
AlexandraRoatis May 14, 2018
0fb29b9
disabling locking on the state database since the JournalPruneDS is l…
AlexandraRoatis May 14, 2018
8aa1133
moving the JournalPruneDS test class in the correct folder
AlexandraRoatis May 14, 2018
ca0c44b
unit tests for conformance to interface
AlexandraRoatis May 14, 2018
d10df63
concurrency tests for JournalPruneDS
AlexandraRoatis May 14, 2018
3205e07
unit tests for pruning functionality
AlexandraRoatis May 14, 2018
0f78cc8
removed dependency on modDbImpl
AlexandraRoatis May 14, 2018
527f382
applying review comments
AlexandraRoatis May 17, 2018
606b1e3
bugfix: incorrect check for presence in the data source
AlexandraRoatis May 22, 2018
d7c1f58
new configuration class and interface for pruning
AlexandraRoatis May 15, 2018
e319850
using new detailed pruning configuration
AlexandraRoatis May 15, 2018
cf6e9d5
bugfix: method parameters should be generic
AlexandraRoatis May 15, 2018
e40b699
new data source class that deletes from the current db only keys not …
AlexandraRoatis May 18, 2018
471614e
refactored journal prune class to take a data store as argument; meth…
AlexandraRoatis May 18, 2018
9c5b552
building the journal prune on top of the data source with archive
AlexandraRoatis May 18, 2018
8e487b5
archiving the world state at runtime
AlexandraRoatis May 18, 2018
631da9e
bugfix: incorrect variable update
AlexandraRoatis May 18, 2018
2e89e3f
bugfix: reseting the root back to its initial value after recovery
AlexandraRoatis May 23, 2018
f8242db
optimizing the sync BACKWARD and FORWARD behavior
AlexandraRoatis May 23, 2018
cf03e54
fixing config file
AlexandraRoatis May 23, 2018
47da914
bugfix: printing the size of the trie states should only consider the…
AlexandraRoatis May 24, 2018
b42dd50
additional stop condition for sync FORWARD mode
AlexandraRoatis May 24, 2018
27a7a52
new pruning configurations and default values
AlexandraRoatis May 24, 2018
ada9502
correcting broken test cases
AlexandraRoatis May 25, 2018
0e4880a
extracting the buildGenesis functionality to a utils class
AlexandraRoatis May 25, 2018
188a483
state pruning as CLI options
AlexandraRoatis May 25, 2018
d87ac87
skipping full batch when in FORWARD mode and last block satisfies req…
AlexandraRoatis May 25, 2018
d415e4a
checking for prune restrictions before importing blocks
AlexandraRoatis May 25, 2018
eeb8627
moved the prune restriction to block filtering
AlexandraRoatis May 28, 2018
8a1160b
bugfix: checking for null keys, values and batches
AlexandraRoatis May 28, 2018
51edfc4
backward sync step as constant
AlexandraRoatis May 29, 2018
9e2d789
extracting repeated code into methods in TaskImportBlocks
AlexandraRoatis May 29, 2018
bff2e1c
prune parameters as constants
AlexandraRoatis May 29, 2018
e0ca459
bugfix: reducing the heap space requirements for the state recovery
AlexandraRoatis May 30, 2018
ab6d339
Merge branch 'master-pre-merge' into db-archive-states
AlexandraRoatis Jun 1, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 56 additions & 71 deletions modAionBase/src/org/aion/base/db/IKeyValueStore.java
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/*******************************************************************************
/* ******************************************************************************
* Copyright (c) 2017-2018 Aion foundation.
*
* This file is part of the aion network project.
Expand Down Expand Up @@ -42,14 +42,11 @@
/**
* Functionality for a key-value cache allowing itemized updates.
*
* @param <K>
* the data type of the keys
* @param <V>
* the data type of the values
* @param <K> the data type of the keys
* @param <V> the data type of the values
* @author Alexandra Roatis
* @implNote For the underlying DB connection, if [isClosed() == true], then all
* function calls which are documented to throw RuntimeException, will
* throw a RuntimeException.
* @implNote For the underlying DB connection, if [isClosed() == true], then all function calls
* which are documented to throw RuntimeException, will throw a RuntimeException.
*/
public interface IKeyValueStore<K, V> extends AutoCloseable {

Expand All @@ -60,115 +57,103 @@ public interface IKeyValueStore<K, V> extends AutoCloseable {
* Returns if the DB is empty or not.
*
* @return True if number of keys > 0, false otherwise
* @throws RuntimeException
* if the data store is closed
* @throws RuntimeException if the data store is closed
*/
boolean isEmpty();

/**
* Returns the set of keys for the database.
*
* @return Set of keys
* @throws RuntimeException
* if the data store is closed
* @apiNote Returns an empty set if the database keys could not be
* retrieved.
* @throws RuntimeException if the data store is closed
* @apiNote Returns an empty set if the database keys could not be retrieved.
*/
Set<K> keys();

/**
* get retrieves a value from the database, returning an optional, it is
* fulfilled if a value was able to be retrieved from the DB, otherwise the
* optional is empty
* get retrieves a value from the database, returning an optional, it is fulfilled if a value
* was able to be retrieved from the DB, otherwise the optional is empty
*
* @param k
* @throws RuntimeException
* if the data store is closed
* @throws IllegalArgumentException
* if the key is null
* @throws RuntimeException if the data store is closed
* @throws IllegalArgumentException if the key is null
*/
Optional<V> get(K k);

// Updates for individual keys
// -------------------------------------------------------------------------------------

/**
* Places or updates a value into the cache at the corresponding key. Makes
* no guarantees about when the value is actually inserted into the
* underlying data store.
* Places or updates a value into the cache at the corresponding key. Makes no guarantees about
* when the value is actually inserted into the underlying data store.
*
* @param k
* the key for the new entry
* @param v
* the value for the new entry
* @throws RuntimeException
* if the underlying data store is closed
* @throws IllegalArgumentException
* if the key is null
* @implNote The choice of when to push the changes to the data store is
* left up to the implementation.
* @param k the key for the new entry
* @param v the value for the new entry
* @throws RuntimeException if the underlying data store is closed
* @throws IllegalArgumentException if the key is null
* @implNote The choice of when to push the changes to the data store is left up to the
* implementation.
* @apiNote Put must have the following properties:
* <ol>
* <li>Creates a new entry in the cache, if the key-value pair does
* not exist in the cache or underlying data store.</li>
* <li>Updates the entry in the cache when the key-value pair
* already exists.
* <li>Deletes the entry when given a {@code null} value.</li>
* </ol>
* <ol>
* <li>Creates a new entry in the cache, if the key-value pair does not exist in the cache
* or underlying data store.
* <li>Updates the entry in the cache when the key-value pair already exists.
* <li>Deletes the entry when given a {@code null} value.
* </ol>
*/
void put(K k, V v);

/**
* Delete an entry from the cache, marking it for deletion inside the data
* store. Makes no guarantees about when the value is actually deleted from
* the underlying data store.
* Delete an entry from the cache, marking it for deletion inside the data store. Makes no
* guarantees about when the value is actually deleted from the underlying data store.
*
* @param k
* the key of the entry to be deleted
* @throws RuntimeException
* if the underlying data store is closed
* @throws IllegalArgumentException
* if the key is null
* @implNote The choice of when to push the changes to the data store is
* left up to the implementation.
* @param k the key of the entry to be deleted
* @throws RuntimeException if the underlying data store is closed
* @throws IllegalArgumentException if the key is null
* @implNote The choice of when to push the changes to the data store is left up to the
* implementation.
*/
void delete(K k);

// Batch Updates
// ---------------------------------------------------------------------------------------------------

/**
* Puts or updates the data store with the given <i>key-value</i> pairs, as
* follows:
* Puts or updates the data store with the given <i>key-value</i> pairs, as follows:
*
* <ul>
* <li>if the <i>key</i> is present in the data store, the stored
* <i>value</i> is overwritten</li>
* <li>if the <i>key</i> is not present in the data store, the new
* <i>key-value</i> pair is stored</li>
* <li>if the <i>value</i> is null, the matching stored <i>key</i> will be
* deleted from the data store, or</li>
* <li>if the <i>key</i> is present in the data store, the stored <i>value</i> is overwritten
* <li>if the <i>key</i> is not present in the data store, the new <i>key-value</i> pair is
* stored
* <li>if the <i>value</i> is null, the matching stored <i>key</i> will be deleted from the
* data store, or
* </ul>
*
* @param inputMap
* a {@link Map} of key-value pairs to be updated in the database
* @throws RuntimeException
* if the data store is closed
* @throws IllegalArgumentException
* if the map contains a null key
* @param inputMap a {@link Map} of key-value pairs to be updated in the database
* @throws RuntimeException if the data store is closed
* @throws IllegalArgumentException if the map contains a null key
*/
void putBatch(Map<K, V> inputMap);

void putToBatch(byte[] key, byte[] value);
void putToBatch(K key, V value);

void commitBatch();

/**
* Similar to delete, except operates on a list of keys
*
* @param keys
* @throws RuntimeException
* if the data store is closed
* @throws IllegalArgumentException
* if the collection contains a null key
* @throws RuntimeException if the data store is closed
* @throws IllegalArgumentException if the collection contains a null key
*/
void deleteBatch(Collection<K> keys);

/**
* Checks that the data store connection is open. Throws a {@link RuntimeException} if the data
* store connection is closed.
*
* @implNote Always do this check after acquiring a lock on the class/data. Otherwise it might
* produce inconsistent results due to lack of synchronization.
*/
void check();
}
37 changes: 37 additions & 0 deletions modAionBase/src/org/aion/base/db/IPruneConfig.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package org.aion.base.db;

/**
* Interface for pruning configuration parameters.
*
* @author Alexandra Roatis
*/
public interface IPruneConfig {

/**
* Indicates if pruning should be enabled or disabled.
*
* @return {@code true} when pruning enabled, {@code false} when pruning disabled.
*/
boolean isEnabled();

/**
* Indicates if archiving should be enabled or disabled.
*
* @return {@code true} when archiving enabled, {@code false} when archiving disabled.
*/
boolean isArchived();

/**
* @return the number of topmost blocks for which the full data should be maintained on disk.
*/
int getCurrentCount();

/**
* Gets the rate at which blocks should be archived (for which the full data should be
* maintained on disk). Blocks that are exact multiples of the returned value should be
* persisted on disk, regardless of other pruning.
*
* @return integer value representing the archive rate
*/
int getArchiveRate();
}
12 changes: 5 additions & 7 deletions modAionBase/src/org/aion/base/db/IRepositoryConfig.java
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/*******************************************************************************
/* ******************************************************************************
* Copyright (c) 2017-2018 Aion foundation.
*
* This file is part of the aion network project.
Expand Down Expand Up @@ -37,19 +37,17 @@
import java.util.Properties;

/**
* Represents a configuration interface accepted that should be accepted by the
* repository to implement necessary configs
* Represents a configuration interface accepted that should be accepted by the repository to
* implement necessary configs
*
* @author yao
*/
public interface IRepositoryConfig {

/**
* @return absolute path to the DB folder containing files
*/
/** @return absolute path to the DB folder containing files */
String getDbPath();

int getPrune();
IPruneConfig getPruneConfig();

IContractDetails contractDetailsImpl();

Expand Down
Loading