ISWC2013Paper2BenchmarkDetails

Table of Contents ISWC 2013 Paper 2 Benchmark Details 1. Benchmark Machine 2. LUBM Benchmark 2.1 onTop-MySQL 2.1.1 Configuration 2.1.2 Load Time 2.2 onTop-DB2 2.2.1 Configuration 2.2.2 Load Time 2.3 Stardog 2.3.1 Configuration 2.3.2 Load Time 2.4 OWLIM 2.4.1 Configuration 2.4.2 Load Time 2.5 Query Performance Results 3. IMDB Benchmark 3.1 onTop-MySQL 3.2 onTop-DB2 3.3 Stardog Loading time 3.4 OWLIM Loading time 3.5 Query Performance Results

ISWC 2013 Paper 2 Benchmark Details

This page presents the test configurations and the detailed results in the paper submitted for ISWC 2013 entitled: "Ontology-Based Data Access: Ontop of Databases".

The following systems were used and presented in the paper:

• -onTop- using MySQL and DB2 as the back-end
• Stardog 1.2
• OWLIM-SE 5.3.5849

1. Benchmark Machine

The tests were run on the same machine with the following specification:

• Hardware

   * Processors: 24 units Intel Xeon 6 cores @3.47GHz
   * Memory: 106GB DDR3
   * Hard Disks: 1TB (15,000 rpm) SCSI and 200GB (15,000 rpm) SCSI

• Software

   * Operating System: Ubuntu 12.04 64-bit, Kernel 3.2.0
      * Filesystem: ext3
      * Separate partitions for application (on 200GB HDD) and databases (on 1TB HDD)
   * Java Version and JVM: Version 1.6.0_27, OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

2. LUBM Benchmark

The benchmark uses three different datasets generated with 50, 100 and 200 universities.

2.1 onTop-MySQL

2.1.1 Configuration

• MySQL: Version 5.6

   * Configuration file changes:

# CACHES AND LIMITS #
tmp_table_size                 = 32M
max_heap_table_size            = 32M
query_cache_type               = 0
query_cache_size               = 0
max_connections                = 500
thread_cache_size              = 50
open_files_limit               = 65535
table_definition_cache         = 1024
table_open_cache               = 2048

# INNODB #
innodb_flush_method            = O_DIRECT
innodb_log_file_size           = 512M
innodb_flush_log_at_trx_commit = 0
innodb_buffer_pool_size        = 80G


sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
innodb_thread_concurrency = 0
innodb_read_io_threads = 32

   * Indexes:

* Indexes on primary keys
* Indexes on columns given by LUBM benchmark

   * Other notes:

After loading the dataset, we performed index generation and did "analyse table" on all tables in the database.
MySQL Connector Version 5.0.8 was used for JDBC access.

2.1.2 Load Time

The table below summarises the load times of the CSV dump files (in hh:mm:ss):

	LUBM 50	LUBM 100	LUBM 200
Total time	00:01:49	00:03:32	00:05:08

2.2 onTop-DB2

2.2.1 Configuration

• MySQL: Version 5.6
• Indexes:

* Indexes on primary key
* Indexes on columns given by bsbmtools data generator

• Other notes:

* Set buffer pool page size 32K
* Set automatic memory, i.e., DB2 will use any available system memory
* After loading the dataset, we performed index generation and did "RUNSTATS" on all tables in the database.

2.2.2 Load Time

2.3 Stardog

2.3.1 Configuration

• Stardog Version 1.2
• Server configuration in stardog.properties:

strict.parsing=false

• Indexing:

Create less index by adding "--index-triples-only" in the loading command.

• Other notes:

* Use multiple input files (bulk load)
* Each input file was compressed using gzip.

2.3.2 Load Time

LUBM 50	LUBM 100	LUBM 200
00:00:20	00:00:47	00:01:36

2.4 OWLIM

2.4.1 Configuration

• OWLIM-SE Version 5.3.5849
• Repository configuration template for BSBM 25:

#
# Sesame configuration template for owlim repository
#
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>.
@prefix rep: <http://www.openrdf.org/config/repository#>.
@prefix sr: <http://www.openrdf.org/config/repository/sail#>.
@prefix sail: <http://www.openrdf.org/config/sail#>.
@prefix owlim: <http://www.ontotext.com/trree/owlim#>.

[] a rep:Repository ;
   rep:repositoryID "LUBM-EX_200" ;
   rdfs:label "LUBM EX 200 Universities" ;
   rep:repositoryImpl [
     rep:repositoryType "openrdf:SailRepository" ;
     sr:sailImpl [
       sail:sailType "owlim:Sail" ; 

       owlim:base-URL "http://swat.cse.lehigh.edu/onto/univ-bench.owl#" ;
       owlim:defaultNS "http://swat.cse.lehigh.edu/onto/univ-bench.owl#" ;
       owlim:entity-index-size "50000000" ;
       owlim:repository-type "file-repository" ;
       owlim:ruleset "owl-horst-optimized" ;
       owlim:imports "/data/datasets/lubm-extended/lubm-ex-200.owl";
       owlim:noPersist "false" ;

       # OWLIM-SE parameters
       owlim:cache-memory "15G" ; 
       
       # Other OWLIM-SE parameters
       owlim:transaction-mode "fast" ;       
      ]
   ].

2.4.2 Load Time

The table below summarises the load time of using a single Turtle TTL file (in hh:mm:ss).

LUBM 50	LUBM 100	LUBM 200
00:14:25	00:33:02	01:23:00

2.5 Query Performance Results

LUBM Query Performance

3. IMDB Benchmark

3.1 onTop-MySQL

3.2 onTop-DB2

3.3 Stardog

Loading time

IMDB
00:02:43

3.4 OWLIM

Loading time

IMDB
00:44:30

3.5 Query Performance Results

IMDB Performance Results