Database Test 5 (DBT-5) Documentation

Introduction

This document provides instructions on how to set up and use the Open Source Development Lab's Database Test 5 (DBT-5) kit. This kit is an open source fair-use implementation of the TPC Benchmark(TM) E (TPC-E) specification, which is an on-line transaction processing benchmark.

An introduction and complete details on the TPC-E can be found at: https://tpc.org/tpce/

Design

DBT-5 implements the workload with three binaries: BrokerageHouse, MarketExchange, and Driver.

BrokerageHouse

BrokerageHouse is responsible for executing all of the database transactions. It implements the EGenTxnHarness and EGenDriverDM.

MarketExchange

MarketExchange emulates stock exchanges and sends messages to the BrokerageHouse for initial Trade Request and Mark Feed transactions. It implements the EGenDriverMEE.

Driver

Driver emulates customers and drives the workload. It implements EGenDriverCE.

Installing DBT-5

The latest stable and development version of the kit can be found on GitHub: https://github.com/osdldbt/dbt5

The TPC's TPC-E Tools cannot be redistributed with DBT-5 and must be downloaded separately : https://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp

Required Software

In addition to the database management system software, the following is also required:

C++ compiler
patch
Database development libraries and header files
- PostgreSQL https://www.postgresql.org
  
  psql
SQLite https://www.sqlite.org/index.html
sar, pidstat http://pagesperso-orange.fr/sebastien.godard/ (While the scripts assume this particular version of sar and pidstat, it is possible to run on non-Linux based operating systems with some modifications to the kit.)

User's Guide

Building TPC-E Tools

Download the TPC-E Tools directly from the TPC: https://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp

The kit requires the TPC-E Tools to be built for the specific database management system to be tested. The TPC-E Tools is developed in such a way that it needs to rebuilt or another copy needs to be built if a different database management system is to be tested.

DBT-5 provides a script to apply patches and code to compile the TPC-E Tools. The patches that are applied are minor code changes and code is supplied to build sponsor supplied code.

For example, to build the TPC-E Tools for PostgreSQL (pgsql), unzip the TPC-E Tools zip file into an empty directory and run dbt5-build-egen against the resulting directory:

mkdir /tmp/egen
cd /tmp/egen
unzip /path/to/*-tpc-e-tool.zip
dbt5-build-egen /tmp/egen

Database Management System Notes

PostgreSQL

Building the Kit

Developed against Postgresql 8.4 and newer. May work with older versions but not quite tested.

By default, the kit will use PL/pgsql stored functions, but you may use C stored functions instead. The C stored functions need to be separately built and installed on the database system before they can be used by the database:

cd storedproc/pgsql/c
make
make install
dbt5 pgsql-load-stored-procs -t c

Build the Database

To build the minimum sized database in a database dbt5:

dbt5 build --tpcetools=/tmp/egen pgsql

Run a Test

Run a quick 120 second (2 minute) test with 1 user:

dbt5 run -d 120 pgsql /tmp/results

Developer Guide

This document is for detailing anything related to the development of this test kit.

Building the Kit

CMake is build system used for this kit. A Makefile.cmake is provided to automate some of the tasks.

Building for debugging:

make -f Makefile.cmake debug

Building for release:

make -f Makefile.cmake release

Building source packages:

make -f Makefile.cmake package

See the AppImage section for details on building an AppImage. There are additional requirements for the appimage target in the Makefile.cmake. Alternatively, the kit provides scripts in the tools diretory to create a container that can create an AppImage.

AppImage

AppImages are only for Linux based systems: https://appimage.org/

The AppImageKit AppImage can be downloaded from: https://github.com/AppImage/AppImageKit/releases

It is recommended to build AppImages on older distributions: https://docs.appimage.org/introduction/concepts.html#build-on-old-systems-run-on-newer-systems

The logo used is the number "5" from the Freeware Metal On Metal Font.

See the README.rst in the tools/ directory for an example of creating an AppImage with a Podman container.

Building the AppImage

The AppImages builds a custom minimally configured PostgreSQL build to reduce library dependency requirements. Part of this reason is to make it easier to include libraries with compatible licences. At least version PostgreSQL 11 should be used for the pg_type_d.h header file.

At the time of this document, PostgreSQL 11 was configured with the following options:

./configure --without-ldap --without-readline --without-zlib \
      --without-gssapi --with-openssl

Don't forget that both PATH and LD_LIBRARY_PATH may need to be set appropriately depending on where the custom build of PostgreSQL is installed.