Integer + Floating Point Compression Filter

• Fastest transpose/shuffle
  • Byte/Nibble transpose/shuffle for improving compression of binary data (ex. floating point data)
  • ✨ Scalar/SIMD Transpose/Shuffle 8,16,32,64,... bits
  • 👍 Dynamic CPU detection and JIT scalar/sse/avx2 switching
  • 100% C (C++ headers), usage as simple as memcpy
• Byte Transpose
  • Fastest byte transpose
• Nibble Transpose
  • nearly as fast as byte transpose
  • more efficient, up to 6 times! faster than Bitshuffle
  • 🆕 better compression (w/ lz77) and
    10 times! faster than one of the best floating-point compressors SPDP
  • can compress/decompress (w/ lz77) better and faster than other domain specific floating point compressors
• Scalar and SIMD Transform
  • Delta encoding for sorted lists
  • Zigzag encoding for unsorted lists
  • Xor encoding
  • 🆕 lossy floating point compression with user-defined error

Transpose Benchmark:

• CPU: Skylake i7-6700 3.4GHz gcc 7.2 single thread

- Speed test

Benchmark w/ 16k buffer

BOLD = pareto frontier.
E:Encode, D:Decode

    ./tpbench -s# file -B16K   (# = 8,4,2)

Transpose/Shuffle benchmark w/ large files.

MB/s: 1,000,000 bytes/second

    ./tpbench -s# file  (# = 8,4,2)

- Compression test (transpose/shuffle+lz4)

🆕 Download IcApp a new benchmark for TurboPFor+TurboTranspose
for testing allmost all integer and floating point file types.
Note: Lossy compression benchmark with icapp only.

• Scientific IEEE 754 32-Bit Single-Precision Floating-Point Datasets

- Speed test (file msg_sweep3d)

    ./tpbench -s4 -z *.sp

Tp8=Byte transpose, Tp4=Nibble transpose, lz = lz4
eTp4Lzt = lossy compression with lzturbo and allowed error = 0.0001 (1e-4)
Slow but best compression: SPDP9 and lzt = lzturbo,39

• Scientific IEEE 754 64-Bit Double-Precision Floating-Point Datasets

    ./tpbench -s8 -z *.trace
  

eTp4Lzt = lossy compression with allowed error = 0.0001

Compile:

    git clone git://github.com/powturbo/TurboTranspose.git
    cd TurboTranspose

Linux + Windows MingW

	make
    or
	make AVX2=1

Windows Visual C++

	nmake /f makefile.vs
    or
	nmake AVX2=1 /f makefile.vs

• benchmark with other libraries
  download or clone bitshuffle or blosc and type

    make AVX2=1 BLOSC=1
    or
    make AVX2=1 BITSHUFFLE=1
  

Testing:

• benchmark "transpose" functions
  

  ./tpbench [-s#] [-z] file
  s# = element size #=2,4,8,16,... (default 4) 
  -z = only lz77 compression benchmark (bitshuffle package mandatory)
  

Function usage:

Byte transpose:

void tpenc( unsigned char *in, unsigned n, unsigned char *out, unsigned esize);
void tpdec( unsigned char *in, unsigned n, unsigned char *out, unsigned esize)
in : input buffer
n : number of bytes
out : output buffer
esize : element size in bytes (2,4,8,...)

Nibble transpose:

void tp4enc( unsigned char *in, unsigned n, unsigned char *out, unsigned esize);
void tp4dec( unsigned char *in, unsigned n, unsigned char *out, unsigned esize)
in : input buffer
n : number of bytes
out : output buffer
esize : element size in bytes (2,4,8,...)

Environment:

OS/Compiler (64 bits):

• Linux: GNU GCC (>=4.6)
• clang (>=3.2)
• Windows: MinGW-w64
• Windows: Visual C++ (>=VS2008)

Multithreading:

• All TurboTranspose functions are thread safe

References:

• BS - Bitshuffle: Filter for improving compression of typed binary data.
  📗 A compression scheme for radio data in high performance computing
• Blosc: A blocking, shuffling and loss-less compression
• SPDP is a compression/decompression algorithm for binary IEEE 754 32/64 bits floating-point data
  📗 SPDP - An Automatically Synthesized Lossless Compression Algorithm for Floating-Point Data + DCC 2018
• 📗 FPC: A High-Speed Compressor for Double-Precision Floating-Point Data

Last update: 11 Jun 2018