The Wayback Machine - https://web.archive.org/web/20201127223724/https://github.com/samtools/htsjdk/pull/922
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ParallelBlockCompressedOutputStream #922

Conversation

@SilinPavel
Copy link
Contributor

@SilinPavel SilinPavel commented Jun 29, 2017

We would like to represent new multithreading implementation of BlockCompressedOutptuStream.
ParallelBlockCompressedOutptuStream provides parallel zipping of GZ-blocks, which leads to performance gain by utilizing CPU cores.
Base AbstractBlockCompressedOutptuStream class was extracted. AbstractBlockCompressedOutptuStream was then extended by singlethread BlockCompressedOutptuStream and by ParallelBlockCompressedOutptuStream implementation.

uml classes

ParallelBlockCompressedOutptuStream implements deflateBlock method, which is called at the moment the buffer is full and GZ-block should be compressed and be written. The ParallelBCOS deflateBlock implementation submits the task of zipping the GZ-block to the ThreadPoolExecutor, so it will be processed in another thread in parallel. The number of threads in ThreadPoolExecutor (number of blocks are processed in parallel) is controlled by setting -Dsamjdk.zip_threads property. If the property is equal to 0 (by default), single thread implementation is used.
After enough (64 * ZIP_THREADS) deflating tasks are submitted, the writing task will be submitted. Writing task will join all previous deflating tasks and write them in the original order.

parallel

Here are benchmarks for comparing performance results of BlockCompressedOutputStream and ParallelBlockCompressedOutputStream:

Benchmark                               (blocks)  Mode  Cnt      Score      Error  Units
BCOSBenchmark.BCOSBenchmark                    2  avgt    5      4.206 ±    0.803  ms/op
BCOSBenchmark.BCOSBenchmark                 1024  avgt    5   1838.131 ±   58.076  ms/op
BCOSBenchmark.BCOSBenchmark                 4096  avgt    5   7791.901 ±  282.288  ms/op
BCOSBenchmark.BCOSBenchmark                16384  avgt    5  29659.399 ±  643.186  ms/op
BCOSBenchmark.parallelBCOSBenchmark4th         2  avgt    5      5.259 ±    6.846  ms/op
BCOSBenchmark.parallelBCOSBenchmark4th      1024  avgt    5    616.716 ±   46.377  ms/op
BCOSBenchmark.parallelBCOSBenchmark4th      4096  avgt    5   3037.837 ±  932.181  ms/op
BCOSBenchmark.parallelBCOSBenchmark4th     16384  avgt    5  10798.186 ± 1611.957  ms/op

We just generate random block of data and write it to the output stream.

Here are also results of second part of SortSam where BlockCompressedOutputStream is used:

master branch realization:
INFO	2017-06-28 13:23:55	SortSam	Wrote    10,000,000 records from a sorting collection.  Elapsed time: 00:20:45s.  Time for last 10,000,000:  118s.  Last read position: 21:24,587,673
INFO	2017-06-28 13:25:49	SortSam	Wrote    20,000,000 records from a sorting collection.  Elapsed time: 00:22:38s.  Time for last 10,000,000:  113s.  Last read position: 7:23,286,339
INFO	2017-06-28 13:27:40	SortSam	Wrote    30,000,000 records from a sorting collection.  Elapsed time: 00:24:30s.  Time for last 10,000,000:  111s.  Last read position: 16:11,025,537
INFO	2017-06-28 13:29:33	SortSam	Wrote    40,000,000 records from a sorting collection.  Elapsed time: 00:26:22s.  Time for last 10,000,000:  112s.  Last read position: 6:143,105,022
INFO	2017-06-28 13:31:27	SortSam	Wrote    50,000,000 records from a sorting collection.  Elapsed time: 00:28:16s.  Time for last 10,000,000:  114s.  Last read position: 1:186,643,572
INFO	2017-06-28 13:33:21	SortSam	Wrote    60,000,000 records from a sorting collection.  Elapsed time: 00:30:10s.  Time for last 10,000,000:  114s.  Last read position: 1:196,714,975
INFO	2017-06-28 13:35:13	SortSam	Wrote    70,000,000 records from a sorting collection.  Elapsed time: 00:32:03s.  Time for last 10,000,000:  112s.  Last read position: 16:49,983,724
INFO	2017-06-28 13:37:06	SortSam	Wrote    80,000,000 records from a sorting collection.  Elapsed time: 00:33:55s.  Time for last 10,000,000:  112s.  Last read position: 1:197,704,968
INFO	2017-06-28 13:38:59	SortSam	Wrote    90,000,000 records from a sorting collection.  Elapsed time: 00:35:48s.  Time for last 10,000,000:  113s.  Last read position: 9:125,905,256
INFO	2017-06-28 13:40:54	SortSam	Wrote   100,000,000 records from a sorting collection.  Elapsed time: 00:37:43s.  Time for last 10,000,000:  114s.  Last read position: 6:140,620,132
INFO	2017-06-28 13:42:46	SortSam	Wrote   110,000,000 records from a sorting collection.  Elapsed time: 00:39:35s.  Time for last 10,000,000:  111s.  Last read position: 9:104,433,180
INFO	2017-06-28 13:44:36	SortSam	Wrote   120,000,000 records from a sorting collection.  Elapsed time: 00:41:25s.  Time for last 10,000,000:  110s.  Last read position: 1:143,767,447
INFO	2017-06-28 13:46:26	SortSam	Wrote   130,000,000 records from a sorting collection.  Elapsed time: 00:43:16s.  Time for last 10,000,000:  110s.  Last read position: 5:168,250,174
INFO	2017-06-28 13:48:20	SortSam	Wrote   140,000,000 records from a sorting collection.  Elapsed time: 00:45:09s.  Time for last 10,000,000:  113s.  Last read position: 17:8,159,993
INFO	2017-06-28 13:50:12	SortSam	Wrote   150,000,000 records from a sorting collection.  Elapsed time: 00:47:01s.  Time for last 10,000,000:  112s.  Last read position: 6:70,942,372
INFO	2017-06-28 13:52:04	SortSam	Wrote   160,000,000 records from a sorting collection.  Elapsed time: 00:48:53s.  Time for last 10,000,000:  112s.  Last read position: 22:50,433,075
INFO	2017-06-28 13:53:56	SortSam	Wrote   170,000,000 records from a sorting collection.  Elapsed time: 00:50:45s.  Time for last 10,000,000:  111s.  Last read position: 6:16,823,069
INFO	2017-06-28 13:55:48	SortSam	Wrote   180,000,000 records from a sorting collection.  Elapsed time: 00:52:38s.  Time for last 10,000,000:  112s.  Last read position: 16:20,636,901
INFO	2017-06-28 13:57:40	SortSam	Wrote   190,000,000 records from a sorting collection.  Elapsed time: 00:54:29s.  Time for last 10,000,000:  111s.  Last read position: 4:180,667,149
INFO	2017-06-28 13:59:33	SortSam	Wrote   200,000,000 records from a sorting collection.  Elapsed time: 00:56:22s.  Time for last 10,000,000:  112s.  Last read position: X:7,434,779
INFO	2017-06-28 14:01:25	SortSam	Wrote   210,000,000 records from a sorting collection.  Elapsed time: 00:58:14s.  Time for last 10,000,000:  112s.  Last read position: 2:60,740,053

With 2 threads for zipping:
INFO	2017-06-28 14:21:05	SortSam	Wrote    10,000,000 records from a sorting collection.  Elapsed time: 00:19:10s.  Time for last 10,000,000:   76s.  Last read position: 21:24,587,673
INFO	2017-06-28 14:22:18	SortSam	Wrote    20,000,000 records from a sorting collection.  Elapsed time: 00:20:23s.  Time for last 10,000,000:   73s.  Last read position: 7:23,286,339
INFO	2017-06-28 14:23:32	SortSam	Wrote    30,000,000 records from a sorting collection.  Elapsed time: 00:21:36s.  Time for last 10,000,000:   73s.  Last read position: 16:11,025,537
INFO	2017-06-28 14:24:45	SortSam	Wrote    40,000,000 records from a sorting collection.  Elapsed time: 00:22:50s.  Time for last 10,000,000:   73s.  Last read position: 6:143,105,022
INFO	2017-06-28 14:25:59	SortSam	Wrote    50,000,000 records from a sorting collection.  Elapsed time: 00:24:03s.  Time for last 10,000,000:   73s.  Last read position: 1:186,643,572
INFO	2017-06-28 14:27:08	SortSam	Wrote    60,000,000 records from a sorting collection.  Elapsed time: 00:25:12s.  Time for last 10,000,000:   69s.  Last read position: 1:196,714,975
INFO	2017-06-28 14:28:21	SortSam	Wrote    70,000,000 records from a sorting collection.  Elapsed time: 00:26:25s.  Time for last 10,000,000:   73s.  Last read position: 16:49,983,724
INFO	2017-06-28 14:29:35	SortSam	Wrote    80,000,000 records from a sorting collection.  Elapsed time: 00:27:40s.  Time for last 10,000,000:   74s.  Last read position: 1:197,704,968
INFO	2017-06-28 14:30:47	SortSam	Wrote    90,000,000 records from a sorting collection.  Elapsed time: 00:28:52s.  Time for last 10,000,000:   71s.  Last read position: 9:125,905,256
INFO	2017-06-28 14:32:01	SortSam	Wrote   100,000,000 records from a sorting collection.  Elapsed time: 00:30:06s.  Time for last 10,000,000:   74s.  Last read position: 6:140,620,132
INFO	2017-06-28 14:33:15	SortSam	Wrote   110,000,000 records from a sorting collection.  Elapsed time: 00:31:20s.  Time for last 10,000,000:   74s.  Last read position: 9:104,433,180
INFO	2017-06-28 14:34:29	SortSam	Wrote   120,000,000 records from a sorting collection.  Elapsed time: 00:32:34s.  Time for last 10,000,000:   73s.  Last read position: 1:143,767,447
INFO	2017-06-28 14:35:42	SortSam	Wrote   130,000,000 records from a sorting collection.  Elapsed time: 00:33:47s.  Time for last 10,000,000:   72s.  Last read position: 5:168,250,174
INFO	2017-06-28 14:36:56	SortSam	Wrote   140,000,000 records from a sorting collection.  Elapsed time: 00:35:01s.  Time for last 10,000,000:   74s.  Last read position: 17:8,159,993
INFO	2017-06-28 14:38:16	SortSam	Wrote   150,000,000 records from a sorting collection.  Elapsed time: 00:36:21s.  Time for last 10,000,000:   79s.  Last read position: 6:70,942,372
INFO	2017-06-28 14:39:35	SortSam	Wrote   160,000,000 records from a sorting collection.  Elapsed time: 00:37:40s.  Time for last 10,000,000:   78s.  Last read position: 22:50,433,075
INFO	2017-06-28 14:40:48	SortSam	Wrote   170,000,000 records from a sorting collection.  Elapsed time: 00:38:53s.  Time for last 10,000,000:   73s.  Last read position: 6:16,823,069
INFO	2017-06-28 14:42:02	SortSam	Wrote   180,000,000 records from a sorting collection.  Elapsed time: 00:40:06s.  Time for last 10,000,000:   73s.  Last read position: 16:20,636,901
INFO	2017-06-28 14:43:18	SortSam	Wrote   190,000,000 records from a sorting collection.  Elapsed time: 00:41:22s.  Time for last 10,000,000:   76s.  Last read position: 4:180,667,149
INFO	2017-06-28 14:44:34	SortSam	Wrote   200,000,000 records from a sorting collection.  Elapsed time: 00:42:39s.  Time for last 10,000,000:   76s.  Last read position: X:7,434,779
INFO	2017-06-28 14:45:56	SortSam	Wrote   210,000,000 records from a sorting collection.  Elapsed time: 00:44:01s.  Time for last 10,000,000:   81s.  Last read position: 2:60,740,053

NOTE:
write(), close() and flush() methods can block the current thread because we have to wait until all tasks for zipping and writing will be completed.
ThreadPoolExecutor is static, for the reason to restrict the number of threads globally (in case several PBCOS are created).

@codecov-io
Copy link

@codecov-io codecov-io commented Jun 29, 2017

Codecov Report

Merging #922 into master will increase coverage by 0.033%.
The diff coverage is 74.074%.

@@               Coverage Diff               @@
##              master      #922       +/-   ##
===============================================
+ Coverage     65.118%   65.151%   +0.033%     
- Complexity      7278      7289       +11     
===============================================
  Files            528       530        +2     
  Lines          31971     32067       +96     
  Branches        5468      5478       +10     
===============================================
+ Hits           20819     20892       +73     
- Misses          8995      9013       +18     
- Partials        2157      2162        +5
Impacted Files Coverage Δ Complexity Δ
...va/htsjdk/samtools/util/AsyncBufferedIterator.java 69.565% <0%> (ø) 17 <0> (ø) ⬇️
src/main/java/htsjdk/samtools/BAMFileWriter.java 46.154% <10%> (-1.099%) 15 <0> (ø)
...ain/java/htsjdk/samtools/SAMFileWriterFactory.java 63.971% <100%> (ø) 41 <1> (ø) ⬇️
src/main/java/htsjdk/samtools/Defaults.java 77.551% <100%> (+0.468%) 6 <0> (ø) ⬇️
...jdk/samtools/util/BlockCompressedOutputStream.java 71.739% <75%> (-4.49%) 8 <0> (-16)
...ools/util/ParallelBlockCompressedOutputStream.java 77.419% <77.419%> (ø) 9 <9> (?)
...ools/util/AbstractBlockCompressedOutputStream.java 78.205% <78.205%> (ø) 18 <18> (?)
...samtools/util/AsyncBlockCompressedInputStream.java 72% <0%> (-4%) 12% <0%> (-1%)
... and 1 more
@SilinPavel SilinPavel force-pushed the EpamLifeSciencesTeam:epam-ls_ParallelBlockCompressedOutputStream branch from 2f9fff3 to d3101b4 Jun 29, 2017
@SilinPavel SilinPavel force-pushed the EpamLifeSciencesTeam:epam-ls_ParallelBlockCompressedOutputStream branch from d3101b4 to b9ec2b4 Jun 29, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
You can’t perform that action at this time.