### Pipeline run code and environment:
* Command: `/home/mjs5kd/.local/bin/refgenie build hg38/bismark_bt2_index`
* Compute host: udc-ba25-17
* Working dir: /sfs/qumulo/qproject/shefflab/deploy/rg.databio.org_full/rg.databio.org
* Outfolder: /project/shefflab/deploy/rg.databio.org_full/genomes/data/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/bismark_bt2_index/default/_refgenie_build/
* Pipeline started at: (04-07 20:45:46) elapsed: 0.0 _TIME_
### Version log:
* Python version: 3.6.6
* Pypiper dir: `/sfs/qumulo/qhome/mjs5kd/.local/lib/python3.6/site-packages/pypiper`
* Pypiper version: 0.12.1
* Pipeline dir: `/sfs/qumulo/qhome/mjs5kd/.local/bin`
* Pipeline version: None
### Arguments passed to pipeline:
* `asset_registry_paths`: `['hg38/bismark_bt2_index']`
* `assets`: `None`
* `command`: `build`
* `config_file`: `refgenie.yaml`
* `docker`: `False`
* `files`: `None`
* `genome`: `None`
* `genome_config`: `None`
* `genome_description`: `None`
* `logdev`: `False`
* `new_start`: `False`
* `outfolder`: `/project/shefflab/deploy/rg.databio.org_full/genomes/data`
* `params`: `None`
* `recipe`: `None`
* `recover`: `False`
* `requirements`: `False`
* `silent`: `False`
* `skip_read_lock`: `False`
* `tag_description`: `None`
* `verbosity`: `None`
* `volumes`: `None`
----------------------------------------
Target to produce: `/project/shefflab/deploy/rg.databio.org_full/genomes/data/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/bismark_bt2_index/default/_refgenie_build/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4_bismark_bt2_index__default.flag`
> `ln -sf /project/shefflab/deploy/rg.databio.org_full/genomes/data/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/fasta/default/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4.fa /project/shefflab/deploy/rg.databio.org_full/genomes/data/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/bismark_bt2_index/default` (2599)
Command completed. Elapsed time: 0:00:00. Running peak memory: 0.0GB.
PID: 2599; Command: ln; Return code: 0; Memory used: 0.0GB
> `bismark_genome_preparation --bowtie2 /project/shefflab/deploy/rg.databio.org_full/genomes/data/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/bismark_bt2_index/default` (2600)
[33mWARNING:[0m Skipping mount /opt/singularity/3.5.2/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
Writing bisulfite genomes out into a single MFA (multi FastA) file
Bisulfite Genome Indexer version v0.22.1 (last modified: 14 April 2019)
Step I - Prepare genome folders - completed
Total number of conversions performed:
C->T: 598683433
G->A: 600854940
Step II - Genome bisulfite conversions - completed
Bismark Genome Preparation - Step III: Launching the Bowtie 2 indexer
Please be aware that this process can - depending on genome size - take several hours!
Settings:
Output files: "BS_CT.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
genome_mfa.CT_conversion.fa
Building a SMALL index
Reading reference sizes
Settings:
Output files: "BS_GA.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
genome_mfa.GA_conversion.fa
Building a SMALL index
Reading reference sizes
Time reading reference sizes: 00:00:40
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time reading reference sizes: 00:00:40
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:32
bmax according to bmaxDivN setting: 733719112
Using parameters --bmax 550289334 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 550289334 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
Building sPrime
Time to join reference sequences: 00:00:33
bmax according to bmaxDivN setting: 733719112
Using parameters --bmax 550289334 --dcv 1024
Doing ahead-of-time memory usage test
Building sPrimeOrder
V-Sorting samples
Passed! Constructing with these parameters: --bmax 550289334 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
Building sPrime
Building sPrimeOrder
V-Sorting samples
V-Sorting samples time: 00:02:24
Allocating rank array
Ranking v-sort output
V-Sorting samples time: 00:02:23
Allocating rank array
Ranking v-sort output
Ranking v-sort output time: 00:00:35
Invoking Larsson-Sadakane on ranks
Ranking v-sort output time: 00:00:35
Invoking Larsson-Sadakane on ranks
Invoking Larsson-Sadakane on ranks time: 00:01:09
Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
(Using difference cover)
Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
Splitting and merging time: 00:00:00
Avg bucket size: 2.93488e+09 (target: 550289333)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
No samples; assembling all-inclusive block
Invoking Larsson-Sadakane on ranks time: 00:01:09
Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
(Using difference cover)
Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
Splitting and merging time: 00:00:00
Avg bucket size: 2.93488e+09 (target: 550289333)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
No samples; assembling all-inclusive block
Sorting block of length 2934876451 for bucket 1
(Using difference cover)
Sorting block of length 2934876451 for bucket 1
(Using difference cover)
Sorting block time: 00:44:27
Returning block of 2934876452 for bucket 1
Sorting block time: 00:45:50
Returning block of 2934876452 for bucket 1
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 1467274941
fchr[G]: 2065958374
fchr[T]: 2065958374
fchr[$]: 2934876451
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 982507425 bytes to primary EBWT file: BS_GA.1.bt2
Wrote 733719120 bytes to secondary EBWT file: BS_GA.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
len: 2934876451
bwtLen: 2934876452
sz: 733719113
bwtSz: 733719113
lineRate: 6
offRate: 4
offMask: 0xfffffff0
ftabChars: 10
eftabLen: 20
eftabSz: 80
ftabLen: 1048577
ftabSz: 4194308
offsLen: 183429779
offsSz: 733719116
lineSz: 64
sideSz: 64
sideBwtSz: 48
sideBwtLen: 192
numSides: 15285815
numLines: 15285815
ebwtTotLen: 978292160
ebwtTotSz: 978292160
color: 0
reverse: 0
Total time for call to driver() for forward index: 00:57:54
Reading reference sizes
Time reading reference sizes: 00:00:30
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:33
Time to reverse reference sequence: 00:00:05
bmax according to bmaxDivN setting: 733719112
Using parameters --bmax 550289334 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 550289334 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
Building sPrime
Building sPrimeOrder
V-Sorting samples
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 866420001
fchr[G]: 866420001
fchr[T]: 1467274941
fchr[$]: 2934876451
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 982507425 bytes to primary EBWT file: BS_CT.1.bt2
Wrote 733719120 bytes to secondary EBWT file: BS_CT.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
len: 2934876451
bwtLen: 2934876452
sz: 733719113
bwtSz: 733719113
lineRate: 6
offRate: 4
offMask: 0xfffffff0
ftabChars: 10
eftabLen: 20
eftabSz: 80
ftabLen: 1048577
ftabSz: 4194308
offsLen: 183429779
offsSz: 733719116
lineSz: 64
sideSz: 64
sideBwtSz: 48
sideBwtLen: 192
numSides: 15285815
numLines: 15285815
ebwtTotLen: 978292160
ebwtTotSz: 978292160
color: 0
reverse: 0
Total time for call to driver() for forward index: 00:59:39
Reading reference sizes
Time reading reference sizes: 00:00:31
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:33
Time to reverse reference sequence: 00:00:05
bmax according to bmaxDivN setting: 733719112
Using parameters --bmax 550289334 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 550289334 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
Building sPrime
Building sPrimeOrder
V-Sorting samples
V-Sorting samples time: 00:02:24
Allocating rank array
Ranking v-sort output
Ranking v-sort output time: 00:00:34
Invoking Larsson-Sadakane on ranks
Invoking Larsson-Sadakane on ranks time: 00:01:10
Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
(Using difference cover)
Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
Splitting and merging time: 00:00:00
Avg bucket size: 2.93488e+09 (target: 550289333)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
No samples; assembling all-inclusive block
V-Sorting samples time: 00:02:24
Allocating rank array
Ranking v-sort output
Sorting block of length 2934876451 for bucket 1
(Using difference cover)
Ranking v-sort output time: 00:00:35
Invoking Larsson-Sadakane on ranks
Invoking Larsson-Sadakane on ranks time: 00:01:09
Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
(Using difference cover)
Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
Splitting and merging time: 00:00:00
Avg bucket size: 2.93488e+09 (target: 550289333)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
No samples; assembling all-inclusive block
Sorting block of length 2934876451 for bucket 1
(Using difference cover)
Sorting block time: 00:44:24
Returning block of 2934876452 for bucket 1
Sorting block time: 00:45:50
Returning block of 2934876452 for bucket 1
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 1467274941
fchr[G]: 2065958374
fchr[T]: 2065958374
fchr[$]: 2934876451
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 982507425 bytes to primary EBWT file: BS_GA.rev.1.bt2
Wrote 733719120 bytes to secondary EBWT file: BS_GA.rev.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
len: 2934876451
bwtLen: 2934876452
sz: 733719113
bwtSz: 733719113
lineRate: 6
offRate: 4
offMask: 0xfffffff0
ftabChars: 10
eftabLen: 20
eftabSz: 80
ftabLen: 1048577
ftabSz: 4194308
offsLen: 183429779
offsSz: 733719116
lineSz: 64
sideSz: 64
sideBwtSz: 48
sideBwtLen: 192
numSides: 15285815
numLines: 15285815
ebwtTotLen: 978292160
ebwtTotSz: 978292160
color: 0
reverse: 1
Total time for backward call to driver() for mirror index: 00:57:53
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 866420001
fchr[G]: 866420001
fchr[T]: 1467274941
fchr[$]: 2934876451
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 982507425 bytes to primary EBWT file: BS_CT.rev.1.bt2
Wrote 733719120 bytes to secondary EBWT file: BS_CT.rev.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
len: 2934876451
bwtLen: 2934876452
sz: 733719113
bwtSz: 733719113
lineRate: 6
offRate: 4
offMask: 0xfffffff0
ftabChars: 10
eftabLen: 20
eftabSz: 80
ftabLen: 1048577
ftabSz: 4194308
offsLen: 183429779
offsSz: 733719116
lineSz: 64
sideSz: 64
sideBwtSz: 48
sideBwtLen: 192
numSides: 15285815
numLines: 15285815
ebwtTotLen: 978292160
ebwtTotSz: 978292160
color: 0
reverse: 1
Total time for backward call to driver() for mirror index: 00:57:50
Command completed. Elapsed time: 1:59:14. Running peak memory: 30.371GB.
PID: 2600; Command: bismark_genome_preparation; Return code: 0; Memory used: 30.371GB
> `touch /project/shefflab/deploy/rg.databio.org_full/genomes/data/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/bismark_bt2_index/default/_refgenie_build/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4_bismark_bt2_index__default.flag` (4711)
Command completed. Elapsed time: 0:00:00. Running peak memory: 30.371GB.
PID: 4711; Command: touch; Return code: 0; Memory used: 0.001GB
Asset digest: 3ad42e5278d5efa9ad62827faa0eb828
Default tag for '2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/bismark_bt2_index' set to: default
### Pipeline completed. Epilogue
* Elapsed time (this run): 1:59:47
* Total elapsed time (all runs): 1:59:14
* Peak memory (this run): 30.3712 GB
* Pipeline completed time: 2021-04-07 22:45:33
Finished building 'bismark_bt2_index' asset
Created alias directories:
- /project/shefflab/deploy/rg.databio.org_full/genomes/alias/hg38/bismark_bt2_index/default