I don't know why they do that. I blame it on Illumina's demultiplexing software defaults. Like on NextSeq, which only has one physical lane (in terms of library separation), we still produce 8 files per library, which is really inefficient from a tracking and labor perspective.
Seqanswers Leaderboard Ad
Collapse
Announcement
Collapse
No announcement yet.
X
-
Originally posted by Brian Bushnell View PostLike on NextSeq, which only has one physical lane (in terms of library separation), we still produce 8 files per library, which is really inefficient from a tracking and labor perspective.
Comment
-
Originally posted by Brian Bushnell View PostBBDuk2 should work fine if you adjust the parameters as I indicated. That said - personally, I would use 2 passes of BBDuk because BBDuk2 is a bit more confusing and less flexible (you can't use different kmer lengths for left and right trimming, for example). I designed BBDuk2 for integration into pipelines that get written once and then run exactly the same way for years, to achieve maximal efficiency, since it can do all kmer operations in a single pass (filtering, left-trimming, right-trimming, and masking). But actually I never use it because I usually want different values of K and a different hamming distance for the different steps.
The issue here is either that you are running OpenJDK, or version 1.6, and probably both combined. I only test with Oracle's JDK, and use version 1.7 and 1.8.
Hi Brian,
So I have decided to use BBDuk instead of BBDul2 and do it twice. The following is my code:
bash bbduk.sh in1=/PATH/M_R1_001.fastq.gz in2=/PATH/M_R2_001.fastq.gz out1=/PATH/M1_R1_001.fastq.gz out2=/PATH/M1_R2_001.fastq.gz ref="/truseq.fa.gz" ktrim=r k=13 mink=11 hdist=1 rcomp=t minlen=25 qtrim=rl trimq=10 tpe tbo
For some of my samples, this works fine and I get a complete output and a summary of the input number of reads and reads left after trimming.
However, for some of my samples, I am not getting this, instead I get the following:
BBDuk version 35.66
maskMiddle was disabled because useShortKmers=true
Initial:
Memory: max=41160m, free=40086m, used=1074m
Added 182 kmers; time: 0.032 seconds.
Memory: max=41160m, free=39012m, used=2148m
Input is being processed as paired
Started output streams: 0.221 seconds.
bbduk.sh: line 282: 888 Killed java -Djava.library.path=/PATH/bbmap/jni/ -ea -Xmx40g -Xms40g -cp /PATH/bbmap/current/ jgi.BBDukF -Xmx40g in1=/PATH/M_R1_001.fastq.gz in2=/PATH/M_R2_001.fastq.gz out1=/scratch/ea11g10/PATH/M1_R1_001.fastq.gz out2=/PATH/M1_R2_001.fastq.gz literal=GCTCTTCCGATCT ktrim=l k=13 mink=11 hdist=1 rcomp=t minlen=25 qtrim=rl trimq=10 tpe tbo
Do you know why it would be getting killed? I am not using Java 1.8 as it has been upgraded:
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
Am I running out of memory and so it is being killed?
Thanks
Comment
-
As this is a short job, I am not sending the job off and install I am running it on one of the login nodes.
I have tried doing the same job more than once and it keeps failing for some reason, sometimes a job which ran fine the first pass of BBDuk, fails when I run it a second time through.
Nope it is not dying right away, it get killed at some point while it is running.
Comment
-
Yea the job that fails does produce a file in the specific output directory, but it is not the expected size (probably around half what it should be)
I have just tried running one of the files that failed previously on another login node and it seems to have run ok and wasn't killed. So I am guessing it might be the node, but what is causing it to fail I am not sure
Comment
-
Running jobs on head nodes is generally frowned upon by most admins.
Unless you want to figure out what is different between those two nodes just start using the job scheduler since you would need to do that anyway for alignments with BBMap.
Comment
-
The most likely problem is memory. Adapter-trimming requires only a little memory, so use the flag -Xmx1g instead of -Xmx40g. Also, by default, BBDuk will try to spawn pigz processes to accelerate compression and decompression, if pigz is installed. This can be disabled with "pigz=f unpigz=f". I think it is the combination of the two things. Essentially, due to some weirdness in Linux, when a process that uses a lot of virtual memory spawns a subprocess, for a split second it looks like it's using twice as much virtual memory. Often clusters are configured to kill jobs that do that.
So, "-Xmx1g" OR "pigz=f unpigz=f" will fix it - you don't need both. Sorry about that! We changed our cluster's configuration specifically so that it would not kill jobs in this circumstance, but I know that, for example, Amazon instances do.
Comment
Latest Articles
Collapse
-
by seqadmin
Proteins are often described as the workhorses of the cell, and identifying their sequences is key to understanding their role in biological processes and disease. Currently, the most common technique used to determine protein sequences is mass spectrometry. While still a valuable tool, mass spectrometry faces several limitations and requires a highly experienced scientist familiar with the equipment to operate it. Additionally, other proteomic methods, like affinity assays, are constrained...-
Channel: Articles
04-04-2024, 04:25 PM -
-
by seqadmin
Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...-
Channel: Articles
03-22-2024, 06:39 AM -
ad_right_rmr
Collapse
News
Collapse
Topics | Statistics | Last Post | ||
---|---|---|---|---|
Started by seqadmin, 04-11-2024, 12:08 PM
|
0 responses
27 views
0 likes
|
Last Post
by seqadmin
04-11-2024, 12:08 PM
|
||
Started by seqadmin, 04-10-2024, 10:19 PM
|
0 responses
30 views
0 likes
|
Last Post
by seqadmin
04-10-2024, 10:19 PM
|
||
Started by seqadmin, 04-10-2024, 09:21 AM
|
0 responses
26 views
0 likes
|
Last Post
by seqadmin
04-10-2024, 09:21 AM
|
||
Started by seqadmin, 04-04-2024, 09:00 AM
|
0 responses
52 views
0 likes
|
Last Post
by seqadmin
04-04-2024, 09:00 AM
|
Comment