SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics
Similar Threads
Thread Thread Starter Forum Replies Last Post
Suggestions on trimming RNA-seq file and compare splice-junction tools results kanewong Bioinformatics 0 04-10-2013 12:30 AM
Adapter trimming and trimming by quality question alisrpp Bioinformatics 5 04-08-2013 04:55 PM
trimming FASTA file baika Bioinformatics 7 03-05-2013 10:03 AM
Trimming WG BAM file for exome tahamasoodi Bioinformatics 0 11-16-2012 11:58 PM
Please Help: What is the differences between standard trimming and adaptive trimming byou678 Bioinformatics 8 08-22-2011 12:05 PM

Reply
 
Thread Tools
Old 11-10-2015, 04:44 AM   #21
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,143
Default

You are doing this on a cluster so I assume your job gets assigned to a random node, correct? Does a job that fails once run fine if you submit it a second (third time)? Is the job dying right away?
GenoMax is offline   Reply With Quote
Old 11-10-2015, 04:47 AM   #22
ea11
Member
 
Location: Southampton

Join Date: Jun 2015
Posts: 36
Default

As this is a short job, I am not sending the job off and install I am running it on one of the login nodes.

I have tried doing the same job more than once and it keeps failing for some reason, sometimes a job which ran fine the first pass of BBDuk, fails when I run it a second time through.

Nope it is not dying right away, it get killed at some point while it is running.
ea11 is offline   Reply With Quote
Old 11-10-2015, 04:56 AM   #23
ea11
Member
 
Location: Southampton

Join Date: Jun 2015
Posts: 36
Default

I am just trying the same job on a different login node to see if it works on that
ea11 is offline   Reply With Quote
Old 11-10-2015, 05:03 AM   #24
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,143
Default

You may need to ask sysadmins if they can find any evidence in systems logs as to why the job fails.

So the job that fails is actually producing some output before it gets killed?
GenoMax is offline   Reply With Quote
Old 11-10-2015, 05:07 AM   #25
ea11
Member
 
Location: Southampton

Join Date: Jun 2015
Posts: 36
Default

Yea the job that fails does produce a file in the specific output directory, but it is not the expected size (probably around half what it should be)

I have just tried running one of the files that failed previously on another login node and it seems to have run ok and wasn't killed. So I am guessing it might be the node, but what is causing it to fail I am not sure
ea11 is offline   Reply With Quote
Old 11-10-2015, 05:32 AM   #26
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,143
Default

Running jobs on head nodes is generally frowned upon by most admins.

Unless you want to figure out what is different between those two nodes just start using the job scheduler since you would need to do that anyway for alignments with BBMap.
GenoMax is offline   Reply With Quote
Old 11-10-2015, 05:37 AM   #27
ea11
Member
 
Location: Southampton

Join Date: Jun 2015
Posts: 36
Default

We have been told to use the login nodes for any jobs less than half an hour, which all the BBDuk jobs are.

And yea, I am already using the job scheduler for alignments and other jobs which require more processing power and time.

Thanks for the help
ea11 is offline   Reply With Quote
Old 11-10-2015, 09:32 AM   #28
Brian Bushnell
Super Moderator
 
Location: Walnut Creek, CA

Join Date: Jan 2014
Posts: 2,707
Default

The most likely problem is memory. Adapter-trimming requires only a little memory, so use the flag -Xmx1g instead of -Xmx40g. Also, by default, BBDuk will try to spawn pigz processes to accelerate compression and decompression, if pigz is installed. This can be disabled with "pigz=f unpigz=f". I think it is the combination of the two things. Essentially, due to some weirdness in Linux, when a process that uses a lot of virtual memory spawns a subprocess, for a split second it looks like it's using twice as much virtual memory. Often clusters are configured to kill jobs that do that.

So, "-Xmx1g" OR "pigz=f unpigz=f" will fix it - you don't need both. Sorry about that! We changed our cluster's configuration specifically so that it would not kill jobs in this circumstance, but I know that, for example, Amazon instances do.
Brian Bushnell is offline   Reply With Quote
Old 11-10-2015, 10:02 AM   #29
ea11
Member
 
Location: Southampton

Join Date: Jun 2015
Posts: 36
Default

Hi Brian, I tried lowering the memory, and it has worked. Thanks for your help. BBDuk has performed much better than cutadapt previously has for me.
ea11 is offline   Reply With Quote
Old 11-10-2015, 10:52 AM   #30
Brian Bushnell
Super Moderator
 
Location: Walnut Creek, CA

Join Date: Jan 2014
Posts: 2,707
Default

You're welcome; sorry it took so much effort
Brian Bushnell is offline   Reply With Quote
Reply

Tags
bbduk2

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



All times are GMT -8. The time now is 07:07 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2022, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO