SEQanswers

Go Back   SEQanswers > Sequencing Technologies/Companies > Pacific Biosciences



Similar Threads
Thread Thread Starter Forum Replies Last Post
Bam output from SMRT Portal cascoamarillo Pacific Biosciences 3 04-29-2015 11:39 AM
PacBio assembly using SMRT portal manjari.deshmukh General 11 04-02-2015 08:53 AM
RS_CeleraAssembler not included in SMRT portal v2.3 macb Pacific Biosciences 5 02-02-2015 07:37 AM
Diary: Assembly in SMRT Portal 2.1.1 with HGAP+CA 8.1 pag Pacific Biosciences 20 09-06-2014 09:20 AM
SMRT portal errors bsp017 Pacific Biosciences 3 05-26-2014 04:57 AM

Reply
 
Thread Tools
Old 05-11-2015, 07:51 AM   #1
cascoamarillo
Senior Member
 
Location: MA

Join Date: Oct 2010
Posts: 160
Default HGAP assembly failed task on SMRT Portal

Hi

So I have 10 SMRT cells that I've been playing with. But de novo assembly with protocols RS_HGAP_Assembly.2 and .3 on SMRT Portal shows an error message during the process. Using smrtanalysis_2.3.0.140936.run with smrtanalysis-patch_2.3.0.140936.p3.run.

Code:
[INFO] 2015-04-28 22:02:21,943 [smrtpipe.status refreshTargets 409] Workflow Completion Status 139/212 in ( ...... 65%) tasks completed.
[ERROR] 2015-04-29 09:52:35,198 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_006of006
[ERROR] 2015-04-29 09:52:35,199 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_004of006
[ERROR] 2015-04-29 09:52:35,199 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_005of006
[ERROR] 2015-04-29 09:52:35,199 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_003of006
[ERROR] 2015-04-29 09:52:35,200 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006
[ERROR] 2015-04-29 09:52:35,200 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_002of006
[INFO] 2015-04-29 09:52:35,212 [smrtpipe.status execute 627] Found 6 failed tasks.
[INFO] 2015-04-29 09:52:35,213 [smrtpipe.status execute 629] task hgapAlignForCorrection_004of006 FAILED
[INFO] 2015-04-29 09:52:35,213 [smrtpipe.status execute 629] task hgapAlignForCorrection_003of006 FAILED
[INFO] 2015-04-29 09:52:35,214 [smrtpipe.status execute 629] task hgapAlignForCorrection_006of006 FAILED
[INFO] 2015-04-29 09:52:35,214 [smrtpipe.status execute 629] task hgapAlignForCorrection_001of006 FAILED
[INFO] 2015-04-29 09:52:35,215 [smrtpipe.status execute 629] task hgapAlignForCorrection_002of006 FAILED
[INFO] 2015-04-29 09:52:35,215 [smrtpipe.status execute 629] task hgapAlignForCorrection_005of006 FAILED
[ERROR] 2015-04-29 09:53:06,302 [SMRTpipe.SmrtPipeMain run 608] SmrtExit task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_004of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_003of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_006of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_002of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_005of006 Failed
that's part of log file where the errors start showing up. Not sure where is the problem.
Could it be due to a genome size limitation? Working with eukaryote genome (ca. 200Mb)
Thanks
cascoamarillo is offline   Reply With Quote
Old 05-11-2015, 09:01 AM   #2
gconcepcion
Member
 
Location: Menlo Park

Join Date: Dec 2010
Posts: 68
Default

Quote:
Originally Posted by cascoamarillo View Post
Hi

So I have 10 SMRT cells that I've been playing with. But de novo assembly with protocols RS_HGAP_Assembly.2 and .3 on SMRT Portal shows an error message during the process. Using smrtanalysis_2.3.0.140936.run with smrtanalysis-patch_2.3.0.140936.p3.run.

Code:
[INFO] 2015-04-28 22:02:21,943 [smrtpipe.status refreshTargets 409] Workflow Completion Status 139/212 in ( ...... 65%) tasks completed.
[ERROR] 2015-04-29 09:52:35,198 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_006of006
[ERROR] 2015-04-29 09:52:35,199 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_004of006
[ERROR] 2015-04-29 09:52:35,199 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_005of006
[ERROR] 2015-04-29 09:52:35,199 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_003of006
[ERROR] 2015-04-29 09:52:35,200 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006
[ERROR] 2015-04-29 09:52:35,200 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_002of006
[INFO] 2015-04-29 09:52:35,212 [smrtpipe.status execute 627] Found 6 failed tasks.
[INFO] 2015-04-29 09:52:35,213 [smrtpipe.status execute 629] task hgapAlignForCorrection_004of006 FAILED
[INFO] 2015-04-29 09:52:35,213 [smrtpipe.status execute 629] task hgapAlignForCorrection_003of006 FAILED
[INFO] 2015-04-29 09:52:35,214 [smrtpipe.status execute 629] task hgapAlignForCorrection_006of006 FAILED
[INFO] 2015-04-29 09:52:35,214 [smrtpipe.status execute 629] task hgapAlignForCorrection_001of006 FAILED
[INFO] 2015-04-29 09:52:35,215 [smrtpipe.status execute 629] task hgapAlignForCorrection_002of006 FAILED
[INFO] 2015-04-29 09:52:35,215 [smrtpipe.status execute 629] task hgapAlignForCorrection_005of006 FAILED
[ERROR] 2015-04-29 09:53:06,302 [SMRTpipe.SmrtPipeMain run 608] SmrtExit task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_004of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_003of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_006of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_002of006 Failed task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_005of006 Failed
that's part of log file where the errors start showing up. Not sure where is the problem.
Could it be due to a genome size limitation? Working with eukaryote genome (ca. 200Mb)
Thanks
At this point the failure is at the alignment-for-correction phase which occurs prior to usage of the genome size information to restrict the number of reads being used for the assembly. This failure is unrelated to the genome size setting, though that will come into play later.

Can you post the contents of:

[JOB_DIR]/log/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006.log
gconcepcion is offline   Reply With Quote
Old 05-11-2015, 09:10 AM   #3
cascoamarillo
Senior Member
 
Location: MA

Join Date: Oct 2010
Posts: 160
Default

here they are:
Code:
Setting up ENV on cluster5-01.bpcservers.private for task hgapAlignForCorrection_001of006
#!/bin/bash
# Setting up SMRTpipe environment
echo "Setting up ENV on $(uname -n)" for task hgapAlignForCorrection_001of006

SEYMOUR_HOME=/smrtanalysis/install/smrtanalysis_2.3.0.140936
source $SEYMOUR_HOME/etc/setup.sh

# Create the local TMP dir if it doesn't exist
tmp_dir=$(readlink -m "/smrtanalysis/tmpdir")
if [ ! -e "$tmp_dir" ]; then
   stat=0
   mkdir -p $tmp_dir || stat=$?
   if [[ $stat -ne 0 ]]; then
       echo "SMRTpipe Unable to create TMP dir '/smrtanalysis/tmpdir' on $(uname -n)" 1>&2
       exit 1
   else
       echo "successfully created or found TMP dir '/smrtanalysis/tmpdir'"
   fi
elif [[ ! -d "$tmp_dir" ]]; then
   echo "SMRTpipe TMP /smrtanalysis/tmpdir must be a directory on $(uname -n)" 1>&2
   exit 1
fi

########### TASK metadata #############
# Task            : hgapAlignForCorrection_001of006
# Module          : P_PreAssemblerDagcon
# Module Version  : 2.1.124285
# TaskType        : None
# URL             : task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006
# createdAt       : 2015-04-28 17:47:34.515890
# createdAt (UTC) : 2015-04-28 21:47:34.515909
# ncmds           : 2
# LogPath         : /smrtanalysis/userdata/jobs/016/016450/log/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006.log
# Script Path     : /smrtanalysis/userdata/jobs/016/016450/workflow/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006.sh

# Input       : /smrtanalysis/userdata/jobs/016/016450/data/nocontrol_filtered_subreads.fasta
# Input       : /smrtanalysis/userdata/jobs/016/016450/filtered_longreads.chunk001of006.fasta
# Output      : /smrtanalysis/userdata/jobs/016/016450/seeds.chunk001of006.m4
# Output      : /smrtanalysis/userdata/jobs/016/016450/seeds.chunk001of006.m4.fofn
#
########### END TASK metadata #############

cd /smrtanalysis/userdata/jobs/016/016450/log/P_PreAssemblerDagcon
# Writing to log file
cat /smrtanalysis/userdata/jobs/016/016450/workflow/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006.sh >> /smrtanalysis/userdata/jobs/016/016450/log/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006.log;



echo "Running task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006 on $(uname -a)"

echo "Started on $(date -u)"
echo 'Validating existence of Input Files'
if [ -e /smrtanalysis/userdata/jobs/016/016450/data/nocontrol_filtered_subreads.fasta ]
then
echo 'Successfully found /smrtanalysis/userdata/jobs/016/016450/data/nocontrol_filtered_subreads.fasta'
else
echo 'WARNING: Unable to find necessary input file, or dir /smrtanalysis/userdata/jobs/016/016450/data/nocontrol_filtered_subreads.fasta.'
fi
if [ -e /smrtanalysis/userdata/jobs/016/016450/filtered_longreads.chunk001of006.fasta ]
then
echo 'Successfully found /smrtanalysis/userdata/jobs/016/016450/filtered_longreads.chunk001of006.fasta'
else
echo 'WARNING: Unable to find necessary input file, or dir /smrtanalysis/userdata/jobs/016/016450/filtered_longreads.chunk001of006.fasta.'
fi
echo 'Successfully validated input files'

# Task hgapAlignForCorrection_001of006 commands:


# Completed writing Task hgapAlignForCorrection_001of006 commands


# Task 1
blasr /smrtanalysis/userdata/jobs/016/016450/data/nocontrol_filtered_subreads.fasta /smrtanalysis/userdata/jobs/016/016450/filtered_longreads.chunk001of006.fasta -out /smrtanalysis/userdata/jobs/016/016450/seeds.chunk001of006.m4 -m 4 -nproc 1 -bestn 10 -nCandidates 10 -noSplitSubreads -minReadLength 200 -maxScore -1000 -maxLCPLength 16 || exit $?
echo "Task 1 completed at $(date)"
# Task 2
echo /smrtanalysis/userdata/jobs/016/016450/seeds.chunk001of006.m4 > /smrtanalysis/userdata/jobs/016/016450/seeds.chunk001of006.m4.fofn || exit $?
echo "Task 2 completed at $(date)"



rcode=$?
echo "Finished on $(date -u)"
echo "Task hgapAlignForCorrection_001of006 with nproc 1 with exit code ${rcode}."
exit ${rcode}Running task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection_001of006 on Linux cluster5-01.bpcservers.private 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Started on Wed Apr 29 01:52:18 UTC 2015
Validating existence of Input Files
Successfully found /smrtanalysis/userdata/jobs/016/016450/data/nocontrol_filtered_subreads.fasta
Successfully found /smrtanalysis/userdata/jobs/016/016450/filtered_longreads.chunk001of006.fasta
Successfully validated input files
[INFO] 2015-04-28T21:52:18 [blasr] started.
# Writing stdout and stderr from Popen:
Your job 19310 ("Phga016450") has been submitted
Job 19310 exited because of signal SIGKILL
SIGKILL??
Thanks
cascoamarillo is offline   Reply With Quote
Old 05-11-2015, 10:00 AM   #4
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,077
Default

What is the hardware specs for the server you are running this on? Are you using a cluster or a stand-alone server?
GenoMax is offline   Reply With Quote
Old 05-11-2015, 10:04 AM   #5
gconcepcion
Member
 
Location: Menlo Park

Join Date: Dec 2010
Posts: 68
Default

Assuming you're running on SGE, one of two things happened:
1) Your sys admin qdel'd your job (unlikely)
2) Your job hit a resource limit, and SGE killed the job automatically either due to it's exceeding the time limit allowed for the job, cpu/memory limits.

Talk with your sys admin and find out why the job may have been killed.

Alternatively if you were running it locally, the job's memory consumption likely exceeded the system hardware.
gconcepcion is offline   Reply With Quote
Old 05-11-2015, 10:15 AM   #6
cascoamarillo
Senior Member
 
Location: MA

Join Date: Oct 2010
Posts: 160
Default

running on a server (CentOS 6.5) with SGE. 32 cpus and 1024 GB.

Thank you for point me in that direction> I'll ask my sys admin.
cascoamarillo is offline   Reply With Quote
Old 05-11-2015, 02:11 PM   #7
cascoamarillo
Senior Member
 
Location: MA

Join Date: Oct 2010
Posts: 160
Default

So the scheduler is killing the job but we are not sure why it's being set. The job is being submitted with a "hard" resource list that includes the parameter h_rt=43200. This means the hard limit real time lifespan of the job is 43200 seconds (12H).
This resource limit isn't something impose on the cluster. It's coming from whatever is submitting the job (SMRT Portal?).
cascoamarillo is offline   Reply With Quote
Old 05-11-2015, 02:15 PM   #8
rhall
Senior Member
 
Location: San Francisco

Join Date: Aug 2012
Posts: 322
Default

Check the sge template scripts in <SMRT Analysis>/analysis/etc/cluster/SGE/*.tmpl
rhall is offline   Reply With Quote
Old 05-11-2015, 02:21 PM   #9
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,077
Default

I recollect that SMRTportal needs to be able to submit sub-jobs from the original job that gets launched. My hunch is that your SGE may not be set up to allow that. You can ask your admins to verify.
GenoMax is offline   Reply With Quote
Old 05-11-2015, 02:34 PM   #10
gconcepcion
Member
 
Location: Menlo Park

Join Date: Dec 2010
Posts: 68
Default

Quote:
Originally Posted by cascoamarillo View Post
So the scheduler is killing the job but we are not sure why it's being set. The job is being submitted with a "hard" resource list that includes the parameter h_rt=43200. This means the hard limit real time lifespan of the job is 43200 seconds (12H).
This resource limit isn't something impose on the cluster. It's coming from whatever is submitting the job (SMRT Portal?).
12H is the default time limit per task in a default SMRTAnalysis install.

You can change that by following rhall's advice in a previous post and modifying the SGE scripts to increase the hard time limit that's already preset here:
[smrtanalysis_install]/analysis/etc/cluster/SGE/interactive.tmpl*
gconcepcion is offline   Reply With Quote
Old 05-12-2015, 07:59 AM   #11
cascoamarillo
Senior Member
 
Location: MA

Join Date: Oct 2010
Posts: 160
Default

Thank you all!
Wow, how do you know where are those config files? Just set a bigger hard time. Anyway; it seems (de novo assembly with pacbio) is quite a long process. I place in parallel a subset of filtered subreads (120,000) to be assembled with CA (using 12 processors and 48G of memory) and still waiting after 4 days.

Last edited by cascoamarillo; 05-12-2015 at 08:30 AM.
cascoamarillo is offline   Reply With Quote
Old 05-12-2015, 08:30 AM   #12
gconcepcion
Member
 
Location: Menlo Park

Join Date: Dec 2010
Posts: 68
Default

Quote:
Originally Posted by cascoamarillo View Post
Thank you all!
Wow, how do you know where are those config files? Just set a bigger hard time. Anyway; it seems (de novo assembly with pacbio) is quite a long process. I place in parallel a subset of filtered subreads to be assembled with CA (using 12 processors and 48G of memory) and still waiting after 4 days.
Glad it worked out for you! rhall and I are both pacbio employees who frequent this forum from time to time and are happy to lend a helping hand when time permits.

Alot of factors influence how long an assembly takes; e.g. cleanliness of library prep, size of genome, repetitiveness of genome, ploidy, quality and quantity of input data and the list goes on...
gconcepcion is offline   Reply With Quote
Old 05-12-2015, 10:11 AM   #13
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,077
Default

Quote:
Originally Posted by cascoamarillo View Post
Thank you all!
Wow, how do you know where are those config files? Just set a bigger hard time. Anyway; it seems (de novo assembly with pacbio) is quite a long process. I place in parallel a subset of filtered subreads (120,000) to be assembled with CA (using 12 processors and 48G of memory) and still waiting after 4 days.
Is the job running for 4 days or is it waiting for 4 days to get on the cluster?
GenoMax is offline   Reply With Quote
Old 05-12-2015, 04:20 PM   #14
cascoamarillo
Senior Member
 
Location: MA

Join Date: Oct 2010
Posts: 160
Default

Quote:
Originally Posted by GenoMax View Post
Is the job running for 4 days or is it waiting for 4 days to get on the cluster?
It's running. Apparently, it has finished with overlapStoreBuild process and now is dealing with the correct-frags.
cascoamarillo is offline   Reply With Quote
Old 05-26-2015, 07:10 AM   #15
cascoamarillo
Senior Member
 
Location: MA

Join Date: Oct 2010
Posts: 160
Default

Hi guys,
Let me continue this post with a question regarding CA output. In 9-terminator/ folder there is a summary of the assembly with Read Depth Histogram. There are contigs (consensus) with more read depth than others. I'd like to extract a subset of contigs with the maximun coverage reported. Is it possible to do that? Thanks.
cascoamarillo is offline   Reply With Quote
Old 05-26-2015, 09:50 AM   #16
rhall
Senior Member
 
Location: San Francisco

Join Date: Aug 2012
Posts: 322
Default

I would suggest using the coverage reported during the resequencing for the most accurate measure of raw read coverage. The following script takes the coverage.bed file and outputs a median coverage for each contig https://gist.github.com/rhallPB/7275c48d6f166e1410df

Last edited by rhall; 05-26-2015 at 09:53 AM.
rhall is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 01:24 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO