I am getting strange memory issues after runCaToUnitig stage is compeleted:
I am using 50GB memory which should be more than enough for a 10Mb organism. I get stuck with the same error while using data from 1 cell or data from 3 cells.
All logs seem to exit with code 0:
The log from runCaToUnitig.log is as follows:
Can anyone please help?
Thanks
Code:
[INFO] 2017-01-30 17:39:23,826 Workflow Completion Status 35/56 in ( ...... 62%) tasks completed. [INFO] 2017-01-30 17:39:28,836 task P_AssembleUnitig.writeRunCASpec successfully completed (4.00 sec (0.07 min)). freed 1 slots, using 1 slots. [INFO] 2017-01-30 17:39:28,837 Workflow Completion Status 36/56 in ( ...... 64%) tasks completed. [INFO] 2017-01-30 17:39:33,845 Starting task://Anonymous/P_AssembleUnitig/runCaToUnitig [INFO] 2017-01-30 17:39:50,860 task P_PreAssemblerDagcon.preAssemblerJsonReport successfully completed (25.00 sec (0.42 min)). freed 1 slots, using 1 slots. [INFO] 2017-01-30 17:39:50,861 Workflow Completion Status 37/56 in ( ...... 66%) tasks completed. slurmstepd: Job 4910074 exceeded memory limit (52468956 > 52428800), being killed slurmstepd: *** JOB 4910074 ON c5a-s17 CANCELLED AT 2017-01-30T17:47:40 *** slurmstepd: Exceeded job memory limit slurmstepd: Exceeded step memory limit at some point.
All logs seem to exit with code 0:
Code:
tail */*.log -n 1 ==> P_AssembleUnitig/genFrgFile.log <== Task genFrgFile with nproc 1 with exit code 0. ==> P_AssembleUnitig/runCaToUnitig.log <== exit ${rcode} ==> P_AssembleUnitig/writeRunCASpec.log <== Task writeRunCASpec with nproc 1 with exit code 0. ==> P_Fetch/adapterRpt.log <== Task adapterRpt with nproc 1 with exit code 0. ==> P_Fetch/overviewRpt.log <== Task overviewRpt with nproc 1 with exit code 0. ==> P_Fetch/toFofn.log <== Task toFofn with nproc 1 with exit code 0. ==> P_Filter/filter_001of009.log <== Task filter_001of009 with nproc 2 with exit code 0. ==> P_Filter/filter_002of009.log <== Task filter_002of009 with nproc 2 with exit code 0. ==> P_Filter/filter_003of009.log <== Task filter_003of009 with nproc 2 with exit code 0. ==> P_Filter/filter_004of009.log <== Task filter_004of009 with nproc 2 with exit code 0. ==> P_Filter/filter_005of009.log <== Task filter_005of009 with nproc 2 with exit code 0. ==> P_Filter/filter_006of009.log <== Task filter_006of009 with nproc 2 with exit code 0. ==> P_Filter/filter_007of009.log <== Task filter_007of009 with nproc 2 with exit code 0. ==> P_Filter/filter_008of009.log <== Task filter_008of009 with nproc 2 with exit code 0. ==> P_Filter/filter_009of009.log <== Task filter_009of009 with nproc 2 with exit code 0. ==> P_Filter/filter.plsFofn.Scatter.log <== Task filter.plsFofn.Scatter with nproc 1 with exit code 0. ==> P_Filter/filter.rgnFofn.Gather.log <== Task filter.rgnFofn.Gather with nproc 1 with exit code 0. ==> P_Filter/filter.summary.Gather.log <== Task filter.summary.Gather with nproc 1 with exit code 0. ==> P_FilterReports/loadingRpt.log <== Task loadingRpt with nproc 1 with exit code 0. ==> P_FilterReports/statsRpt.log <== Task statsRpt with nproc 1 with exit code 0. ==> P_FilterReports/subreadRpt.log <== Task subreadRpt with nproc 1 with exit code 0. ==> P_Filter/subreads_001of009.log <== Task subreads_001of009 with nproc 1 with exit code 0. ==> P_Filter/subreads_002of009.log <== Task subreads_002of009 with nproc 1 with exit code 0. ==> P_Filter/subreads_003of009.log <== Task subreads_003of009 with nproc 1 with exit code 0. ==> P_Filter/subreads_004of009.log <== Task subreads_004of009 with nproc 1 with exit code 0. ==> P_Filter/subreads_005of009.log <== Task subreads_005of009 with nproc 1 with exit code 0. ==> P_Filter/subreads_006of009.log <== Task subreads_006of009 with nproc 1 with exit code 0. ==> P_Filter/subreads_007of009.log <== Task subreads_007of009 with nproc 1 with exit code 0. ==> P_Filter/subreads_008of009.log <== Task subreads_008of009 with nproc 1 with exit code 0. ==> P_Filter/subreads_009of009.log <== Task subreads_009of009 with nproc 1 with exit code 0. ==> P_Filter/subreads.subreadFastq.Gather.log <== Task subreads.subreadFastq.Gather with nproc 1 with exit code 0. ==> P_Filter/subreads.subreads.Gather.log <== Task subreads.subreads.Gather with nproc 1 with exit code 0. ==> P_Filter/subreadSummary.log <== Task subreadSummary with nproc 1 with exit code 0. ==> P_PreAssemblerDagcon/filterLongReadsByLength.log <== Task filterLongReadsByLength with nproc 1 with exit code 0. ==> P_PreAssemblerDagcon/hgapAlignForCorrection.log <== Task hgapAlignForCorrection with nproc 12 with exit code 0. ==> P_PreAssemblerDagcon/hgapCorrection.log <== Task hgapCorrection with nproc 12 with exit code 0. ==> P_PreAssemblerDagcon/hgapFilterM4.log <== Task hgapFilterM4 with nproc 1 with exit code 0. ==> P_PreAssemblerDagcon/preAssemblerJsonReport.log <== Task preAssemblerJsonReport with nproc 1 with exit code 0.
Code:
#!/bin/bash # Setting up SMRTpipe environment echo "Setting up ENV on $(uname -n)" for task runCaToUnitig SEYMOUR_HOME=/apps/pacbio/2.3.0.140936.p5/install/smrtanalysis_2.3.0.140936 source $SEYMOUR_HOME/etc/setup.sh # Create the local TMP dir if it doesn't exist tmp_dir=$(readlink -m "/apps/pacbio/2.3.0.140936.p5/tmpdir") if [ ! -e "$tmp_dir" ]; then stat=0 mkdir -p $tmp_dir || stat=$? if [[ $stat -ne 0 ]]; then echo "SMRTpipe Unable to create TMP dir '/apps/pacbio/2.3.0.140936.p5/tmpdir' on $(uname -n)" 1>&2 exit 1 else echo "successfully created or found TMP dir '/apps/pacbio/2.3.0.140936.p5/tmpdir'" fi elif [[ ! -d "$tmp_dir" ]]; then echo "SMRTpipe TMP /apps/pacbio/2.3.0.140936.p5/tmpdir must be a directory on $(uname -n)" 1>&2 exit 1 fi ########### TASK metadata ############# # Task : runCaToUnitig # Module : P_AssembleUnitig # Module Version : 2.2.UnknownVersion # TaskType : None # URL : task://Anonymous/P_AssembleUnitig/runCaToUnitig # createdAt : 2017-01-30 11:18:51.164802 # createdAt (UTC) : 2017-01-30 16:18:51.164828 # ncmds : 2 # LogPath : /smrtpipe2/matt_review/new/log/P_AssembleUnitig/runCaToUnitig.log # Script Path : /smrtpipe2/matt_review/new/workflow/P_AssembleUnitig/runCaToUnitig.sh # Input : /smrtpipe2/matt_review/new/data/corrected.frg # Input : /smrtpipe2/matt_review/new/data/runCA.spec # Output :/smrtpipe2/matt_review/new/data/ca_finished # ########### END TASK metadata ############# cd /smrtpipe2/matt_review/new/log/P_AssembleUnitig # Writing to log file cat /smrtpipe2/matt_review/new/workflow/P_AssembleUnitig/runCaToUnitig.sh >> /smrtpip e2/matt_review/new/log/P_AssembleUnitig/runCaToUnitig.log; echo "Running task://Anonymous/P_AssembleUnitig/runCaToUnitig on $(uname -a)" echo "Started on $(date -u)" echo 'Validating existence of Input Files' if [ -e /smrtpipe2/matt_review/new/data/corrected.frg ] then echo 'Successfully found /smrtpipe2/matt_review/new/data/corrected.frg' else echo 'WARNING: Unable to find necessary input file, or dir /smrtpipe2/matt_review/new/data/corrected.frg.' fi if [ -e /smrtpipe2/matt_review/new/data/runCA.spec ] then echo 'Successfully found /smrtpipe2/matt_review/new/data/runCA.spec' else echo 'WARNING: Unable to find necessary input file, or dir /smrtpipe2/matt_review/new/data/runCA.spec.' fi echo 'Successfully validated input files' # Task runCaToUnitig commands: # Completed writing Task runCaToUnitig commands # Task 1 runCA -d /smrtpipe2/matt_review/new/data -p celera-assembler -s /smrtpipe2/matt_revie w/new/data/runCA.spec /smrtpipe2/matt_review/new/data/corrected.frg || exit $? echo "Task 1 completed at $(date)" # Task 2 touch /smrtpipe2/matt_review/new/data/ca_finished || exit $? echo "Task 2 completed at $(date)" rcode=$? echo "Finished on $(date -u)" echo "Task runCaToUnitig with nproc 1 with exit code ${rcode}." exit ${rcode}
Thanks