Tweet
I am getting strange memory issues after runCaToUnitig stage is compeleted:
I am using 50GB memory which should be more than enough for a 10Mb organism. I get stuck with the same error while using data from 1 cell or data from 3 cells.
All logs seem to exit with code 0:
The log from runCaToUnitig.log is as follows:
Can anyone please help?
Thanks
Code:
[INFO] 2017-01-30 17:39:23,826 Workflow Completion Status 35/56 in ( ...... 62%) tasks completed. [INFO] 2017-01-30 17:39:28,836 task P_AssembleUnitig.writeRunCASpec successfully completed (4.00 sec (0.07 min)). freed 1 slots, using 1 slots. [INFO] 2017-01-30 17:39:28,837 Workflow Completion Status 36/56 in ( ...... 64%) tasks completed. [INFO] 2017-01-30 17:39:33,845 Starting task://Anonymous/P_AssembleUnitig/runCaToUnitig [INFO] 2017-01-30 17:39:50,860 task P_PreAssemblerDagcon.preAssemblerJsonReport successfully completed (25.00 sec (0.42 min)). freed 1 slots, using 1 slots. [INFO] 2017-01-30 17:39:50,861 Workflow Completion Status 37/56 in ( ...... 66%) tasks completed. slurmstepd: Job 4910074 exceeded memory limit (52468956 > 52428800), being killed slurmstepd: *** JOB 4910074 ON c5a-s17 CANCELLED AT 2017-01-30T17:47:40 *** slurmstepd: Exceeded job memory limit slurmstepd: Exceeded step memory limit at some point.
All logs seem to exit with code 0:
Code:
tail */*.log -n 1
==> P_AssembleUnitig/genFrgFile.log <==
Task genFrgFile with nproc 1 with exit code 0.
==> P_AssembleUnitig/runCaToUnitig.log <==
exit ${rcode}
==> P_AssembleUnitig/writeRunCASpec.log <==
Task writeRunCASpec with nproc 1 with exit code 0.
==> P_Fetch/adapterRpt.log <==
Task adapterRpt with nproc 1 with exit code 0.
==> P_Fetch/overviewRpt.log <==
Task overviewRpt with nproc 1 with exit code 0.
==> P_Fetch/toFofn.log <==
Task toFofn with nproc 1 with exit code 0.
==> P_Filter/filter_001of009.log <==
Task filter_001of009 with nproc 2 with exit code 0.
==> P_Filter/filter_002of009.log <==
Task filter_002of009 with nproc 2 with exit code 0.
==> P_Filter/filter_003of009.log <==
Task filter_003of009 with nproc 2 with exit code 0.
==> P_Filter/filter_004of009.log <==
Task filter_004of009 with nproc 2 with exit code 0.
==> P_Filter/filter_005of009.log <==
Task filter_005of009 with nproc 2 with exit code 0.
==> P_Filter/filter_006of009.log <==
Task filter_006of009 with nproc 2 with exit code 0.
==> P_Filter/filter_007of009.log <==
Task filter_007of009 with nproc 2 with exit code 0.
==> P_Filter/filter_008of009.log <==
Task filter_008of009 with nproc 2 with exit code 0.
==> P_Filter/filter_009of009.log <==
Task filter_009of009 with nproc 2 with exit code 0.
==> P_Filter/filter.plsFofn.Scatter.log <==
Task filter.plsFofn.Scatter with nproc 1 with exit code 0.
==> P_Filter/filter.rgnFofn.Gather.log <==
Task filter.rgnFofn.Gather with nproc 1 with exit code 0.
==> P_Filter/filter.summary.Gather.log <==
Task filter.summary.Gather with nproc 1 with exit code 0.
==> P_FilterReports/loadingRpt.log <==
Task loadingRpt with nproc 1 with exit code 0.
==> P_FilterReports/statsRpt.log <==
Task statsRpt with nproc 1 with exit code 0.
==> P_FilterReports/subreadRpt.log <==
Task subreadRpt with nproc 1 with exit code 0.
==> P_Filter/subreads_001of009.log <==
Task subreads_001of009 with nproc 1 with exit code 0.
==> P_Filter/subreads_002of009.log <==
Task subreads_002of009 with nproc 1 with exit code 0.
==> P_Filter/subreads_003of009.log <==
Task subreads_003of009 with nproc 1 with exit code 0.
==> P_Filter/subreads_004of009.log <==
Task subreads_004of009 with nproc 1 with exit code 0.
==> P_Filter/subreads_005of009.log <==
Task subreads_005of009 with nproc 1 with exit code 0.
==> P_Filter/subreads_006of009.log <==
Task subreads_006of009 with nproc 1 with exit code 0.
==> P_Filter/subreads_007of009.log <==
Task subreads_007of009 with nproc 1 with exit code 0.
==> P_Filter/subreads_008of009.log <==
Task subreads_008of009 with nproc 1 with exit code 0.
==> P_Filter/subreads_009of009.log <==
Task subreads_009of009 with nproc 1 with exit code 0.
==> P_Filter/subreads.subreadFastq.Gather.log <==
Task subreads.subreadFastq.Gather with nproc 1 with exit code 0.
==> P_Filter/subreads.subreads.Gather.log <==
Task subreads.subreads.Gather with nproc 1 with exit code 0.
==> P_Filter/subreadSummary.log <==
Task subreadSummary with nproc 1 with exit code 0.
==> P_PreAssemblerDagcon/filterLongReadsByLength.log <==
Task filterLongReadsByLength with nproc 1 with exit code 0.
==> P_PreAssemblerDagcon/hgapAlignForCorrection.log <==
Task hgapAlignForCorrection with nproc 12 with exit code 0.
==> P_PreAssemblerDagcon/hgapCorrection.log <==
Task hgapCorrection with nproc 12 with exit code 0.
==> P_PreAssemblerDagcon/hgapFilterM4.log <==
Task hgapFilterM4 with nproc 1 with exit code 0.
==> P_PreAssemblerDagcon/preAssemblerJsonReport.log <==
Task preAssemblerJsonReport with nproc 1 with exit code 0.
Code:
#!/bin/bash
# Setting up SMRTpipe environment
echo "Setting up ENV on $(uname -n)" for task runCaToUnitig
SEYMOUR_HOME=/apps/pacbio/2.3.0.140936.p5/install/smrtanalysis_2.3.0.140936
source $SEYMOUR_HOME/etc/setup.sh
# Create the local TMP dir if it doesn't exist
tmp_dir=$(readlink -m "/apps/pacbio/2.3.0.140936.p5/tmpdir")
if [ ! -e "$tmp_dir" ]; then
stat=0
mkdir -p $tmp_dir || stat=$?
if [[ $stat -ne 0 ]]; then
echo "SMRTpipe Unable to create TMP dir '/apps/pacbio/2.3.0.140936.p5/tmpdir' on $(uname -n)" 1>&2
exit 1
else
echo "successfully created or found TMP dir '/apps/pacbio/2.3.0.140936.p5/tmpdir'"
fi
elif [[ ! -d "$tmp_dir" ]]; then
echo "SMRTpipe TMP /apps/pacbio/2.3.0.140936.p5/tmpdir must be a directory on $(uname -n)" 1>&2
exit 1
fi
########### TASK metadata #############
# Task : runCaToUnitig
# Module : P_AssembleUnitig
# Module Version : 2.2.UnknownVersion
# TaskType : None
# URL : task://Anonymous/P_AssembleUnitig/runCaToUnitig
# createdAt : 2017-01-30 11:18:51.164802
# createdAt (UTC) : 2017-01-30 16:18:51.164828
# ncmds : 2
# LogPath : /smrtpipe2/matt_review/new/log/P_AssembleUnitig/runCaToUnitig.log
# Script Path : /smrtpipe2/matt_review/new/workflow/P_AssembleUnitig/runCaToUnitig.sh
# Input : /smrtpipe2/matt_review/new/data/corrected.frg
# Input : /smrtpipe2/matt_review/new/data/runCA.spec
# Output :/smrtpipe2/matt_review/new/data/ca_finished
#
########### END TASK metadata #############
cd /smrtpipe2/matt_review/new/log/P_AssembleUnitig
# Writing to log file
cat /smrtpipe2/matt_review/new/workflow/P_AssembleUnitig/runCaToUnitig.sh >> /smrtpip
e2/matt_review/new/log/P_AssembleUnitig/runCaToUnitig.log;
echo "Running task://Anonymous/P_AssembleUnitig/runCaToUnitig on $(uname -a)"
echo "Started on $(date -u)"
echo 'Validating existence of Input Files'
if [ -e /smrtpipe2/matt_review/new/data/corrected.frg ]
then
echo 'Successfully found /smrtpipe2/matt_review/new/data/corrected.frg'
else
echo 'WARNING: Unable to find necessary input file, or dir /smrtpipe2/matt_review/new/data/corrected.frg.'
fi
if [ -e /smrtpipe2/matt_review/new/data/runCA.spec ]
then
echo 'Successfully found /smrtpipe2/matt_review/new/data/runCA.spec'
else
echo 'WARNING: Unable to find necessary input file, or dir /smrtpipe2/matt_review/new/data/runCA.spec.'
fi
echo 'Successfully validated input files'
# Task runCaToUnitig commands:
# Completed writing Task runCaToUnitig commands
# Task 1
runCA -d /smrtpipe2/matt_review/new/data -p celera-assembler -s /smrtpipe2/matt_revie
w/new/data/runCA.spec /smrtpipe2/matt_review/new/data/corrected.frg || exit $?
echo "Task 1 completed at $(date)"
# Task 2
touch /smrtpipe2/matt_review/new/data/ca_finished || exit $?
echo "Task 2 completed at $(date)"
rcode=$?
echo "Finished on $(date -u)"
echo "Task runCaToUnitig with nproc 1 with exit code ${rcode}."
exit ${rcode}
Thanks