Hello,
I successfully got an assembly using the whitelisting method for one of my eight BACs a couple days ago.
I then generated whitelists for my other 7 BACs, and attempted to repeat the HGAP assembly process, however all of a sudden my whitelists are of an 'invalid format'.
My workflow fails at the filter001_of_001 step & fetch adatperrpt fails.
I went into the filter001_of_001 error log file and pinpointed the following message:
The whitelist is a .txt file, the same as my other whitelist, and has no different internal formatting, for example:
is the whitelist which is NOT working, and
is the whitelist which IS working.
They are both named whitelist.txt and I have specified this in my run parameters, they just lie in different directories ../45/whitelist.txt and ../102/whitelist.txt.
I have re-run my original whitelist run and it still works (protocol seems to be fine), but when I run the same protocol with a different whitelist I recieve the same error.
Here are more relevant log files:
THE END OF MY MASTER.LOG
I have no idea why this is happening, I have been comparing my files for hours now and can't find any differences.
Any help is greatly appreciated!
Thanks,
Nick
I successfully got an assembly using the whitelisting method for one of my eight BACs a couple days ago.
I then generated whitelists for my other 7 BACs, and attempted to repeat the HGAP assembly process, however all of a sudden my whitelists are of an 'invalid format'.
My workflow fails at the filter001_of_001 step & fetch adatperrpt fails.
I went into the filter001_of_001 error log file and pinpointed the following message:
Code:
Invalid whitelist format for '' in file /opt/smrtanalysis/data/45/whitelist.txt.
Code:
m130725_204700_42158_c100524682550000001823080609281380_s1_p0/0 m130725_204700_42158_c100524682550000001823080609281380_s1_p0/1 m130725_204700_42158_c100524682550000001823080609281380_s1_p0/2 m130725_204700_42158_c100524682550000001823080609281380_s1_p0/3 m130725_204700_42158_c100524682550000001823080609281380_s1_p0/4
Code:
m130726_012611_42158_c100524682550000001823080609281382_s1_p0/0 m130726_012611_42158_c100524682550000001823080609281382_s1_p0/1 m130726_012611_42158_c100524682550000001823080609281382_s1_p0/2 m130726_012611_42158_c100524682550000001823080609281382_s1_p0/3 m130726_012611_42158_c100524682550000001823080609281382_s1_p0/4
They are both named whitelist.txt and I have specified this in my run parameters, they just lie in different directories ../45/whitelist.txt and ../102/whitelist.txt.
I have re-run my original whitelist run and it still works (protocol seems to be fine), but when I run the same protocol with a different whitelist I recieve the same error.
Here are more relevant log files:
Code:
#!/bin/bash # Setting up SMRTpipe environment echo "Setting up ENV on $(uname -n)" for task filter_001of001 SEYMOUR_HOME=/opt/smrtanalysis/current source $SEYMOUR_HOME/etc/setup.sh # Create the local TMP dir if it doesn't exist tmp_dir=$(readlink -m "/opt/smrtanalysis/tmpdir") if [ ! -e "$tmp_dir" ]; then stat=0 mkdir -p $tmp_dir || stat=$? if [[ $stat -ne 0 ]]; then echo "SMRTpipe Unable to create TMP dir '/opt/smrtanalysis/tmpdir' on $(uname -n)" 1>&2 exit 1 else echo "successfully created or found TMP dir '/opt/smrtanalysis/tmpdir'" fi elif [[ ! -d "$tmp_dir" ]]; then echo "SMRTpipe TMP /opt/smrtanalysis/tmpdir must be a directory on $(uname -n)" 1>&2 exit 1 fi ########### TASK metadata ############# # Task : filter_001of001 # Module : P_Filter # Module Version : 1.7.133178 # TaskType : None # URL : task://016450/P_Filter/filter_001of001 # createdAt : 2014-07-02 13:32:29.396088 # createdAt (UTC) : 2014-07-02 17:32:29.396095 # ncmds : 1 # LogPath : /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log # Script Path : /opt/smrtanalysis/current/common/jobs/016/016450/workflow/P_Filter/filter_001of001.sh # Input : /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn # Output : /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv # Output : /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions # Output : /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn # ########### END TASK metadata ############# # Writing to log file cat /opt/smrtanalysis/current/common/jobs/016/016450/workflow/P_Filter/filter_001of001.sh >> /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log; echo "Running task://016450/P_Filter/filter_001of001 on $(uname -a)" echo "Started on $(date -u)" echo 'Validating existence of Input Files' if [ -e /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn ] then echo 'Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn' else echo 'WARNING: Unable to find necessary input file, or dir /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn.' fi echo 'Successfully validated input files' # Task filter_001of001 commands: # Completed writing Task filter_001of001 commands # Task 1 filter_plsh5.py --debug --filter='MinReadScore=0.8000,MinSRL=500,MinRL=100,ReadWhitelist=/opt/smrtanalysis/data/45/whitelist.txt' --trim='True' --outputDir=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions --outputSummary=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv --outputFofn=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn || exit $? echo "Task 1 completed at $(date)" rcode=$? echo "Finished on $(date -u)" echo "Task filter_001of001 with nproc 2 with exit code ${rcode}." exit ${rcode}# Writing stdout and stderr from Popen: Setting up ENV on nick for task filter_001of001 Running task://016450/P_Filter/filter_001of001 on Linux nick 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Started on Wed Jul 2 17:32:43 UTC 2014 Validating existence of Input Files Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn Successfully validated input files No handlers could be found for logger "pbfilter.filter_plsh5" Traceback (most recent call last): File "/opt/smrtanalysis/current/analysis/bin/filter_plsh5.py", line 8, in <module> load_entry_point('pbfilter==1.2', 'console_scripts', 'filter_plsh5.py')() File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 636, in main filters.extend(_to_filters(filter_str)) File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 581, in _to_filters filters.append(SUPPORTED_FILTERS[filterName][2](threshold)) File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 392, in __init__ raise ValueError(emsg) ValueError: Invalid whitelist format for '' in file /opt/smrtanalysis/data/45/whitelist.txt.
Code:
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > ########### END TASK metadata ############# [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Writing to log file [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > cat /opt/smrtanalysis/current/common/jobs/016/016450/workflow/P_Filter/filter_001of001.sh >> /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log; [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Running task://016450/P_Filter/filter_001of001 on $(uname -a)" [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Started on $(date -u)" [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'Validating existence of Input Files' [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > if [ -e /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn ] [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > then [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn' [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > else [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'WARNING: Unable to find necessary input file, or dir /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn.' [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > fi [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'Successfully validated input files' [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Task filter_001of001 commands: [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Completed writing Task filter_001of001 commands [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Task 1 [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > filter_plsh5.py --debug --filter='MinReadScore=0.8000,MinSRL=500,MinRL=100,ReadWhitelist=/opt/smrtanalysis/data/45/whitelist.txt' --trim='True' --outputDir=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions --outputSummary=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv --outputFofn=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn || exit $? [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Task 1 completed at $(date)" [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > rcode=$? [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Finished on $(date -u)" [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Task filter_001of001 with nproc 2 with exit code ${rcode}." [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > exit ${rcode}# Writing stdout and stderr from Popen: [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Setting up ENV on nick for task filter_001of001 [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Running task://016450/P_Filter/filter_001of001 on Linux nick 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Started on Wed Jul 2 17:32:43 UTC 2014 [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Validating existence of Input Files [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Successfully validated input files [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > No handlers could be found for logger "pbfilter.filter_plsh5" [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Traceback (most recent call last): [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/analysis/bin/filter_plsh5.py", line 8, in <module> [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > load_entry_point('pbfilter==1.2', 'console_scripts', 'filter_plsh5.py')() [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 636, in main [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > filters.extend(_to_filters(filter_str)) [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 581, in _to_filters [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > filters.append(SUPPORTED_FILTERS[filterName][2](threshold)) [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 392, in __init__ [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > raise ValueError(emsg) [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > ValueError: Invalid whitelist format for '' in file /opt/smrtanalysis/data/45/whitelist.txt. [DEBUG] 2014-07-02 13:32:48,253 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 372] Grabbed task://016450/P_Filter/filter_001of001 with message started, runflag: 1 from Queue [INFO] 2014-07-02 13:32:48,254 [smrtpipe.status refreshTargets 419] Starting task://016450/P_Filter/filter_001of001 [DEBUG] 2014-07-02 13:32:48,254 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 372] Grabbed task://016450/P_Filter/filter_001of001 with message fail from Queue [ERROR] 2014-07-02 13:32:48,254 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_Filter/filter_001of001 [INFO] 2014-07-02 13:32:48,255 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 323] Exiting Workflow while loop. No Jobs to be submitted and No alive threads. [INFO] 2014-07-02 13:32:48,255 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/genFrgFile, None [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/getUnitigs, None [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/runCaToUnitig, None [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/unitigConsensus, None [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/writeRunCASpec, None [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/callConsensus, None [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/enrichAlnSummary, None [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/polishedJsonReport, None [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/topCorrectionsJsonReport, None [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/variantsJsonReport, None [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/zipPolishedFasta, None [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/adapterRpt, fail [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/getChemistry, done [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/overviewRpt, done [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/toFofn, done [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter.plsFofn.Scatter, done [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter.rgnFofn.Gather, None [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter.summary.Gather, None [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter_001of001, fail [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreadSummary, None [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreads.subreadFastq.Gather, None [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreads.subreads.Gather, None [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreads_001of001, None [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_FilterReports/loadingRpt, None [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_FilterReports/statsRpt, None [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_FilterReports/subreadRpt, None [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/align, None [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/covGFF, None [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/gff2Bed, None [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/loadChemistry, None [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/repack, None [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/samBam, None [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/sort, None [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/unmapped, None [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_MappingReports/coverageJsonReport, None [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_MappingReports/statsJsonReport, None [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/filterLongReadsByLength, None [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection, None [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/hgapCorrection, None [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/hgapFilterM4, None [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/preAssemblerJsonReport, None [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_ReferenceUploader/runUploaderUnitig, None [DEBUG] 2014-07-02 13:32:48,262 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 448] Number of task2thead 6 [INFO] 2014-07-02 13:32:48,262 [smrtpipe.status execute 627] Found 2 failed tasks. [INFO] 2014-07-02 13:32:48,263 [smrtpipe.status execute 629] task adapterRpt FAILED [DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 630] task adapterRpt FAILED with error task://016450/P_Fetch/adapterRpt Failed [DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 631] task adapterRpt FAILED with errors {} [INFO] 2014-07-02 13:32:48,263 [smrtpipe.status execute 629] task filter_001of001 FAILED [DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 630] task filter_001of001 FAILED with error task://016450/P_Filter/filter_001of001 Failed [DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 631] task filter_001of001 FAILED with errors {'Unable to locate BaseCalls within hdf5 file': 'No Base Calls found in Primary; Pulse2Base may have failed.'} [ERROR] 2014-07-02 13:32:48,264 [SMRTpipe.engine.SmrtPipeWorkflow execute 637] task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed [DEBUG] 2014-07-02 13:32:48,264 [SMRTpipe.engine.SmrtPipeWorkflow _update 602] Updating Workflow state [DEBUG] 2014-07-02 13:32:48,274 [SMRTpipe.engine.SmrtPipeTasks _extractComputeTime 864] <P_Fetch/adapterRpt> Found Started : Started on Wed Jul 2 17:32:35 UTC 2014 [WARNING] 2014-07-02 13:32:48,275 [SMRTpipe.engine.SmrtPipeTasks computeTime 884] Unable to extract computeTime from /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Fetch/adapterRpt.log. [DEBUG] 2014-07-02 13:32:48,280 [SMRTpipe.engine.SmrtPipeTasks _extractComputeTime 864] <P_Filter/filter_001of001> Found Started : Started on Wed Jul 2 17:32:43 UTC 2014 [WARNING] 2014-07-02 13:32:48,280 [SMRTpipe.engine.SmrtPipeTasks computeTime 884] Unable to extract computeTime from /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log. [DEBUG] 2014-07-02 13:32:48,318 [SMRTpipe.engine.SmrtPipeWorkflow _writeWorkflow 578] Writing DAG to /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.rdf [DEBUG] 2014-07-02 13:32:48,321 [SMRTpipe.engine.SmrtPipeWorkflow _writeWorkflow 578] Writing DAG to /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.dot [DEBUG] 2014-07-02 13:32:48,447 [SMRTpipe.engine.SmrtPipeWorkflow _writeWorkflow 578] Writing DAG to /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.dot [DEBUG] 2014-07-02 13:32:48,452 [SMRTpipe.SmrtWorker backticks 188] Running on nick with cmd dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg; addSVGLinks.py --size 1400,770 /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg [DEBUG] 2014-07-02 13:32:48,501 [SMRTpipe.SmrtWorker backticks 212] Successful output (Return code = 0) in 0.05 sec (0.00 min) of dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg; addSVGLinks.py --size 1400,770 /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg [DEBUG] 2014-07-02 13:32:48,506 [SMRTpipe.SmrtWorker backticks 188] Running on nick with cmd dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg; addSVGLinks.py --size 2800,1540 --linkTasks /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg [DEBUG] 2014-07-02 13:32:48,557 [SMRTpipe.SmrtWorker backticks 212] Successful output (Return code = 0) in 0.05 sec (0.00 min) of dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg; addSVGLinks.py --size 2800,1540 --linkTasks /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg [DEBUG] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeWorkflow cleanChunkedFiles 607] cleaning up chunked Workflow Workflow files [WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn for deleting. [DEBUG] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 202] Removing chunk file /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn [WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv for deleting. [WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_subreads.chunk001of001.fasta for deleting. [WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_subreads.chunk001of001.fastq for deleting. [ERROR] 2014-07-02 13:32:48,558 [SMRTpipe.SmrtPipeMain run 608] SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed Traceback (most recent call last): File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/SmrtPipeMain.py", line 571, in run self._runTasks(pModules) File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/SmrtPipeMain.py", line 300, in _runTasks workflow.execute() File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/engine/SmrtPipeWorkflow.py", line 640, in execute raise SmrtExit(str(e)) SmrtExit: SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed [ERROR] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain exit 690] SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed [INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain exit 692] Beginning exit process [INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain shutdown 716] beginning Shutdown Process [INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain _generate_final_report 651] Writing final TOC page with error message : SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed [DEBUG] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService merge_from_serialized_datastore 277] data.items.pickle merging item <DataItem name:Chemistry group:General > [DEBUG] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService merge_from_serialized_datastore 281] ('Chemistry', 'General') [DEBUG] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService generateFinalReport 722] jobInfo <JobInfo id:016450 name:45 Assembly 4 > dataUrl xml:/opt/smrtanalysis/current/common/jobs/016/016450/input.xml paramsFile /opt/smrtanalysis/current/common/jobs/016/016450/settings.xml [INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService generateFinalReport 757] Skipping JNLP creation because no job id or no cmp.h5 [DEBUG] 2014-07-02 13:32:48,560 [SMRTpipe.SmrtReportService _generateIndexHtml 699] Writing Report XML to /opt/smrtanalysis/current/common/jobs/016/016450/toc.xml [DEBUG] 2014-07-02 13:32:48,560 [SMRTpipe.SmrtReportService _generateIndexHtml 708] Generating html file from xsl /opt/smrtanalysis/current/analysis/etc/xsl/toc.xsl [DEBUG] 2014-07-02 13:32:48,564 [SMRTpipe.SmrtWorker backticks 188] Running on nick with cmd saxonb9 -xsl:/opt/smrtanalysis/current/analysis/etc/xsl/toc.xsl -s:/opt/smrtanalysis/current/common/jobs/016/016450/toc.xml -o:/opt/smrtanalysis/current/common/jobs/016/016450/index.html [DEBUG] 2014-07-02 13:32:49,368 [SMRTpipe.SmrtWorker backticks 212] Successful output (Return code = 0) in 0.80 sec (0.01 min) of saxonb9 -xsl:/opt/smrtanalysis/current/analysis/etc/xsl/toc.xsl -s:/opt/smrtanalysis/current/common/jobs/016/016450/toc.xml -o:/opt/smrtanalysis/current/common/jobs/016/016450/index.html [DEBUG] 2014-07-02 13:32:49,368 [SMRTpipe.SmrtReportService _toRdf 360] Adding report node <ReportNode group:General title:Workflow > to metadata.rdf. [DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _toRdf 360] Adding report node <ReportNode group:Diagnostic title:Json Overview Report > to metadata.rdf. [DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _toRdf 396] Adding Auxiliary Files to RDF. [DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _toRdf 437] No barcodeFasta to add to ReportService [DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _pullJsonAttributes 579] Json Graph attributes: [GraphAttribute name:SMRT Cells value:1 id:overview.ncells, GraphAttribute name:Movies value:1 id:overview.nmovies] [DEBUG] 2014-07-02 13:32:49,371 [SMRTpipe.SmrtReportService generateFinalReport 768] Writing Metadata XML to /opt/smrtanalysis/current/common/jobs/016/016450/metadata.rdf [INFO] 2014-07-02 13:32:49,373 [SMRTpipe.SmrtPipeMain shutdown 752] Completed shutdown gracefully [INFO] 2014-07-02 13:32:49,373 [SMRTpipe.SmrtPipeMain exit 702] Sending message Exiting smrtpipe last error: SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed [INFO] 2014-07-02 13:32:49,373 [smrtpipe.status failed 104] Sending Job 016450 failed message 'Exiting smrtpipe last error: SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed' to url http://130.15.147.123:8080/smrtportal/api/jobs/016450/status [INFO] 2014-07-02 13:32:49,386 [SMRTpipe.HttpProgress _post_progress 151] Job Progress 'Failed' event POSTED to http://130.15.147.123:8080/smrtportal/api/jobs/016450/status [DEBUG] 2014-07-02 13:32:49,386 [SMRTpipe.HttpProgress _post_progress 152] The data string is progress=%7B%22message%22%3A+%22Exiting+smrtpipe+last+error%3A+SmrtExit+task%3A%2F%2F016450%2FP_Fetch%2FadapterRpt+Failed+task%3A%2F%2F016450%2FP_Filter%2Ffilter_001of001+Failed%22%2C+%22code%22%3A+%22Failed%22%2C+%22jobStage%22%3A+null%2C+%22value%22%3A+100%2C+%22moduleName%22%3A+null%7D [DEBUG] 2014-07-02 13:32:49,386 [SMRTpipe.SmrtPipeMain _releaseLock 368] -- Released lock (/opt/smrtanalysis/current/common/jobs/016/016450/smrt.lock) for this output directory. [DEBUG] 2014-07-02 13:32:49,386 [SMRTpipe.SmrtPipeMain run 636] Shutting down heartbeat service [INFO] 2014-07-02 13:32:49,386 [SMRTpipe.SmrtPipeMain run 641] completed SmrtPipeMain.run with exit code -1 [INFO] 2014-07-02 13:32:49,386 [root main 336] Failed smrtpipe version v1.85.133289 with exit code -1 in 21.26 seconds (0.35 minutes) [INFO] 2014-07-02 13:33:28,170 [SMRTpipe.Heartbeat _startHeartbeat 65] Heartbeat attempting to stop
Any help is greatly appreciated!
Thanks,
Nick
Comment