Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
Collapse
 
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
Previous template Next
  • #1

    Invalid Whitelist Format (filtering step)

    Hello,

    I successfully got an assembly using the whitelisting method for one of my eight BACs a couple days ago.

    I then generated whitelists for my other 7 BACs, and attempted to repeat the HGAP assembly process, however all of a sudden my whitelists are of an 'invalid format'.

    My workflow fails at the filter001_of_001 step & fetch adatperrpt fails.

    I went into the filter001_of_001 error log file and pinpointed the following message:

    Code:
    Invalid whitelist format for '' in file /opt/smrtanalysis/data/45/whitelist.txt.
    The whitelist is a .txt file, the same as my other whitelist, and has no different internal formatting, for example:

    Code:
    m130725_204700_42158_c100524682550000001823080609281380_s1_p0/0
m130725_204700_42158_c100524682550000001823080609281380_s1_p0/1
m130725_204700_42158_c100524682550000001823080609281380_s1_p0/2
m130725_204700_42158_c100524682550000001823080609281380_s1_p0/3
m130725_204700_42158_c100524682550000001823080609281380_s1_p0/4
    is the whitelist which is NOT working, and

    Code:
    m130726_012611_42158_c100524682550000001823080609281382_s1_p0/0
m130726_012611_42158_c100524682550000001823080609281382_s1_p0/1
m130726_012611_42158_c100524682550000001823080609281382_s1_p0/2
m130726_012611_42158_c100524682550000001823080609281382_s1_p0/3
m130726_012611_42158_c100524682550000001823080609281382_s1_p0/4
    is the whitelist which IS working.

    They are both named whitelist.txt and I have specified this in my run parameters, they just lie in different directories ../45/whitelist.txt and ../102/whitelist.txt.

    I have re-run my original whitelist run and it still works (protocol seems to be fine), but when I run the same protocol with a different whitelist I recieve the same error.

    Here are more relevant log files:


    Code:
    #!/bin/bash
# Setting up SMRTpipe environment
echo "Setting up ENV on $(uname -n)" for task filter_001of001

SEYMOUR_HOME=/opt/smrtanalysis/current
source $SEYMOUR_HOME/etc/setup.sh

# Create the local TMP dir if it doesn't exist
tmp_dir=$(readlink -m "/opt/smrtanalysis/tmpdir")
if [ ! -e "$tmp_dir" ]; then
   stat=0
   mkdir -p $tmp_dir || stat=$?
   if [[ $stat -ne 0 ]]; then
       echo "SMRTpipe Unable to create TMP dir '/opt/smrtanalysis/tmpdir' on $(uname -n)" 1>&2
       exit 1
   else
       echo "successfully created or found TMP dir '/opt/smrtanalysis/tmpdir'"
   fi
elif [[ ! -d "$tmp_dir" ]]; then
   echo "SMRTpipe TMP /opt/smrtanalysis/tmpdir must be a directory on $(uname -n)" 1>&2
   exit 1
fi

########### TASK metadata #############
# Task            : filter_001of001
# Module          : P_Filter
# Module Version  : 1.7.133178
# TaskType        : None
# URL             : task://016450/P_Filter/filter_001of001
# createdAt       : 2014-07-02 13:32:29.396088
# createdAt (UTC) : 2014-07-02 17:32:29.396095
# ncmds           : 1
# LogPath         : /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log
# Script Path     : /opt/smrtanalysis/current/common/jobs/016/016450/workflow/P_Filter/filter_001of001.sh

# Input       : /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn
# Output      : /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv
# Output      : /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions
# Output      : /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn
#
########### END TASK metadata #############

# Writing to log file
cat /opt/smrtanalysis/current/common/jobs/016/016450/workflow/P_Filter/filter_001of001.sh >> /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log;



echo "Running task://016450/P_Filter/filter_001of001 on $(uname -a)"

echo "Started on $(date -u)"
echo 'Validating existence of Input Files'
if [ -e /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn ]
then
echo 'Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn'
else
echo 'WARNING: Unable to find necessary input file, or dir /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn.'
fi
echo 'Successfully validated input files'

# Task filter_001of001 commands:


# Completed writing Task filter_001of001 commands


# Task 1
filter_plsh5.py --debug --filter='MinReadScore=0.8000,MinSRL=500,MinRL=100,ReadWhitelist=/opt/smrtanalysis/data/45/whitelist.txt' --trim='True' --outputDir=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions --outputSummary=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv --outputFofn=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn || exit $?
echo "Task 1 completed at $(date)"



rcode=$?
echo "Finished on $(date -u)"
echo "Task filter_001of001 with nproc 2 with exit code ${rcode}."
exit ${rcode}# Writing stdout and stderr from Popen:
Setting up ENV on nick for task filter_001of001
Running task://016450/P_Filter/filter_001of001 on Linux nick 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Started on Wed Jul  2 17:32:43 UTC 2014
Validating existence of Input Files
Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn
Successfully validated input files
No handlers could be found for logger "pbfilter.filter_plsh5"
Traceback (most recent call last):
  File "/opt/smrtanalysis/current/analysis/bin/filter_plsh5.py", line 8, in <module>
    load_entry_point('pbfilter==1.2', 'console_scripts', 'filter_plsh5.py')()
  File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 636, in main
    filters.extend(_to_filters(filter_str))
  File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 581, in _to_filters
    filters.append(SUPPORTED_FILTERS[filterName][2](threshold))
  File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 392, in __init__
    raise ValueError(emsg)
ValueError: Invalid whitelist format for '' in file /opt/smrtanalysis/data/45/whitelist.txt.
    THE END OF MY MASTER.LOG
    Code:
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > ########### END TASK metadata #############
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Writing to log file
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > cat /opt/smrtanalysis/current/common/jobs/016/016450/workflow/P_Filter/filter_001of001.sh >> /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log;
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Running task://016450/P_Filter/filter_001of001 on $(uname -a)"
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Started on $(date -u)"
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'Validating existence of Input Files'
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > if [ -e /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn ]
[ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > then
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn'
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > else
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'WARNING: Unable to find necessary input file, or dir /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn.'
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > fi
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'Successfully validated input files'
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Task filter_001of001 commands:
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Completed writing Task filter_001of001 commands
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Task 1
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > filter_plsh5.py --debug --filter='MinReadScore=0.8000,MinSRL=500,MinRL=100,ReadWhitelist=/opt/smrtanalysis/data/45/whitelist.txt' --trim='True' --outputDir=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions --outputSummary=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv --outputFofn=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn || exit $?
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Task 1 completed at $(date)"
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > rcode=$?
[ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Finished on $(date -u)"
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Task filter_001of001 with nproc 2 with exit code ${rcode}."
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > exit ${rcode}# Writing stdout and stderr from Popen:
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Setting up ENV on nick for task filter_001of001
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Running task://016450/P_Filter/filter_001of001 on Linux nick 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Started on Wed Jul  2 17:32:43 UTC 2014
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Validating existence of Input Files
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Successfully validated input files
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > No handlers could be found for logger "pbfilter.filter_plsh5"
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Traceback (most recent call last):
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/analysis/bin/filter_plsh5.py", line 8, in <module>
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > load_entry_point('pbfilter==1.2', 'console_scripts', 'filter_plsh5.py')()
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 636, in main
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > filters.extend(_to_filters(filter_str))
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 581, in _to_filters
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > filters.append(SUPPORTED_FILTERS[filterName][2](threshold))
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 392, in __init__
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > raise ValueError(emsg)
[ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > ValueError: Invalid whitelist format for '' in file /opt/smrtanalysis/data/45/whitelist.txt.
[DEBUG] 2014-07-02 13:32:48,253 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 372] Grabbed task://016450/P_Filter/filter_001of001 with message started, runflag: 1 from Queue
[INFO] 2014-07-02 13:32:48,254 [smrtpipe.status refreshTargets 419] Starting task://016450/P_Filter/filter_001of001
[DEBUG] 2014-07-02 13:32:48,254 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 372] Grabbed task://016450/P_Filter/filter_001of001 with message fail from Queue
[ERROR] 2014-07-02 13:32:48,254 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_Filter/filter_001of001
[INFO] 2014-07-02 13:32:48,255 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 323] Exiting Workflow while loop. No Jobs to be submitted and No alive threads.
[INFO] 2014-07-02 13:32:48,255 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/genFrgFile, None
[INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/getUnitigs, None
[INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/runCaToUnitig, None
[INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/unitigConsensus, None
[INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/writeRunCASpec, None
[INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/callConsensus, None
[INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/enrichAlnSummary, None
[INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/polishedJsonReport, None
[INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/topCorrectionsJsonReport, None
[INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/variantsJsonReport, None
[INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/zipPolishedFasta, None
[INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/adapterRpt, fail
[INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/getChemistry, done
[INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/overviewRpt, done
[INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/toFofn, done
[INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter.plsFofn.Scatter, done
[INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter.rgnFofn.Gather, None
[INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter.summary.Gather, None
[INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter_001of001, fail
[INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreadSummary, None
[INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreads.subreadFastq.Gather, None
[INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreads.subreads.Gather, None
[INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreads_001of001, None
[INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_FilterReports/loadingRpt, None
[INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_FilterReports/statsRpt, None
[INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_FilterReports/subreadRpt, None
[INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/align, None
[INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/covGFF, None
[INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/gff2Bed, None
[INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/loadChemistry, None
[INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/repack, None
[INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/samBam, None
[INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/sort, None
[INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/unmapped, None
[INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_MappingReports/coverageJsonReport, None
[INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_MappingReports/statsJsonReport, None
[INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/filterLongReadsByLength, None
[INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection, None
[INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/hgapCorrection, None
[INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/hgapFilterM4, None
[INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/preAssemblerJsonReport, None
[INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_ReferenceUploader/runUploaderUnitig, None
[DEBUG] 2014-07-02 13:32:48,262 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 448] Number of task2thead 6
[INFO] 2014-07-02 13:32:48,262 [smrtpipe.status execute 627] Found 2 failed tasks.
[INFO] 2014-07-02 13:32:48,263 [smrtpipe.status execute 629] task adapterRpt FAILED 
[DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 630] task adapterRpt FAILED with error task://016450/P_Fetch/adapterRpt Failed
[DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 631] task adapterRpt FAILED with errors {}
[INFO] 2014-07-02 13:32:48,263 [smrtpipe.status execute 629] task filter_001of001 FAILED 
[DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 630] task filter_001of001 FAILED with error task://016450/P_Filter/filter_001of001 Failed
[DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 631] task filter_001of001 FAILED with errors {'Unable to locate BaseCalls within hdf5 file': 'No Base Calls found in Primary; Pulse2Base may have failed.'}
[ERROR] 2014-07-02 13:32:48,264 [SMRTpipe.engine.SmrtPipeWorkflow execute 637] task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
[DEBUG] 2014-07-02 13:32:48,264 [SMRTpipe.engine.SmrtPipeWorkflow _update 602] Updating Workflow state
[DEBUG] 2014-07-02 13:32:48,274 [SMRTpipe.engine.SmrtPipeTasks _extractComputeTime 864] <P_Fetch/adapterRpt> Found Started : Started on Wed Jul  2 17:32:35 UTC 2014
[WARNING] 2014-07-02 13:32:48,275 [SMRTpipe.engine.SmrtPipeTasks computeTime 884] Unable to extract computeTime from /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Fetch/adapterRpt.log.
[DEBUG] 2014-07-02 13:32:48,280 [SMRTpipe.engine.SmrtPipeTasks _extractComputeTime 864] <P_Filter/filter_001of001> Found Started : Started on Wed Jul  2 17:32:43 UTC 2014
[WARNING] 2014-07-02 13:32:48,280 [SMRTpipe.engine.SmrtPipeTasks computeTime 884] Unable to extract computeTime from /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log.
[DEBUG] 2014-07-02 13:32:48,318 [SMRTpipe.engine.SmrtPipeWorkflow _writeWorkflow 578] Writing DAG to /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.rdf
[DEBUG] 2014-07-02 13:32:48,321 [SMRTpipe.engine.SmrtPipeWorkflow _writeWorkflow 578] Writing DAG to /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.dot
[DEBUG] 2014-07-02 13:32:48,447 [SMRTpipe.engine.SmrtPipeWorkflow _writeWorkflow 578] Writing DAG to /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.dot
[DEBUG] 2014-07-02 13:32:48,452 [SMRTpipe.SmrtWorker backticks 188] Running on nick with cmd dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg; addSVGLinks.py --size 1400,770 /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg
[DEBUG] 2014-07-02 13:32:48,501 [SMRTpipe.SmrtWorker backticks 212] Successful output (Return code = 0) in 0.05 sec (0.00 min) of dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg; addSVGLinks.py --size 1400,770 /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg
[DEBUG] 2014-07-02 13:32:48,506 [SMRTpipe.SmrtWorker backticks 188] Running on nick with cmd dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg; addSVGLinks.py --size 2800,1540 --linkTasks /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg
[DEBUG] 2014-07-02 13:32:48,557 [SMRTpipe.SmrtWorker backticks 212] Successful output (Return code = 0) in 0.05 sec (0.00 min) of dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg; addSVGLinks.py --size 2800,1540 --linkTasks /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg
[DEBUG] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeWorkflow cleanChunkedFiles 607] cleaning up chunked Workflow Workflow files
[WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn for deleting.
[DEBUG] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 202] Removing chunk file /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn
[WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv for deleting.
[WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_subreads.chunk001of001.fasta for deleting.
[WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_subreads.chunk001of001.fastq for deleting.
[ERROR] 2014-07-02 13:32:48,558 [SMRTpipe.SmrtPipeMain run 608] SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
Traceback (most recent call last):
  File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/SmrtPipeMain.py", line 571, in run
    self._runTasks(pModules)
  File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/SmrtPipeMain.py", line 300, in _runTasks
    workflow.execute()
  File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/engine/SmrtPipeWorkflow.py", line 640, in execute
    raise SmrtExit(str(e))
SmrtExit: SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
[ERROR] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain exit 690] SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
[INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain exit 692] Beginning exit process
[INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain shutdown 716] beginning Shutdown Process
[INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain _generate_final_report 651] Writing final TOC page with error message : SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
[DEBUG] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService merge_from_serialized_datastore 277] data.items.pickle merging item <DataItem name:Chemistry group:General >
[DEBUG] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService merge_from_serialized_datastore 281] ('Chemistry', 'General')
[DEBUG] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService generateFinalReport 722] jobInfo <JobInfo id:016450 name:45 Assembly 4 > dataUrl xml:/opt/smrtanalysis/current/common/jobs/016/016450/input.xml paramsFile /opt/smrtanalysis/current/common/jobs/016/016450/settings.xml
[INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService generateFinalReport 757] Skipping JNLP creation because no job id or no cmp.h5
[DEBUG] 2014-07-02 13:32:48,560 [SMRTpipe.SmrtReportService _generateIndexHtml 699] Writing Report XML to /opt/smrtanalysis/current/common/jobs/016/016450/toc.xml
[DEBUG] 2014-07-02 13:32:48,560 [SMRTpipe.SmrtReportService _generateIndexHtml 708] Generating html file from xsl /opt/smrtanalysis/current/analysis/etc/xsl/toc.xsl
[DEBUG] 2014-07-02 13:32:48,564 [SMRTpipe.SmrtWorker backticks 188] Running on nick with cmd saxonb9 -xsl:/opt/smrtanalysis/current/analysis/etc/xsl/toc.xsl -s:/opt/smrtanalysis/current/common/jobs/016/016450/toc.xml -o:/opt/smrtanalysis/current/common/jobs/016/016450/index.html
[DEBUG] 2014-07-02 13:32:49,368 [SMRTpipe.SmrtWorker backticks 212] Successful output (Return code = 0) in 0.80 sec (0.01 min) of saxonb9 -xsl:/opt/smrtanalysis/current/analysis/etc/xsl/toc.xsl -s:/opt/smrtanalysis/current/common/jobs/016/016450/toc.xml -o:/opt/smrtanalysis/current/common/jobs/016/016450/index.html
[DEBUG] 2014-07-02 13:32:49,368 [SMRTpipe.SmrtReportService _toRdf 360] Adding report node <ReportNode group:General title:Workflow > to metadata.rdf.
[DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _toRdf 360] Adding report node <ReportNode group:Diagnostic title:Json Overview Report > to metadata.rdf.
[DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _toRdf 396] Adding Auxiliary Files to RDF.
[DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _toRdf 437] No barcodeFasta to add to ReportService
[DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _pullJsonAttributes 579] Json Graph attributes: [GraphAttribute name:SMRT Cells value:1 id:overview.ncells, GraphAttribute name:Movies value:1 id:overview.nmovies]
[DEBUG] 2014-07-02 13:32:49,371 [SMRTpipe.SmrtReportService generateFinalReport 768] Writing Metadata XML to /opt/smrtanalysis/current/common/jobs/016/016450/metadata.rdf
[INFO] 2014-07-02 13:32:49,373 [SMRTpipe.SmrtPipeMain shutdown 752] Completed shutdown gracefully
[INFO] 2014-07-02 13:32:49,373 [SMRTpipe.SmrtPipeMain exit 702] Sending message Exiting smrtpipe last error: SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
[INFO] 2014-07-02 13:32:49,373 [smrtpipe.status failed 104] Sending Job 016450 failed message 'Exiting smrtpipe last error: SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed' to url http://130.15.147.123:8080/smrtportal/api/jobs/016450/status
[INFO] 2014-07-02 13:32:49,386 [SMRTpipe.HttpProgress _post_progress 151] Job Progress 'Failed' event POSTED to http://130.15.147.123:8080/smrtportal/api/jobs/016450/status
[DEBUG] 2014-07-02 13:32:49,386 [SMRTpipe.HttpProgress _post_progress 152] The data string is progress=%7B%22message%22%3A+%22Exiting+smrtpipe+last+error%3A+SmrtExit+task%3A%2F%2F016450%2FP_Fetch%2FadapterRpt+Failed+task%3A%2F%2F016450%2FP_Filter%2Ffilter_001of001+Failed%22%2C+%22code%22%3A+%22Failed%22%2C+%22jobStage%22%3A+null%2C+%22value%22%3A+100%2C+%22moduleName%22%3A+null%7D
[DEBUG] 2014-07-02 13:32:49,386 [SMRTpipe.SmrtPipeMain _releaseLock 368] -- Released lock (/opt/smrtanalysis/current/common/jobs/016/016450/smrt.lock) for this output directory.
[DEBUG] 2014-07-02 13:32:49,386 [SMRTpipe.SmrtPipeMain run 636] Shutting down heartbeat service
[INFO] 2014-07-02 13:32:49,386 [SMRTpipe.SmrtPipeMain run 641] completed SmrtPipeMain.run with exit code -1
[INFO] 2014-07-02 13:32:49,386 [root main 336] Failed smrtpipe version v1.85.133289 with exit code -1 in 21.26 seconds (0.35 minutes)
[INFO] 2014-07-02 13:33:28,170 [SMRTpipe.Heartbeat _startHeartbeat 65] Heartbeat attempting to stop
    I have no idea why this is happening, I have been comparing my files for hours now and can't find any differences.

    Any help is greatly appreciated!

    Thanks,
    Nick
    Tags: None
  • #2
    A mere possibility, but have you looked to see if there are any non-printable/control character differences between the two whitelist files?

    Comment

    • #3
      Hey Genomax,

      I'm pretty sure both files are identical from a character point of view -- I used the same script to generate all of the files except I just hard code in my run name (mxxxx_xxxx_xxxxxxxxx).

      The only difference between the way I generated my original (working) whitelist and my 7 new (non-working white lists) was that:

      In my original, I copy/pasted all of the printed lines directly from the terminal, into a new 'whitelist.txt' file.

      In the new ones, I used the output [(script) > whitelist.txt] argument. So I just re-generated one of my new whitelists and copy/pasted instead. I'll let you know if it works once the current job in queue is finished.

      Comment

      • #4
        Wow,

        For some reason, outputting it to a .txt file via the '>' when I ran my script caused it to mess up.

        Copy+paste directly from the terminal seemed to do the trick...

        Why would that ever occur?

        Comment

        • #5
          Did you move the whitelist file between two OS's (e.g. OS X to Unix)?

          Comment

          • #6
            No sir, everything has stayed on my Ubuntu OS (12.04)

            Comment

            • #7
              So in what way was the file different between the copy/paste and script methods (e.g. http://alvinalexander.com/blog/post/...s-in-text-file)
              Last edited by GenoMax; 07-02-2014, 11:49 AM.

              Comment

              • #8
                Neither of them had any non-ASCII characters

                Comment

                • #9
                  Glad you got it working, my guess from the error
                  Code:
                  Invalid whitelist format for ''
                  a blank line at the end of the file? Looking at the code, the line being read that is invalid is returned between the quotes:
                  Code:
                   emsg = "Invalid whitelist format for '{s}' in file {f}.".format(s=line.strip(), f=self._whitelist_file)

                  Comment

                  • #10
                    I just checked all of them that I generated using the >output and none of them have a blank line anywhere. Very strange. However, I don't want to take up your time trying to solve something that's practically already fixed. I just generated all of mine manually via copy/paste.

                    Thanks,
                    Nick

                    Comment

                    • #11
                      It's not time wasted if we can find and fix a bug

                      If you want to attach examples of the text files that work and do not work (hopefully they won't get reformatted in the attachment) I can take a look at what is happening. Alternatively I can give you my email and we can transfer the files that way.

                      Comment

                      • #12
                        Sure, I can e-mail them to you -- since the text files are >5mb and the limit here is <20kb

                        Comment

                        • #13
                          I 'figured it out'. May have been a stupid thing to do on my end, but basically the whitelist must be generated as the smrtanalysis user. Make sure you do the following:

                          Code:
                          su smrtanalyis
                          Code:
                          > enter password
                          Code:
                          /usr/local/bin/python.decon$ python whitelist.py 147_woot.align 81740 > /opt/smrtanalysis/data/147/whitelist.txt
                          Note the space between the '>' and the beginning of the directory. It didn't work when I didn't include the space.

                          Once the file is created, do not change the permissions whatsoever (chmod/chown), or it will not work. I was changing the permissions on the file to be readable/writeable by all users which I think was the problem.

                          Cheers,
                          Nick

                          Comment

                          Previous template Next

                          Latest Articles

                          Collapse
                          • seqadmin
                            Essential Discoveries and Tools in Epitranscriptomics
                            by seqadmin




                            The field of epigenetics has traditionally concentrated more on DNA and how changes like methylation and phosphorylation of histones impact gene expression and regulation. However, our increased understanding of RNA modifications and their importance in cellular processes has led to a rise in epitranscriptomics research. “Epitranscriptomics brings together the concepts of epigenetics and gene expression,” explained Adrien Leger, PhD, Principal Research Scientist...
                            Yesterday, 07:01 AM
                          • seqadmin
                            Current Approaches to Protein Sequencing
                            by seqadmin


                            Proteins are often described as the workhorses of the cell, and identifying their sequences is key to understanding their role in biological processes and disease. Currently, the most common technique used to determine protein sequences is mass spectrometry. While still a valuable tool, mass spectrometry faces several limitations and requires a highly experienced scientist familiar with the equipment to operate it. Additionally, other proteomic methods, like affinity assays, are constrained...
                            04-04-2024, 04:25 PM
                          View All

                          ad_right_rmr

                          Collapse

                          News

                          Collapse
                          Topics Statistics Last Post
                          Cancer Metastasis: A Deep Dive into Cellular Plasticity by seqadmin
                          Started by seqadmin, 04-11-2024, 12:08 PM
                          0 responses
                          57 views
                          0 likes
                          		 Last Post seqadmin
                          by seqadmin
                          04-11-2024, 12:08 PM  
                          Proteogenomic Profiles Offer New Clues in Prostate Cancer by seqadmin
                          Started by seqadmin, 04-10-2024, 10:19 PM
                          0 responses
                          53 views
                          0 likes
                          		 Last Post seqadmin
                          by seqadmin
                          04-10-2024, 10:19 PM  
                          Novel Diagnostic Assay Enhances Ovarian Cancer Detection by seqadmin
                          Started by seqadmin, 04-10-2024, 09:21 AM
                          0 responses
                          45 views
                          0 likes
                          		 Last Post seqadmin
                          by seqadmin
                          04-10-2024, 09:21 AM  
                          Evolutionary Dynamics of Centromeres: A Comparative Genomic Analysis by seqadmin
                          Started by seqadmin, 04-04-2024, 09:00 AM
                          0 responses
                          55 views
                          0 likes
                          		 Last Post seqadmin
                          by seqadmin
                          04-04-2024, 09:00 AM  
                          View All
                          Working...
                          X