Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    Thanks for the guiding Dr. Hall,

    That is very good, mean while we talked with the sequencing provider, and they will do resequencing and I hope that I will have better result.

    meanwhile I am trying to get familiar with the protocols I used this protocol RS_Resquensing.1 whith the cells I have,
    But it failed in the mapping process

    [ERROR] 2014-09-04 20:48:10,531 [smrtpipe.status refreshTargets 413] *** Failed task task://016452/P_Mapping/align
    [INFO] 2014-09-04 20:48:10,562 [smrtpipe.status execute 627] Found 1 failed tasks.
    [INFO] 2014-09-04 20:48:10,562 [smrtpipe.status execute 629] task align FAILED
    [ERROR] 2014-09-04 20:48:12,710 [SMRTpipe.SmrtPipeMain run 608] SmrtExit task://016452/P_Mapping/align Failed
    Traceback (most recent call last):
    Is it hardware problem , data or software?
    Last edited by Medhat; 09-05-2014, 01:09 AM.

    Comment


    • #17
      Did process fail in spite of providing a reference for the RS_resequencing protocol?

      Comment


      • #18
        yes I provided a reference of the genome and yet it failed

        Comment


        • #19
          You can check ./jobs/016/016452/logs/P_Mapping/align.log for a more meaningful log of the process that failed. How much memory does the machine have? running none distributed on a single machine can be quite memory intensive, particularly for a large genome.

          Comment


          • #20
            on my PC single machine which is 12 cpu 32G memory

            in this directory:
            /data/smrtAnalysis/current/common/jobs/016/016452/log/P_Mapping

            log is :

            # Task 1
            pbalign.py "/data/smrtAnalysis/current/common/jobs/016/016452/input.fofn" "/data/smrtAnalysis/current/common/references/Maize" "/data/smrtAnalysis/current/common/jobs/016/016452/data/aligned_reads.cmp.h5" --seed=1 --minAccuracy=0.75 --minLength=50 --concordant --algorithmOptions="-useQuality" --algorithmOptions=' -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa' --hitPolicy=randombest --tmpDir=/data/smrtAnalysis/tmpdir --nproc=11 --regionTable=/data/smrtAnalysis/current/common/jobs/016/016452/data/filtered_regions.fofn || exit $?
            echo "Task 1 completed at $(date)"
            # Task 2
            echo 'Alignment Complete' || exit $?
            echo "Task 2 completed at $(date)"
            # Task 3
            loadPulses /data/smrtAnalysis/current/common/jobs/016/016452/input.fofn /data/smrtAnalysis/current/common/jobs/016/016452/data/aligned_reads.cmp.h5 -metrics DeletionQV,IPD,InsertionQV,PulseWidth,QualityValue,MergeQV,SubstitutionQV,DeletionTag -byread || exit $?
            echo "Task 3 completed at $(date)"
            # Task 4
            echo 'LoadPulses Complete' || exit $?
            echo "Task 4 completed at $(date)"



            rcode=$?
            echo "Finished on $(date -u)"
            echo "Task align with nproc 11 with exit code ${rcode}."
            exit ${rcode}# Writing stdout and stderr from Popen:
            Setting up ENV on medhat for task align
            Running task://016452/P_Mapping/align on Linux medhat 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Started on Thu Sep 4 16:06:40 UTC 2014
            Validating existence of Input Files
            Successfully found /data/smrtAnalysis/current/common/jobs/016/016452/input.fofn
            Successfully found /data/smrtAnalysis/current/common/jobs/016/016452/data/filtered_regions.fofn
            Successfully validated input files
            2014-09-04 20:48:07,804 [ERROR] BlaserService returned a non-zero exit status. [INFO] 2014-09-04T18:06:46 [blasr] started.
            Killed

            Comment


            • #21
              After sourcing setup.sh:
              Code:
              source /data/smrtAnalysis/current/etc/setup.sh
              Try running the pbalign command with debug turned on:
              Code:
              pbalign.py "/data/smrtAnalysis/current/common/jobs/016/016452/input.fofn" "/data/smrtAnalysis/current/common/references/Maize" "/data/smrtAnalysis/current/common/jobs/016/016452/data/aligned_reads.cmp.h5" --seed=1 --minAccuracy=0.75 --minLength=50 --concordant --algorithmOptions="-useQuality" --algorithmOptions=' -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa' --hitPolicy=randombest --tmpDir=/data/smrtAnalysis/tmpdir --nproc=11 --regionTable=/data/smrtAnalysis/current/common/jobs/016/016452/data/filtered_regions.fofn --debug
              This should give us more info on why blasr crashed.

              Comment


              • #22
                OK but why i need to source the setup.sh ?

                Comment


                • #23
                  Originally posted by Medhat View Post
                  OK but why i need to source the setup.sh ?
                  That sets up appropriate environment variables so you can run the alignment job.

                  Comment


                  • #24
                    medhat@medhat:/data/smrtAnalysis$ source /data/smrtAnalysis/current/etc/setup.sh

                    medhat@medhat:/data/smrtAnalysis$ pbalign.py "/data/smrtAnalysis/current/common/jobs/016/016452/input.fofn" "/data/smrtAnalysis/current/common/references/Maize" "/data/smrtAnalysis/current/common/jobs/016/016452/data/aligned_reads.cmp.h5" --seed=1 --minAccuracy=0.75 --minLength=50 --concordant --algorithmOptions="-useQuality" --algorithmOptions=' -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa' --hitPolicy=randombest --tmpDir=/data/smrtAnalysis/tmpdir --nproc=11 --regionTable=/data/smrtAnalysis/current/common/jobs/016/016452/data/filtered_regions.fofn --debug

                    > <string>(1)<module>()
                    (Pdb) continue

                    2014-09-09 15:59:07,050 [ERROR] BlaserService returned a non-zero exit status. [INFO] 2014-09-09T14:41:36 [blasr] started.
                    Killed
                    Last edited by Medhat; 09-09-2014, 06:00 AM.

                    Comment


                    • #25
                      I though that would give more meaningful output, you could try running blasr directly:
                      Code:
                      blasr /data/smrtAnalysis/current/common/jobs/016/016452/input.fofn /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta  -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa -nproc 11

                      Comment


                      • #26
                        Using this command

                        /data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/blasr /data/smrtAnalysis/current/common/jobs/016/016452/input.fofn /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa -nproc 11 > ~/Downloads/blasr_run_log2.txt 2>&1
                        every thing is ok and this is the last line of the output

                        m140709_111451_42149_c100664292550000001823130112201495_s1_p0/163190/16535_18492 2 0 0 -6508 82.8514 133857058 133858856 237068873 16536 18488 18493 42573
                        [INFO] 2014-09-11T23:32:01 [blasr] ended.

                        Comment


                        • #27
                          any new sugeestions?

                          Comment


                          • #28
                            After chatting with the developer of pbalign.py the blasr crash when being ran via pbalign is most likely related to a known memory issue. The workaround is to force the splitting up of the alignment job. Via command line, create a new directory and copy the settings.xml and input.xml file from the original SMRT Portal job directory:
                            Code:
                            source /data/smrtAnalysis/current/etc/setup.sh
                            
                            smrtpipe.py --params=settings.xml xml:input.xml --dist -D NPROC=17 -D CLUSTER_MANAGER=BASH
                            All the reports are generated when running via command line instead of SMRT portal, just point a browser at <job dir>/index.html

                            If this works it is possible to make changes to SMRT Portal to distribute jobs by changing <smtanalysis>/analysis/etc/smrtpipe.rc to reflect the values of the -D parameters, and editing <smrtanalysis>redist/tomcat/webapps/smrtportal/WEB-INF/web.xml, setting <param-name>jobsAreDistributed</param-name> to 'true'.

                            Comment


                            • #29
                              so to make sure that I understand correctly first I will copy the files from this directory;
                              cp /data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/common/userdata/jobs/016/016452/setting.xml /data/smrtAnalysis/settingCopy/
                              Note:
                              I have in
                              /data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/common/userdata/jobs/016/
                              four jobs
                              016452 016451 016450 016438

                              then after copying those files I will source and run this command

                              source /data/smrtAnalysis/current/etc/setup.sh

                              smrtpipe.py --params=settings.xml xml:input.xml --dist -D NPROC=17 -D CLUSTER_MANAGER=BASH
                              but in the prvious command shall I point to the files in the copied directory or in the job directory? "I mean here the setting and input.xml file "

                              After success of the previous command I shall navigate to the index file in
                              /data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/common/userdata/jobs/016/016452/index.html
                              right?


                              After all of that
                              I need to change
                              /data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/analysis/etc/smrtpipe.rc to reflect the values of the -D parameters
                              from
                              NPROC = 11
                              to
                              NPROC = 17
                              right?

                              and

                              /data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/redist/apache-tomcat-7.0.23/webapps/smrtportal/WEB-INF/web.xml

                              <context-param>
                              <param-name>jobsAreDistributed</param-name>
                              <param-value>false</param-value>
                              </context-param>

                              to

                              <context-param>
                              <param-name>jobsAreDistributed</param-name>
                              <param-value>True</param-value>
                              </context-param>
                              Last edited by Medhat; 09-17-2014, 12:50 AM.

                              Comment


                              • #30
                                Exactly, run the source and smrtpipe.py command in the new directory.

                                Comment

                                Latest Articles

                                Collapse

                                • seqadmin
                                  Strategies for Sequencing Challenging Samples
                                  by seqadmin


                                  Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                                  03-22-2024, 06:39 AM
                                • seqadmin
                                  Techniques and Challenges in Conservation Genomics
                                  by seqadmin



                                  The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                                  Avian Conservation
                                  Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                                  03-08-2024, 10:41 AM

                                ad_right_rmr

                                Collapse

                                News

                                Collapse

                                Topics Statistics Last Post
                                Started by seqadmin, Yesterday, 06:37 PM
                                0 responses
                                8 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, Yesterday, 06:07 PM
                                0 responses
                                8 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 03-22-2024, 10:03 AM
                                0 responses
                                49 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 03-21-2024, 07:32 AM
                                0 responses
                                66 views
                                0 likes
                                Last Post seqadmin  
                                Working...
                                X