Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Crossbow 1.2.1 On Hadoop - question

    I am running this on a CDH3 cluster currently.

    Crossbow prerequisites mention the following among others:

    1. A bowtie v0.12.8 executable must exist at the same path on all cluster nodes (including the master).
    2. A Crossbow-customized version of soapsnp v1.02 must be installed

    I cannot seem to find these specific functions. Would newer versions of these components work - Bowtie-1.0.0 and SOAPsnp-v1.03?

    Also, running this command:
    ./cb_hadoop --preprocess --input=hdfs:///crossbow/example/e_coli/small.manifest --output=hdfs:///crossbow/example/e_coli/output_small --reference=hdfs:///crossbow-refs/e_coli.jar --all-haploids --streaming-jar=/usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.jar

    And consistently getting this error:
    at SRR014475.fastq_1.out.gz | /usr/bin/md5sum | cut -d' ' -f 1
    Error: No command named `-version' was found. Perhaps you meant `hadoop version'
    hadoop could not be found in HADOOP_HOME or PATH; please specify --hadoop
    java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:390)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)

    As per Michael S, I have created a symlink for hadoop streaming jar to hadoop-streaming-0.20.jar and HADOOP_HOME is set to /usr/lib/hadoop-0.20

    Any help is appreciated! I have unsuccessfully tried to run this on CDH4 and IDH clusters before and attempting to get to a running baseline with CDH3 as it was only tested on this version.

    Thanks.

  • #2
    Thanks to Micheal S, I was able to obtain the specific bowtie 0.12.8 and everything seems to work fine. Hadoop errors are really bogus!

    Comment


    • #3
      Thanks for sahring for this valuble information and it is useful for hadoop learners.hadoop online trainings also provides best http://hadooponlinetrainings.com/had...line-training/hadoop online training

      Comment

      Latest Articles

      Collapse

      • seqadmin
        Strategies for Sequencing Challenging Samples
        by seqadmin


        Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
        03-22-2024, 06:39 AM
      • seqadmin
        Techniques and Challenges in Conservation Genomics
        by seqadmin



        The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

        Avian Conservation
        Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
        03-08-2024, 10:41 AM

      ad_right_rmr

      Collapse

      News

      Collapse

      Topics Statistics Last Post
      Started by seqadmin, 03-27-2024, 06:37 PM
      0 responses
      12 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, 03-27-2024, 06:07 PM
      0 responses
      11 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, 03-22-2024, 10:03 AM
      0 responses
      53 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, 03-21-2024, 07:32 AM
      0 responses
      69 views
      0 likes
      Last Post seqadmin  
      Working...
      X