Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • bbtools: how to properly set up rqcfilter

    Dear all,
    I've recently obtained some PE RNA-seq data from JGI. They have been cleaned using the rqcfilter tool (v. 37.17). Since I would like to uniform the cleaning method extending it to other previously obtained Illumina data, I would like to set up my server to be able to run rqcfilter. I'm aware that this script has been hardcoded to run on on Genepool...but it worth to give a try.

    Here follow the steps run by the RQCfilter, as deduced by the reproduce outputs, on my PE data:

    >clumpify.sh pigz=t unpigz=t zl=4 reorder in1=OMISSIS.fastq.gz out1=TEMP_CLUMP_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz passes=1

    >bbduk.sh ktrim=r ordered minlen=49 minlenfraction=0.33 mink=11 tbo tpe rcomp=f overwrite=true k=23 hdist=1 hdist2=1 ftm=5 pigz=t unpigz=t zl=4 ow=true in1=TEMP_CLUMP_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz out1=TEMP_TRIM_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz rqc=hashmap outduk=ktrim_kmerStats1.txt stats=ktrim_scaffoldStats1.txt loglog ref=adapters2.fa

    >rm TEMP_CLUMP_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz

    >bbduk.sh maq=10,0 trimq=6 qtrim=r ordered overwrite=true maxns=1 minlen=49 minlenfraction=0.33 k=25 hdist=1 pigz=t unpigz=t zl=6 cf=t barcodefilter=crash ow=true in1=TEMP_TRIM_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz out1=TEMP_FILTER1_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz outm=synth1.fq.gz rqc=hashmap outduk=kmerStats1.txt stats=scaffoldStats1.txt loglog ref=pJET1.2.fasta

    >rm TEMP_TRIM_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz

    >bbduk.sh ordered overwrite=true k=20 hdist=1 pigz=t unpigz=t zl=6 ow=true in1=TEMP_FILTER1_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz out1=TEMP_FILTER2_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz outm=synth2.fq.gz outduk=kmerStats2.txt stats=scaffoldStats2.txt loglog ref=short.fa

    >rm TEMP_FILTER1_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz

    >bbmap.sh ordered quickmatch k=13 idtag=t printunmappedcount ow=true qtrim=rl trimq=10 untrim build=1 null path= pigz=t unpigz=t zl=6 minid=.95 idfilter=.95 maxindel=3 minhits=2 bw=12 bwr=0.16 null maxsites2=10 tipsearch=0 in1=TEMP_FILTER2_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz outu1=TEMP_MICROBE_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz outm=microbes.fq.gz scafstats=commonMicrobes.txt

    >rm TEMP_FILTER2_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz

    >bbmap.sh ordered k=14 idtag=t usemodulo printunmappedcount ow=true qtrim=rl trimq=10 untrim kfilter=25 maxsites=1 tipsearch=0 minratio=.9 maxindel=3 minhits=2 bw=12 bwr=0.16 fast=true maxsites2=10 outm=human.fq.gz path= refstats=refStats.txt pigz=t unpigz=t zl=9 in1=TEMP_MICROBE_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz outu1=OMISSIS.anqrpht.fastq.gz

    >rm TEMP_MICROBE_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz

    >bbmerge.sh loose overwrite=true in1=OMISSIS.anqrpht.fastq.gz ihist=ihist_merge.txt outc=cardinality.txt pigz=t unpigz=t zl=9 adapters=adapters2.fa

    >kmercountexact.sh overwrite=true in1=OMISSIS.anqrpht.fastq.gz khist=khist.txt peaks=peaks.txt unpigz=t

    I have highlighted in bold the files that, at a first glance, are missing. Where can I find them? while it's pretty clear what could be the content of adapters2.fa and pJET1.2.fasta, what's the content of the short.fa file?

    about the contaminant removal, should I retrieve also human and cat/dog/mouse/microbial references or the script does it itself?

    Thanks for any hints you may provide!
    Cheers,
    Stefano

  • #2
    I think you are better off asking this question to JGI tech support who gave you this data.

    @Brian Bushnell (author of BBMap) works at JGI and contributes regularly on this forum. He has been busy with something for last few days and I have not seen him here. You could also try emailing him directly.

    Comment


    • #3
      I supposed he has been busy...I 've wrote him last week and got no replies, that's why I posted here. I'll give another try by email and with JGI staff.
      Thanks!
      s.-

      Comment


      • #4
        I'm going to report here my experience, along with some private reply from Brian, while setting up bbmap on my machine. This will turn in a guide, as soon I got the whole package working.

        Some of this [read, missing file] is unnecessary. adapters2.fa is just adapters.fa concatenated with the reverse-complement of itself and is strictly unnecessary because BBDuk automatically looks for reverse-complements of kmers; thus, the results will be identical if you use adapters.fa (in /bbmap/resources/) instead and change the rcomp flag:

        bbduk.sh ktrim=r ordered minlen=49 minlenfraction=0.33 mink=11 tbo tpe rcomp=t overwrite=true k=23 hdist=1 hdist2=1 ftm=5 pigz=t unpigz=t zl=4 ow=true in1=TEMP_CLUMP_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz out1=TEMP_TRIM_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz rqc=hashmap outduk=ktrim_kmerStats1.txt stats=ktrim_scaffoldStats1.txt loglog ref=adapters.fa

        pjet is now in BBMap's \resources\ folder (pJET1.2.fasta) as of BBMap 37.68

        short.fa is contaminant sequences shorter than 31bp, the kmer length used for filtering most contaminants. It can be made like this:

        reformat.sh in=sequencing_artifacts.fa.gz out=short.fa maxlen=30

        masked human/cat/dog/mouse references, as well as microbial contaminants, are located on my google drive. The locations are now given in resources/remote_files.txt
        Step ahead. Since I'm interested in set up the the RQCfilter program, I've found that the source java file in current/jgi/RQCfilter.java, bears the paths to the reference file needed.

        In Green are the file I was able to fix. In red, the files still missing. Others in black are in unknown state.

        Code:
        	/*--------------------------------------------------------------*/
        	/*----------------        Reference Files       ----------------*/
        	/*--------------------------------------------------------------*/
        	
        	private String shortArtifactFile = "/global/projectb/sandbox/gaag/bbtools/data/[COLOR="Lime"]short.fa[/COLOR]";
        	private String shortArtifactFile_noNextera = "/global/projectb/sandbox/gaag/bbtools/data/[COLOR="Red"]short_noNextera.fa[/COLOR]";
        	
        	private String mainArtifactFile_noNextera = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/[COLOR="red"]Illumina.artifacts.2013.12.no_DNA_RNA_spikeins_no_Nextera_junction.fa.gz[/COLOR]";
        	private String mainArtifactFile = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/[COLOR="red"]Illumina.artifacts.2013.12.no_DNA_RNA_spikeins.fa[/COLOR]";
        	private String artifactFileRna = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/[COLOR="red"]RNA_spikeins.artifacts.2012.10.NoPolyA.fa[/COLOR]";
        	private String artifactFileDna = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/[COLOR="red"]DNA_spikeins.artifacts.2012.10.fa[/COLOR]";
        	private String artifactFileDna_noNextera = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/[COLOR="red"]DNA_spikeins.artifacts_no_Nextera_junction.2012.10.fa.gz[/COLOR]";
        	
        	private String phixRef = "/global/dna/shared/rqc/ref_databases/qaqc/databases/[COLOR="lime"]phix174_ill.ref.fa[/COLOR]";
        	private String lambdaRef = "/global/dna/shared/rqc/ref_databases/qaqc/databases/[COLOR="Lime"]lambda.fa.gz[/COLOR]";
        	private String lfpeLinker = "/global/dna/shared/rqc/ref_databases/qaqc/databases/lfpe.linker.fa";
        	private String clrsLinker = "/global/dna/shared/rqc/ref_databases/qaqc/databases/crelox.fa";
        	private String clipLinker = clipLinkerDefault; //A literal string; "CATG" is supposed to be the normal linker.
        	
        	private String pjetRef = "/global/dna/shared/rqc/ref_databases/qaqc/databases/[COLOR="lime"]pJET1.2.fasta[/COLOR]";
        	private String riboKmers = "/global/projectb/sandbox/gaag/bbtools/ribo/[COLOR="red"]merged_ribokmers20.fa.gz[/COLOR]";
        	private String allArtifactsLatest = "/global/projectb/sandbox/rqc/qcdb/illumina.artifacts/[COLOR="red"]Illumina.artifacts.fa[/COLOR]";
        	private String fragAdapter = "/global/projectb/sandbox/gaag/bbtools/data/[COLOR="Lime"]adapters2.fa[/COLOR]";
        	private String rnaAdapter = "/global/projectb/sandbox/gaag/bbtools/data/[COLOR="lime"]truseq_rna.fa.gz[/COLOR]";
        	private String mtstRef = "/global/projectb/sandbox/gaag/bbtools/data/[COLOR="lime"]mtst.fa[/COLOR]";
        
        	private String [COLOR="lime"]humanPath[/COLOR] = "/global/projectb/sandbox/gaag/bbtools/hg19/";
        	private String [COLOR="lime"]catPath[/COLOR] = "/global/projectb/sandbox/gaag/bbtools/cat_genome/";
        	private String [COLOR="lime"]dogPath[/COLOR] = "/global/projectb/sandbox/gaag/bbtools/dog_genome/";
        	private String [COLOR="lime"]mousePath[/COLOR] = "/global/projectb/sandbox/gaag/bbtools/mouse_genome/";
        	private String humanRef = null;
        
        	private String catDogHumanPath = "/global/projectb/sandbox/gaag/bbtools/catdoghuman/";
        	private String mouseCatDogHumanPath = "/global/projectb/sandbox/gaag/bbtools/mousecatdoghuman/";
        
        	private String chloroplastRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/plastid.fa.gz";
        	private String mitoRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/mitochondrion.fa.gz";
        	private String chloroMitoRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/chloro_mito.fa.gz";
        	private String chloroMitoRiboRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/chloro_mito_ribo.fa.gz";
        	private String riboRef = "/global/projectb/sandbox/gaag/bbtools/silva/both_clumped_renamed.fa.gz";
        	private String mitoRiboRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/mito_ribo.fa.gz";
        	private String chloroRiboRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/chloro_ribo.fa.gz";
        
        	private String commonMicrobesPath = "/global/projectb/sandbox/gaag/bbtools/commonMicrobes/";
        	private String [COLOR="lime"]commonMicrobesRef[/COLOR] = "/global/projectb/sandbox/gaag/bbtools/commonMicrobes/[COLOR="lime"]fusedERPBBmasked.fa.gz[/COLOR]";
        	private int commonMicrobesBuild = 1;
        	private String taxTree=TaxTree.defaultTreeFile();
        	private String giTable=TaxTree.defaultTableFile();
        Once fixed, I've recompiled the package with

        Code:
        javac -cp current current/jgi/RQCFilter.java

        At the moment, the script stops at the second BBduk step, while it's looking for Illumina artifacts files. Hope to fix it soon.

        Comment


        • #5
          I'm going to report here my experience, along with some private reply from Brian, while setting up bbmap on my machine. This will turn in a guide, as soon I got the whole package working.

          Some of this [read, missing file] is unnecessary. adapters2.fa is just adapters.fa concatenated with the reverse-complement of itself and is strictly unnecessary because BBDuk automatically looks for reverse-complements of kmers; thus, the results will be identical if you use adapters.fa (in /bbmap/resources/) instead and change the rcomp flag:

          bbduk.sh ktrim=r ordered minlen=49 minlenfraction=0.33 mink=11 tbo tpe rcomp=t overwrite=true k=23 hdist=1 hdist2=1 ftm=5 pigz=t unpigz=t zl=4 ow=true in1=TEMP_CLUMP_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz out1=TEMP_TRIM_8352bfa6997ae09cc4039681816c_OMISSIS.anqrpht.fastq.gz rqc=hashmap outduk=ktrim_kmerStats1.txt stats=ktrim_scaffoldStats1.txt loglog ref=adapters.fa

          pjet is now in BBMap's \resources\ folder (pJET1.2.fasta) as of BBMap 37.68

          short.fa is contaminant sequences shorter than 31bp, the kmer length used for filtering most contaminants. It can be made like this:

          reformat.sh in=sequencing_artifacts.fa.gz out=short.fa maxlen=30

          masked human/cat/dog/mouse references, as well as microbial contaminants, are located on my google drive. The locations are now given in resources/remote_files.txt
          Step ahead. Since I'm interested in set up the the RQCfilter program, I've found that the source jaav file in current/jgi/RQCfilter.java, bears the paths to the reference file needed.

          Green are the file I have fixed.

          Code:
          	/*--------------------------------------------------------------*/
          	/*----------------        Reference Files       ----------------*/
          	/*--------------------------------------------------------------*/
          	
          	private String shortArtifactFile = "/global/projectb/sandbox/gaag/bbtools/data/[COLOR="Lime"]short.fa[/COLOR]";
          	private String shortArtifactFile_noNextera = "/global/projectb/sandbox/gaag/bbtools/data/short_noNextera.fa";
          	
          	private String mainArtifactFile_noNextera = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/Illumina.artifacts.2013.12.no_DNA_RNA_spikeins_no_Nextera_junction.fa.gz";
          	private String mainArtifactFile = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/Illumina.artifacts.2013.12.no_DNA_RNA_spikeins.fa";
          	private String artifactFileRna = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/RNA_spikeins.artifacts.2012.10.NoPolyA.fa";
          	private String artifactFileDna = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/DNA_spikeins.artifacts.2012.10.fa";
          	private String artifactFileDna_noNextera = "/global/dna/shared/rqc/ref_databases/qaqc/databases/illumina.artifacts/DNA_spikeins.artifacts_no_Nextera_junction.2012.10.fa.gz";
          	
          	private String phixRef = "/global/dna/shared/rqc/ref_databases/qaqc/databases/[COLOR="lime"]phix174_ill.ref.fa[/COLOR]";
          	private String lambdaRef = "/global/dna/shared/rqc/ref_databases/qaqc/databases/[COLOR="lime"]lambda.fa.gz[/COLOR]";
          	private String lfpeLinker = "/global/dna/shared/rqc/ref_databases/qaqc/databases/lfpe.linker.fa";
          	private String clrsLinker = "/global/dna/shared/rqc/ref_databases/qaqc/databases/crelox.fa";
          	private String clipLinker = clipLinkerDefault; //A literal string; "CATG" is supposed to be the normal linker.
          	
          	private String pjetRef = "/global/dna/shared/rqc/ref_databases/qaqc/databases/[COLOR="lime"]pJET1.2.fasta[/COLOR]";
          	private String riboKmers = "/global/projectb/sandbox/gaag/bbtools/ribo/merged_ribokmers20.fa.gz";
          	private String allArtifactsLatest = "/global/projectb/sandbox/rqc/qcdb/illumina.artifacts/Illumina.artifacts.fa";
          	private String fragAdapter = "/global/projectb/sandbox/gaag/bbtools/data/[COLOR="lime"]adapters2.fa[/COLOR]";
          	private String rnaAdapter = "/global/projectb/sandbox/gaag/bbtools/data/[COLOR="lime"]truseq_rna.fa.gz[/COLOR]";
          	private String mtstRef = "/global/projectb/sandbox/gaag/bbtools/data/mtst.fa";
          
          	private String [COLOR="lime"]humanPath[/COLOR] = "/global/projectb/sandbox/gaag/bbtools/hg19/";
          	private String [COLOR="lime"]catPath[/COLOR] = "/global/projectb/sandbox/gaag/bbtools/cat_genome/";
          	private String [COLOR="lime"]dogPath[/COLOR] = "/global/projectb/sandbox/gaag/bbtools/dog_genome/";
          	private String [COLOR="lime"]mousePath[/COLOR] = "/global/projectb/sandbox/gaag/bbtools/mouse_genome/";
          	private String humanRef = null;
          
          	private String catDogHumanPath = "/global/projectb/sandbox/gaag/bbtools/catdoghuman/";
          	private String mouseCatDogHumanPath = "/global/projectb/sandbox/gaag/bbtools/mousecatdoghuman/";
          
          	private String chloroplastRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/plastid.fa.gz";
          	private String mitoRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/mitochondrion.fa.gz";
          	private String chloroMitoRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/chloro_mito.fa.gz";
          	private String chloroMitoRiboRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/chloro_mito_ribo.fa.gz";
          	private String riboRef = "/global/projectb/sandbox/gaag/bbtools/silva/both_clumped_renamed.fa.gz";
          	private String mitoRiboRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/mito_ribo.fa.gz";
          	private String chloroRiboRef = "/global/projectb/sandbox/gaag/bbtools/refseqMicrobial/chloro_ribo.fa.gz";
          
          	private String commonMicrobesPath = "/global/projectb/sandbox/gaag/bbtools/commonMicrobes/";
          	private String commonMicrobesRef = "/global/projectb/sandbox/gaag/bbtools/commonMicrobes/[COLOR="lime"]fusedERPBBmasked.fa.gz[/COLOR]";
          	private int commonMicrobesBuild = 1;
          	private String taxTree=TaxTree.defaultTreeFile();
          	private String giTable=TaxTree.defaultTableFile();
          Once fixed, I've recompiled the package with

          Code:
          javac -cp current current/jgi/RQCFilter.java

          At the moment, the script stops at the second BBduk step, while it's looking for Illumina artifacts files. Hope to fix it soon.

          Comment

          Latest Articles

          Collapse

          • seqadmin
            Strategies for Sequencing Challenging Samples
            by seqadmin


            Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
            03-22-2024, 06:39 AM
          • seqadmin
            Techniques and Challenges in Conservation Genomics
            by seqadmin



            The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

            Avian Conservation
            Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
            03-08-2024, 10:41 AM

          ad_right_rmr

          Collapse

          News

          Collapse

          Topics Statistics Last Post
          Started by seqadmin, Yesterday, 06:37 PM
          0 responses
          10 views
          0 likes
          Last Post seqadmin  
          Started by seqadmin, Yesterday, 06:07 PM
          0 responses
          9 views
          0 likes
          Last Post seqadmin  
          Started by seqadmin, 03-22-2024, 10:03 AM
          0 responses
          51 views
          0 likes
          Last Post seqadmin  
          Started by seqadmin, 03-21-2024, 07:32 AM
          0 responses
          67 views
          0 likes
          Last Post seqadmin  
          Working...
          X