Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • bcftools is slow

    Hi,

    I am subsetting a vcf by positions stored in a tab delimited file using bcftools. I noticed the program is very slow. Here is the command:

    bcftools view -R ./chr1.passing.markers.txt chr1.vcf.gz -Oz -o ./chr1.reduced.vcf.gz

    where chr1.passing markers is tab delimited chromosome and position for muliple positions, no header. 68K positions. Original vcf has 524K positions.

    The bcftools command is not using NFS (reading/writing to local disk, executable running from the analysis directory), no competing jobs. It is taking a really long time. Still running after 120 minutes.

    I wrote an equivalent perl script that completes this in 10 minutes but uses flat files and so should be even slower than bcftools with its binary file format.

    bcftools version is up to date.

    Does anyone have an idea how I can speed this up or what might be wrong?

    Thanks,
    Craig

  • #2
    RDBMS are usually more efficient with these types of queries...

    It looks like it tries to ungzip and scan whole input file for each chromosome postion range...
    So to do it 68 thousand times... it takes a bit of time...

    to fix:
    1. Your perl script need the following in the VCF open section in order to be able to read gzipped files:

    if($vcf_file_in=~m/\.gz$/i or $vcf_file_in=~m/\.Z$/i){
    open (VCF_IN, "zcat $vcf_file_in |") or die "\nUnable to open gzipped vcf input file: $vcf_file_in\n";
    }else{
    open (VCF_IN, $vcf_file_in) or die "\nUnable to open vcf input file: $vcf_file_in\n";
    }


    OR:
    2. try running it on ungzipped input file...

    OR:
    3. If you are good with perl, DBI, SQL and MySQL/postgres/etc you can try loading input vcf file into mysql table(s), index it properly and run a set of SQL queries to select needed data.
    PS: Make sure to crank up the MySQL server memory limits (/etc/my.cnf) before attempting to do it...
    Last edited by Markiyan; 05-17-2018, 01:17 AM. Reason: Refinement

    Comment


    • #3
      "It looks like it tries to ungzip and scan whole input file for each chromosome postion range...
      So to do it 68 thousand times... it takes a bit of time..."

      > It is my impression that the program doesn't unzip anything, I believe it streams an indexed binary file and writes a binary file, that can also be indexed.

      The call I made is very standard, if it is poorly phrased, let me know.

      Has anyone experienced this issue of slow performance with bcftools compared to some other benchmark program and been able to resolve it?

      Craig

      Comment


      • #4
        If you don't have an index or have *many* regions, you're probably better off using -T instead of -R which will stream and filter. I have vague recollections though that there may be a difference too regarding overlapping regions and whether records in the overlaps get reported once or twice. (Using an index would report them twice IIRC.)

        -R attempts to use the index, which *should* be faster than streaming, but only if the density is low.

        Eg if you're pulling 68000 positions each of which will mean seeking and uncompressing a gzip block (ideally it'd cache, but this may not be done) then possibly it's actually decoding more than the entire files worth of data, in which case streaming (-T) becomes more efficient. On the flip side, if you have a 10Gb file and 2 positions, seeking and decoding will obviously be faster.

        Comment


        • #5
          Thank you for your response. I am pulling a lot of regions from the original file. I will try -T and see if there is some improvement.

          Thanks again!

          Craig

          Comment

          Latest Articles

          Collapse

          • seqadmin
            Strategies for Sequencing Challenging Samples
            by seqadmin


            Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
            03-22-2024, 06:39 AM
          • seqadmin
            Techniques and Challenges in Conservation Genomics
            by seqadmin



            The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

            Avian Conservation
            Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
            03-08-2024, 10:41 AM

          ad_right_rmr

          Collapse

          News

          Collapse

          Topics Statistics Last Post
          Started by seqadmin, Yesterday, 06:37 PM
          0 responses
          11 views
          0 likes
          Last Post seqadmin  
          Started by seqadmin, Yesterday, 06:07 PM
          0 responses
          10 views
          0 likes
          Last Post seqadmin  
          Started by seqadmin, 03-22-2024, 10:03 AM
          0 responses
          51 views
          0 likes
          Last Post seqadmin  
          Started by seqadmin, 03-21-2024, 07:32 AM
          0 responses
          67 views
          0 likes
          Last Post seqadmin  
          Working...
          X