Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • fastest way to 'parse' fasta or fastq?

    I am looking for a real (High performace computing / HPC) fast fasta or fastq parsing program. I just want the most simple statistics imaginable:
    - number of reads
    - total nr of bases.
    Other stuff like average length/ATCG composition is nice, but not required.

    I searched the software page, tried some packages, wrote my own parsers but they are all slow.
    I am looking for something in C code, which can be super fast I hope.
    I also tried this simple bash code:
    " time grep -v '^>' ./test.fa | wc -m -l"

    which is 'fast' ( 30 seconds to scan 1 GB fasta (file in memory)
    My simple python script takes over a minute to scan this file. But I hope this can be done faster, or all in one script.


    If you want to scan gigabytes of files, it would be nice to have a very fast parser.

    Anyone who is aware of such program? Or, what do you think is the fastest program you know?

  • #2
    Hi,

    You could try the FASTQC package if you haven't already. It can take fastq/bam/sam files and gives most of the important statistics for a NGS run.

    Comment


    • #3
      I suggest using native linux tools such as grep, sed, awk in multithreaded environment also 64 bit may be useful in some applications where it is supported. There is option of using CUDA with GPU to do super fast calculations.

      Comment


      • #4
        PRINSEQ and FASTQC

        Comment


        • #5
          If you're up for moding a couple of lines of code for your needs
          this should do the trick ...
          Code:
          #include <stdio.h>
          #include <string.h>
          #include <ctype.h>
          unsigned long int sum[5];
          unsigned long int basecount;
          unsigned long int readcount = 0;
          char s[512];
          int main()
          {
              register int i,j;
              char ch;
              basecount = 0;
              memset(sum,0,sizeof(sum));
              while (gets(s))
              {
                  if (s[0] == '>') continue; // skip fasta entry header
                  readcount++;
                  for (i=0;i<s[i];i++)
                  {
                      ch = toupper(s[i]);
                      if (ch == 'A') { sum[0]++; basecount++; }
                      else if (ch == 'C') { sum[1]++; basecount++; }
                      else if (ch == 'G') { sum[2]++; basecount++; }
                      else if (ch == 'T') { sum[3]++; basecount++; }
                      else if (ch == 'N') { sum[4]++; basecount++; }
                  }
                  memset(s,0,sizeof(s));
              }
              for (j=0;j<5;j++)
              {
                  if (j == 0) printf("A ");
                  else if (j == 1) printf("C ");
                  else if (j == 2) printf("G ");
                  else if (j == 3) printf("T ");
                  else if (j == 4) printf("N ");
                  printf("%ld ",sum[j]);
                  printf("\n");
              }
              printf("bases = %ld \n",basecount);
              printf("reads = %ld \n",readcount);
              return 0;
          }

          Comment


          • #6
            If you don't want error checking Heng Li has a very fast FASTA/FASTQ parser in C which could easily be used for the basic information you requested (read count and total bases):

            Comment

            Latest Articles

            Collapse

            • seqadmin
              Strategies for Sequencing Challenging Samples
              by seqadmin


              Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
              03-22-2024, 06:39 AM
            • seqadmin
              Techniques and Challenges in Conservation Genomics
              by seqadmin



              The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

              Avian Conservation
              Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
              03-08-2024, 10:41 AM

            ad_right_rmr

            Collapse

            News

            Collapse

            Topics Statistics Last Post
            Started by seqadmin, Yesterday, 06:37 PM
            0 responses
            12 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, Yesterday, 06:07 PM
            0 responses
            10 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 03-22-2024, 10:03 AM
            0 responses
            51 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 03-21-2024, 07:32 AM
            0 responses
            68 views
            0 likes
            Last Post seqadmin  
            Working...
            X