Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Splitting a FastQ file into two

    Hello folks,

    I happen to have a small problem which seemed to be trivial at first, but keeps me busy for a while now already. Maybe you can help...

    Problem:

    I need to split 615M paired reads currently in two FastQ files into two file pairs with 308M reads each.

    Solution attempt A:

    I unsuccessfully tried to use line count based tools like split or awk, but since newline characters occur in the quality scores, these tools respectively I screwed up badly.

    Solution attempt B:
    Code:
    bbmap/reformat.sh  in=...  in2=... out=... out2=... reads=308000000
    bbmap/reformat.sh  in=...  in2=... out=... out2=... skipreads=308000000
    resulted in


    Input is being processed as paired
    Input: 615307122 reads 91326404105 bases
    Output: 615307122 reads (100.00%) 91326404105 bases (100.00%)

    for the first command and in

    Input is being processed as paired
    Input: 615307122 reads 91326404105 bases
    Output: 0 reads (0.00%) 0 bases (0.00%)

    for the second command. Effectively those read commands seem to be ignored (BBMap Version 38.76).


    Solution attempt C:
    Code:
    famas --in=...  --in2=... --out=.XXXXXX.fq.gz  --out2=.XXXXXX.fq.gz  -x 308000000
    flooded the output directory with thousands of subfiles files (instead of the actually needed two files each) until the file system couldn't cope with the number of open files anymore and ran out of file descriptors (famas version 0.0.12).


    ERROR(famas.c|open_output_one:1056): Couldn't open =...compressed.065534.fq.gz
    ERROR(famas.c|main:1163): Couldn't open output files. Exiting...


    Since it took me a while to clean that mess up on the cluster again, I am somewhat reluctant to try out more now. Any ideas what I made wrong or suggestions which tools work better?

    Thanks a lot for reading and help!
    Thias

  • #2
    Cross-posted and answered at: https://www.biostars.org/p/451453/

    Comment


    • #3
      Indeed, the issue is solved by now using seqkit split2.

      My apologies for not indicating this here. I had posted here first but the thread was lingering in moderation for ~24h and thus I decided to ask for help on Biostars. It had not yet shown up when I got the answer on Biostars and I subsequently forgot to check back here. Thanks a lot to everyone none the less!

      Comment

      Latest Articles

      Collapse

      • seqadmin
        Strategies for Sequencing Challenging Samples
        by seqadmin


        Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
        03-22-2024, 06:39 AM
      • seqadmin
        Techniques and Challenges in Conservation Genomics
        by seqadmin



        The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

        Avian Conservation
        Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
        03-08-2024, 10:41 AM

      ad_right_rmr

      Collapse

      News

      Collapse

      Topics Statistics Last Post
      Started by seqadmin, Yesterday, 06:37 PM
      0 responses
      12 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, Yesterday, 06:07 PM
      0 responses
      10 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, 03-22-2024, 10:03 AM
      0 responses
      52 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, 03-21-2024, 07:32 AM
      0 responses
      68 views
      0 likes
      Last Post seqadmin  
      Working...
      X