Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Processing a very large soil dataset

    I’ve been tasked with assembling a very large soil metagenomic dataset, so large that I’m having second thoughts about my regular processing pipeline, and was hoping some of you fine folk might have some advice:

    I have:

    15 high depth samples, totalling around 4.5 billion PE reads at 2x150

    45 lower depth samples, ~725 million PE reads at 2x150

    all were sequenced on a hiseq 2500

    I should also mention that the end goal here is to assemble as many near-complete genomes as possible for mapping of metatranscriptomic reads from the same samples

    ordinarily my pipeline would look something like:
    1. adapter and quality trim to q ≥10 using bbduk
    2. merge reads with bbmerge, if merge rate is low then proceed with PE reads only
    3. concatenate files and normalise to coverage of 100 with bbnorm, remove low depth kmers ≤ 5
    4. co-assemble all samples using megahit and / or spades
    5. Map raw reads to assembly with bbmap
    6. downstream analysis, annotation/binning etc.


    I’m currently at step 2, and finding that only ~30% of the reads can be merged, so I’m currently thinking that I should just proceed with the unmerged PE reads. Where I’m really struggling is deciding whether or not co-assembly is a viable option in this case, given the size of the dataset. I’m finding that bbnorm is taking a very long time to run even on a single file, near 24 hours with 16 threads and 400gb RAM, using the following settings:
    bbnorm.sh in=infile.fastq.gz out=normalised.fastq.gz hist=hist.txt prefilter=t mindepth=5 target=100 threads=16

    Loglog.sh tells me that there are ~15 billion unique kmers per sample for the high depth samples, and ~2.5 billion for the low depth samples

    So my main question is: would it make more sense for me to assemble the files individually, concatenate the results and deduplicate using dedupe.sh or something similar? My major concern is that I’ll attempt to co-assemble all the reads only to have it crash out after taking up several days on our cluster

    Does anyone have any other time saving tips/advice for processing and assembly of large datasets?

    I've never worked on a dataset this large before so apologies if I'm missing something obvious here!

    Many thanks

Latest Articles

Collapse

  • seqadmin
    Current Approaches to Protein Sequencing
    by seqadmin


    Proteins are often described as the workhorses of the cell, and identifying their sequences is key to understanding their role in biological processes and disease. Currently, the most common technique used to determine protein sequences is mass spectrometry. While still a valuable tool, mass spectrometry faces several limitations and requires a highly experienced scientist familiar with the equipment to operate it. Additionally, other proteomic methods, like affinity assays, are constrained...
    04-04-2024, 04:25 PM
  • seqadmin
    Strategies for Sequencing Challenging Samples
    by seqadmin


    Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
    03-22-2024, 06:39 AM

ad_right_rmr

Collapse

News

Collapse

Topics Statistics Last Post
Started by seqadmin, 04-04-2024, 09:00 AM
0 responses
34 views
0 likes
Last Post seqadmin  
Started by seqadmin, 04-04-2024, 08:48 AM
0 responses
28 views
0 likes
Last Post seqadmin  
Started by seqadmin, 04-01-2024, 06:45 AM
0 responses
45 views
0 likes
Last Post seqadmin  
Started by seqadmin, 03-27-2024, 06:37 PM
0 responses
32 views
0 likes
Last Post seqadmin  
Working...
X