Split fastq into smaller files

[Guest]

Dear all,

I'm looking into splitting a FASTQ read file into several smaller sized files. It's basically just distributing batches of 4 lines into a certain number of files.

I'm trying with

split -l <number of lines per file> <FASTQ>

which works of course, but is too slooooooooooow on a HiSeq read file.

Any recommendations for faster splitting? awk, sed?

Thanks!

[ehlin]

I haven't tried this, but would it be faster if you specified a file size rather than a line number?

[pallevillesen]

I really don't see anything faster than split (unless you want to parallelize it and let each subroutine extract certain parts of the file) (using e.g. awk).

But for really large files the time for counting the lines (for input to awk) would also take a lot of time...

I would just split it as you do...

[palle@s01n11 3_adapter_trimmed]$ time split -l 4000000 fastqfile

real 2m17.853s
user 0m2.640s
sys 0m17.980s

For a 16 gig file - that is ok.

>cat fatq |grep -e "^@# |wc -l

69332456 fastq records

Primer for awk:

fq=...
from=0
to=4000000
time cat $fq | awk "NR > $from && NR < $to" >xaa
cat xaa | grep -e "^@" |wc -l

You could do this in a simple for loop in bash and submit each cat|awk to seperate nodes of a cluster.... but I doubt it's worth the hassle.... submit all your splits to a cluster and go grab a cup of coffee...

Edit: time cat $fq | awk "{ if (NR < $from) next; if (NR < $to) print; if (NR >= $to) exit;} " >xaa

Will exit after you extracted the wanted part and is much faster for large files (well - only for the first splits - for the last parts it has to read through the file first).

[lorendarith]

I haven't tried this, but would it be faster if you specified a file size rather than a line number?

Haven't tried it, but wouldn't this result in truncated FASTQ entries, especially if you are doing it on compressed files to save time?

I really don't see anything faster than split (unless you want to parallelize it and let each subroutine extract certain parts of the file) (using e.g. awk).

Thanks! Though... 2mins on a 16 Gb file? Tried splitting a 32Gb file and it took HOURS! There must have been something seriously wrong with our file system server...

[apredeus]

Any recommendations for faster splitting? awk, sed?

Thanks!

well it's just reading it into memory and then writing it back, it should be very fast yes you can use awk, how big you want your small files?

you can do something like (bash syntax)

for  i in `seq 1 10`
do
  awk -v v=$i '{if (NR>(v-1)*400000 && NR<=v*400000) print}' > $i.fastq 
done

That will break 1M read fastq file into ten 100K files.

And it should be very quick, few minutes even for very big files.

PS sorry - you already got the question answered, I'm still asleep apparently

[lorendarith]

PS sorry - you already got the question answered, I'm still asleep apparently

ALL suggestions are welcomed and appreciated! Thanks

[apredeus]

You're welcome. I've just changed the code a bit, I messed up a variable name within awk.

[pallevillesen]

Haven't tried it, but wouldn't this result in truncated FASTQ entries, especially if you are doing it on compressed files to save time?

Thanks! Though... 2mins on a 16 Gb file? Tried splitting a 32Gb file and it took HOURS! There must have been something seriously wrong with our file system server...

Well... Our cluster is brand new with 80 Gbit network between nodes and the fileserver - that may cause things to run extremely fast here...

Anyway: your problem was solved.

[sklages]

Haven't tried it, but wouldn't this result in truncated FASTQ entries, especially if you are doing it on compressed files to save time?



Thanks! Though... 2mins on a 16 Gb file? Tried splitting a 32Gb file and it took HOURS! There must have been something seriously wrong with our file system server...

No local storage? NFS?

[gsgs]

there should be a solution to just change the directory list, file names,
file sizes, while keeping the data where it is

[sklages]

there should be a solution to just change the directory list, file names,
file sizes, while keeping the data where it is

Sure, but reading a 32G file and maybe rewriting it (in chunks) is terribly slow via NFS ...