Combining fastq files from paired-end WES prior to trimming and alignment

[reliscu]

Hi there!

I've tried to find answers to questions that apply to my scenario, but I'm still unsure about the kosher way to approach this. I should say, I'm very new to this side of bioinformatics.

I have data from four samples, with fastq files per read (R1 and R2) and per lane (lane 1 and lane 2)

For example, the fastq files from my first sample:

SF10711_9-1-22_CAGATC_L001_R1_001.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_001.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_002.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_002.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_003.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_003.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_004.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_004.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_005.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_005.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_006.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_006.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_007.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_007.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_008.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_008.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_009.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_009.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_010.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_010.fastq.gz
SF10711_9-1-22_CAGATC_L001_R1_011.fastq.gz SF10711_9-1-22_CAGATC_L002_R1_011.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_001.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_001.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_002.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_002.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_003.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_003.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_004.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_004.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_005.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_005.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_006.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_006.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_007.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_007.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_008.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_008.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_009.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_009.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_010.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_010.fastq.gz
SF10711_9-1-22_CAGATC_L001_R2_011.fastq.gz SF10711_9-1-22_CAGATC_L002_R2_011.fastq.gz

I was told generally that I should concatenate the fastq files prior to trimming and alignment because it would be easier to interpret the trimming and alignment statistics. My question then, is how can these be combined?

I.e. Should I combine all fastq files from a sample (e.g. all of the files above into a single file), combine all files within a sample by R1 and R2, or combine all files within a sample by L1 and L2.

Apologies for the simple questions. Again, I have done some research prior to posting this question, but still feel unsure about how to proceed.

Thanks in advance for any insight!

[floem7]

As of joining fastq's, you can do it using bash cat command:
cat file1.fastq.gz file2.fastq.gz ... filen.fastq.gz > joined.fastq.gz
Generally it's better to keep R1 and R2 reads in separate files.
Nevertheless R1 and R2 read can be interleaved in one file, you could try
reformat.sh in1=read1.fq in2=read2.fq out=reads.fq
from the bbmap suite reformat
As of this whole idea. I rather advice to trim reads for each pair separately, i.e. R1 and R2 pairs together for each library. Again, you can use bbduk from bbmap suite.
Each run is separate reaction, so it can give different problems, some reads could have more adapter contamination, for example. Or in a given file there could be more overrepresented sequences, or other bias. So each pair of read should be trimmed together and separately from read from other libraries. You can generate quality metrics using FastQC

[reliscu]

As of joining fastq's, you can do it using bash cat command:
cat file1.fastq.gz file2.fastq.gz ... filen.fastq.gz > joined.fastq.gz
Generally it's better to keep R1 and R2 reads in separate files.
Nevertheless R1 and R2 read can be interleaved in one file, you could try
reformat.sh in1=read1.fq in2=read2.fq out=reads.fq
from the bbmap suite reformat
As of this whole idea. I rather advice to trim reads for each pair separately, i.e. R1 and R2 pairs together for each library. Again, you can use bbduk from bbmap suite.
Each run is separate reaction, so it can give different problems, some reads could have more adapter contamination, for example. Or in a given file there could be more overrepresented sequences, or other bias. So each pair of read should be trimmed together and separately from read from other libraries. You can generate quality metrics using FastQC

Thank you for your response!

To clarify, I could do something like this:

Combine all R1 fastq files from L1 >> R1_L1.fastq
Combine all R2 fastq files from L1 >> R2_L1.fastq
>> Then trim these two files together

Combine all R1 fastq files from L2 >> R1_L2.fastq
Combine all R2 fastq files from L2 >> R2_L2.fastq
>> Then trim these two files together

Is this correct?

As a followup, having done the trimming, is it reasonable then to combine all the trimmed reads into a SINGLE fastq, or should the separation of libraries and/or reads be maintained?

Thanks again.

[floem7]

Your commands are correct.

As a followup, having done the trimming, is it reasonable then to combine all the trimmed reads into a SINGLE fastq, or should the separation of libraries and/or reads be maintained?

I work with sequencing since 2015 and frankly I think leaving separate fastq files are standard. I think joining fastq's is useful (after trimming) if you have more than one sequencing run for the same library*. So later you can use this one file for alignment, as opposed to using few files, what is also possible (bbwrap from bbmap suite can do that).

* although some advice to rather join resulting bam's

Nevertheless, I hope anyone more experienced could join this post...