fastq format questions

[asb2718]

Dear All,
We are a small research group working on compressing fastq file information. We have read through the format specifications (as described in http://nar.oxfordjournals.org/content/38/6/1767) and have the following questions:
1. How significant is each field in a fastq file? We know that the sequence data is important. What about quality data? Do we need to accurately reproduce it?
2. Title line: What aspects of title line are important? What does the title line typically signify. What aspects of title should be retained and what can be dropped? We want to know, what information in the title line is most important to those working with the fastq file.
3. Is the order of reads as stored in a fastq file important or can the reads be reordered to make a new fastq file. Note that the new fastq file will contain the same but reordered information.

Thank you all very much in advance for any inputs that we can get from you.
Best Regards
Ajit

[ulz_peter]

Do you know that article?:

http://www.ncbi.nlm.nih.gov/pubmed/20605925

Regarding your questions (from my point of view):
1) Quality data is important as well for quality-aware alignment software and especially for SNP-calling (noone would waste that much disk space for unnecessary information)

2) I am not sure about the information of the title line the only thing that comes to my mind is that in case of paired end sqeuencing the first and the second mate need to have the same name. Not sure about the rest of the title line.

3) I don't think the order plays an important role but appreciate any comments correcting me...

[gaffa]

3. Generally speaking, the order of sequences in a file carries no significance. However, in Next Generation Sequencing it is common that reads produced by a paired-end experiment are stored in two separate fastq files, with the two reads of a pair being found on the same line in the two corresponding files. Clearly in this situation order is crucial.

[asb2718]

Thank you 'ulz_peter' and 'gaffa' for your answers. We appreciate your feedback. With respect to quality values, is there a particular threshold above which quality values always mean 'high quality'? If there is such a limit, what ascii value corresponds to it?

[ulz_peter]

Regarding the FastQ read name: Illumina has its own way of naming the reads:
see here: http://en.wikipedia.org/wiki/FASTQ_format
Generally I don't know any fastq naming convention but overriding the titles for illumina data would cause loss of information (afaik the coordinates can be used for optical duplicate detection)

Concerning the qualities: that depends on many factor I don't think there is a general threshold value. Moreover there are different quality encoding standards each using it's own range of ascii values...

[vbholaiit]

Random access

Hi all, I gave a query regarding importance of random access to individual reads in a compressed FASTQ file? If so then what can be the possible applications which random access can be used for? I'll appreciate any help in this regard. Thank you in advance