With higher density runs, I'm curious to know how people are handling data. We have Illumina and AB instruments and handling the data off the instruments is becoming very cumbersome. We currently have 1.2PB of storage which is constantly being pruned for space since we're ramping up production.
We perform the analysis of Illumina runs from Intensities (I.e. Running Bustard from Intensities) and do not work from the on instrument Basecalls. No Images are copied over other than thumbnails for troubleshooting purposes. Prior to backing up our data we perform a cleanup (delete the on instrument Basecall directory, leave the export, sequence and Summary.htm related files in the Gerald directory).
For AB data, we keep only the csfasta and qual files and the analyzed data output. All other intermediate files are deleted prior to backup.
Any advice from others on handling all the data?
We perform the analysis of Illumina runs from Intensities (I.e. Running Bustard from Intensities) and do not work from the on instrument Basecalls. No Images are copied over other than thumbnails for troubleshooting purposes. Prior to backing up our data we perform a cleanup (delete the on instrument Basecall directory, leave the export, sequence and Summary.htm related files in the Gerald directory).
For AB data, we keep only the csfasta and qual files and the analyzed data output. All other intermediate files are deleted prior to backup.
Any advice from others on handling all the data?
Comment