1000 Genomes in the Amazon cloud?

[throwaway]

These slides (p. 12) claim that the 1000 genomes BAM files were to be made available in the Amazon cloud in Febuary. That would be hugely useful to me, if so, but I can't seem to find them mentioned in the AWS Public Data Catalog. The claimed sizes for the Ensembl data sets listed there only come to about 250G combined.

Anyone know whether the BAM files actually were made publically available, and if so, how I can access them?

[nilshomer]

These slides (p. 12) claim that the 1000 genomes BAM files were to be made available in the Amazon cloud in Febuary. That would be hugely useful to me, if so, but I can't seem to find them mentioned in the AWS Public Data Catalog. The claimed sizes for the Ensembl data sets listed there only come to about 250G combined.

Anyone know whether the BAM files actually were made publically available, and if so, how I can access them?

Have you checked http://1000genomes.org/page.php?

[throwaway]

Yes. None of the information on that page or the data access page seems pertinent to Amazon storage. Searching for "amazon" or "aws" only turns up a reference to the Ensembl dataset, and doesn't make it cleaer how to access the BAM files.

[spenthil]

FYI - having the data in the AWS Public Data Catalog/S3 would be neat for people analyzing data on AWS EC2 (their cloud computing infrastructure) because transfering data within a region is free and very fast.

http://aws.amazon.com/s3/#pricing

There is no Data Transfer charge for data transferred between Amazon EC2 and Amazon S3 within the same Region or for data transferred between the Amazon EC2 Northern Virginia Region and the Amazon S3 US Standard Region.

[mndoci]

Location of 1000 genomes data on s3

s3://1000genomes

[bioinfosm]

How does one decrypt this s3 link to actually view/download the data?

[mndoci]

I would recommend installing S3fox or similar S3 browser. Since the bucket is public, just type /1000genomes into the location windows (every bucket ID in S3 is unique)

Screenshot: http://img.skitch.com/20100622-geb3s...ngw3rrecc1.jpg

and

http://img.skitch.com/20100622-nixij1xs2ni92jdeud3myrcskq.jpg

Each individual BAM file is addressable, e.g.

http://1000genomes.s3.amazonaws.com/Pilots_Bam/NA20828/NA20828.SLX.maq.SRP000033.2009_09.chrX_1_154913754.bam

(added later)

Also if you use curl or a browser and point to http://1000genomes.s3.amazonaws.com/ you'll get the XML response

[cope]

S3fox is great. I also like Bucket Explorer (commerical, but there's a 30-day trial). If the analysis tools you are using are expecting a filesystem, you could create an AMI and try using s3fs or subcloud. Due to the large size of the current dataset, EBS is just not an option as it is for other public datasets which are less than 1 TB.

On thing to be aware of is that because S3 does not natively understand directories, it is up to the clients to infer the directory structure. Unfortunately some clients differ in this, and so when you mount the bucket using something like s3fs, the directory structure may not appear correctly.

[spenthil]

Also, the AWS console now allows access to S3: http://aws.amazon.com/console/#s3

Haven't used it myself, but it looks nice enough.

[genesquared]

any update on this issue?

thanks!

[aliman]

not directly on-topic, but you can get past the 1TB limit for ebs volumes if you use raid 0 and stripe data across multiple volumes - there's a nice article by eric hammond - http://alestic.com/2009/06/ec2-ebs-raid