Free & Open Environment for NGS analysis: Galaxy (http://usegalaxy.org)

[nekrut]

Originally Posted by tweist

This is a great platform having many useful tools available from the same place and interoperable. However, uploading big dataset to Galaxy is a challenge and a road block. I tried to upload a ILMN dataset of a few Gb and not yet done after a day and half. Any plan to enable Aspera-like tools to accelerate the data transfer speed?

Also, I think a blast tool that allows search against arbitrary database would be useful too. Any plan to add this feature?

Great job!

Hi, upload limitations are going to be a problem for all of us as it is not Galaxy issue per se. To upload your files try this:
- gzip them
- put them to a web-accessible loocation
- paste a URL pointing to your files to upload tool as shown in the attached image

Aspera is incredibly expensive to license, and we're not convinced it really accelerates things that much. There are some open source alternative we will be taking a look at

Yes, blasting against custom db is on the to do list as numerous people want this to happen.

[tweist]

Hi,

Have you looked at open source alternative such as UDT to address the data transfer bottleneck? Also, is there a limit on how much data that I can save in my galaxy account? Thanks!

[nekrut]

Originally Posted by tweist

Hi,
Also, is there a limit on how much data that I can save in my galaxy account? Thanks!

No limit as of now

[ezilybored]

Originally Posted by nekrut

Hi, upload limitations are going to be a problem for all of us as it is not Galaxy issue per se. To upload your files try this:
- gzip them
- put them to a web-accessible loocation
- paste a URL pointing to your files to upload tool as shown in the attached image

Aspera is incredibly expensive to license, and we're not convinced it really accelerates things that much. There are some open source alternative we will be taking a look at

Yes, blasting against custom db is on the to do list as numerous people want this to happen.

Hi there, sorry if i am being naive but will galaxy be able to read the files as gzips? a colleague of mine tried recently and said that it wasnt able to read the file format.

Ben

[KevinLam]

I have uploaded a gzipped 300 MB .qual file and it works for doing box plot. perhaps your fren didn't complete the upload?

[nekrut]

Originally Posted by ezilybored

Hi there, sorry if i am being naive but will galaxy be able to read the files as gzips? a colleague of mine tried recently and said that it wasnt able to read the file format.

Ben

Yes, you can. The best way to get data in is to upload gzipped files via URL (http and ftp both work). See post #21 in the thread (scroll up).

[James]

I'm having a similar problem to Ben, I have illumina fastq files that are gzipped. I can upload them in a few minutes but then when I try to do the groomer i get this error:

An error occurred running this job: Traceback (most recent call last):
File "/galaxy/home/g2main/galaxy_main/tools/fastq/fastq_groomer.py", line 37, in <module>
if __name__ == "__main__": main()
File "/galaxy/home/g2main/galaxy_main/tools/fastq/fastq_groomer.py", line 18, in main

Is there some sort of facility to unzip gz files? Thanks, J

[blankenberg]

Originally Posted by ezilybored

Hi there, sorry if i am being naive but will galaxy be able to read the files as gzips? a colleague of mine tried recently and said that it wasnt able to read the file format.

Ben

Originally Posted by James

I'm having a similar problem to Ben, I have illumina fastq files that are gzipped. I can upload them in a few minutes but then when I try to do the groomer i get this error:

An error occurred running this job: Traceback (most recent call last):
File "/galaxy/home/g2main/galaxy_main/tools/fastq/fastq_groomer.py", line 37, in <module>
if __name__ == "__main__": main()
File "/galaxy/home/g2main/galaxy_main/tools/fastq/fastq_groomer.py", line 18, in main

Is there some sort of facility to unzip gz files? Thanks, J

Galaxy does support the upload of gzip'd files.

Galaxy does not yet accept .tar(.gz) nor .zip files. Each file needs to be gzip'd individually (without tar'ing) in order for it it work with the upload tool at this time.

Support for .tar(.gz) and .zip files is being investigated.

Please don't hesitate to let us know if we can be of further help. Thanks for using Galaxy.

[Calico]

Is there any way that Galaxy can be used to analyze data from a Helicos machine?

[nekrut]

Originally Posted by Calico

Is there any way that Galaxy can be used to analyze data from a Helicos machine?

We have not see the data ourselves yet, but I just asked Helicos guys to send us a few files. Do you have any to share?

[Calico]

Not yet, but I am about to subject some samples to Helicos sequencing.

[steven]

Originally Posted by nekrut

We have not see the data ourselves yet, but I just asked Helicos guys to send us a few files. Do you have any to share?

I do not know about the raw output from the Heliscope, but what i have seen was fasta format for the reads and BED format for the alignments. Some Helicos datasets are available there: http://open.helicosbio.com/mwiki/index.php/Datasets (a free registration is maybe required though). Actually, Helicos RNA-seq data can also be obtained from the UCSC Table Browser and sent to Galaxy, or from the UCSC ftp download page.

[ohofmann]

Problem with Helicos is that it doesn't come as FASTQ (quality scores are assigned to reads, not nucleotides within the reads). The best (only?) way to handle this is to convert the original format to FASTA after filtering at a given quality cutoff and take it from there. There are also tools to create 'fake' FASTQ files by setting the quality score to a uniform value if a tool absolutely requires FASTQ.

You almost need to use their filtering workflow anyway as it gets rid of a number of artifacts which are otherwise difficult to remove.

[brian.g]

UDT (udt.sf.net) should work. It is a library with C++/JNI/.net API and only has a simple command line tool. You can also look at HSCP (http://sourceforge.net/projects/hscp/), which is a SCP-like tool but uses UDT for data transfer. Both UDT and HSCP are open source and free to use.

Originally Posted by nekrut

Hi, upload limitations are going to be a problem for all of us as it is not Galaxy issue per se. To upload your files try this:
- gzip them
- put them to a web-accessible loocation
- paste a URL pointing to your files to upload tool as shown in the attached image

Aspera is incredibly expensive to license, and we're not convinced it really accelerates things that much. There are some open source alternative we will be taking a look at

Yes, blasting against custom db is on the to do list as numerous people want this to happen.

[nekrut]

Originally Posted by brian.g

UDT (udt.sf.net) should work. It is a library with C++/JNI/.net API and only has a simple command line tool. You can also look at HSCP (http://sourceforge.net/projects/hscp/), which is a SCP-like tool but uses UDT for data transfer. Both UDT and HSCP are open source and free to use.

Just talked to Bob Grossman. Will try...

[nilshomer]

I have been playing with Galaxy on my local machine for the past few days. It is great piece of software, and was relatively easy to plug in new tools. Thank-you again.

[nekrut]

Nils:

Thanks for sending us your wrappers. They will be on public site shortly!

[Abigail41]

If there is anything the site or community can do to support this effort, don't hesitate

[Abigail41]

I think it is important to include bfast as perm/bowtie do not do gapped alignment.