SEQanswers

Go Back   SEQanswers > Bioinformatics > Open Genomics Engine Project



Similar Threads
Thread Thread Starter Forum Replies Last Post
HTSeq: A Python framework to work with high-throughput sequencing data Simon Anders Bioinformatics 191 10-29-2015 06:33 AM
PubMed: An overview of the Hadoop/MapReduce/HBase framework and its current applicati Newsbot! Literature Watch 1 12-26-2013 12:48 AM
ChIP-Seq: coMOTIF: A Mixture Framework for Identifying Transcription Factor and a Co- Newsbot! Literature Watch 0 07-22-2011 02:00 AM
PubMed: A framework for analysis of metagenomic sequencing data. Newsbot! Literature Watch 0 12-02-2010 02:00 AM
Java framework Goby 1.8 released Fabien Campagne Bioinformatics 0 11-04-2010 02:25 PM

Reply
 
Thread Tools
Old 06-11-2012, 07:51 AM   #1
desaila
Junior Member
 
Location: Boston

Join Date: Oct 2010
Posts: 7
Default yet another framework?

Anyone who follows the web technologies is very familiar with the # of web frameworks there are, and the ever increasing number of them.

I was pretty excited when I saw the announcement, but was quickly disappointed to find out this doesn't do much, if anything, beyond what Picard and GATK already provide.

What are the advantages of using this framework over using Picard and GATK with BioJava? Development time will be faster in that scenario, there is more documentation, and they have a more liberal license (for now) for commercial users.
desaila is offline   Reply With Quote
Old 06-11-2012, 09:49 AM   #2
zee
NGS specialist
 
Location: Malaysia

Join Date: Apr 2008
Posts: 249
Default

This is a good point. How does openge make it easier us to run good genomics workflows. Perhaps an easyrun function for exome, RNASeq,etc, analysis will be a great thing to have in this system.

desaila, I think the current version of openge will provide some enhanced functions for doing thing with alignments, etc, since it is written in C.

Anybody else care to comment?
zee is offline   Reply With Quote
Old 06-11-2012, 11:28 AM   #3
oiiio
Senior Member
 
Location: USA

Join Date: Jan 2011
Posts: 104
Default

@desaila, i'd like to chime in on your thread here, which is a good distinguishment question-

While there is overlap in the ultimate goals of Picard/GATK and the openge framework, as zee mentioned there are great differences in functionality. In terms of HPC, the openge binaries that were released are much more suited for parallelized performance (both CPU and GPU) in all of its available tools, and can also stream data similar to samtools and many our other favourite programs. Not mention that as a C program, the hassle of matching java versions is gone (for example, between your cluster nodes and your workspace), and can be opened on all 'nix distros with minimal worry.

Aside from performance and command line usability, there is also the workflow creation aspect of openge. Although still in development, an example of its use would be that a user could potentially put together a workflow for a specific type of genomic data that he has, and share it via some kind of devblog or SEQanswers thread with others who may need the exact same thing - saving this subset of analysts many hours. The openge 'wrapped' workflow would then be able to execute on these different machines without trouble. Not to mention advantages of using the openge wrappers, such as automated error handling, and other features of an appealing workflow manager.

Hopefully this can clear up your separation of the Picard/GATK and openge. You are also correct that there is more documentation, etc. but openge should be given its chance to grow as well It is a concept that has its own HPC functionality, but also a big potential for carrying a community.
oiiio is offline   Reply With Quote
Old 06-11-2012, 12:19 PM   #4
desaila
Junior Member
 
Location: Boston

Join Date: Oct 2010
Posts: 7
Default

@oiiio
What I don't understand, in your reply is how is OpenGE differentiating itself in it's parallelization model? GATK uses map/reduce and scales with the # of cpu's you have. It looks like this does the same thing by actually porting that functionality from GATK.

Let me clarify and say that I'm not trying to be pessimistic, and that I'm completely in support of this framework -- especially if they open up the commercial side of their license. However, it realistically does nothing that GATK and Picard don't already do. You can create workflows and things in GATK/Picard too.

Here's where I'm optimistic: the fact that Nvidia is touting this as a compute solution is implying that there will be cuda/openCL elements in the future and _that_ is certainly something GATK will not have.
desaila is offline   Reply With Quote
Old 06-11-2012, 01:14 PM   #5
adaptivegenome
Super Moderator
 
Location: US

Join Date: Nov 2009
Posts: 437
Default

Quote:
Originally Posted by desaila View Post
@oiiio
What I don't understand, in your reply is how is OpenGE differentiating itself in it's parallelization model? GATK uses map/reduce and scales with the # of cpu's you have. It looks like this does the same thing by actually porting that functionality from GATK.
I would encourage you to take a look at the code. It appears you have posted only twice to SEQanswers prior to posting this thread. So, I have no way to gauge your technical level but I think you are misunderstanding the implementation. OpenGE implements multithreading that takes advantage of newer CPUs (and GPUs) that have lots of cores. This is not something that is ported from GATK and it is not map/reduce.

The comparisons to GATK and other tools are inevitable. It's an easy question to ask. We have no interest in replacing or competing with GATK or other tools though. We are looking to complement existing tools, but our primary focus or interest is:

(1) High performance code that maximizes compute power than be utilized on commodity hardware, compute nodes, etc.

(2) A workflow to integrate OpenGE with other tools (and vice versa). Here I can't say too much more (because I do not want to sell you on tales of the future) but the idea is to build graphical and sharable analysis pipelines. And I do not mean scripts. I mean fully managed workflows that track progress, detect and correct errors, etc.

I again encourage you to check out the code, follow the forum and give the code a chance if you want. Furthermore, we would welcome your thoughts and everyone else's on how to decide upon and prioritize future features.
adaptivegenome is offline   Reply With Quote
Old 06-11-2012, 01:44 PM   #6
desaila
Junior Member
 
Location: Boston

Join Date: Oct 2010
Posts: 7
Default

Quote:
Originally Posted by genericforms View Post
I would encourage you to take a look at the code. It appears you have posted only twice to SEQanswers prior to posting this thread. So, I have no way to gauge your technical level but I think you are misunderstanding the implementation. OpenGE implements multithreading that takes advantage of newer CPUs (and GPUs) that have lots of cores. This is not something that is ported from GATK and it is not map/reduce.
https://github.com/adaptivegenome/op...thm_module.cpp and the .h

Correct me if I'm wrong, but inheriting classes of AlgorithmModule are simply asynchronous threads, or they might use a ThreadPool -- but browsing through the code it seems like the only thing that uses the ThreadPools is the read_sorter.{h, cpp}

So, asynchronous threads on a single box is the breakthrough here? If that's the case, GATK scales to the same levels as OpenGE does right now. Clearly that will change once this starts using CUDA.
desaila is offline   Reply With Quote
Old 06-11-2012, 01:57 PM   #7
adaptivegenome
Super Moderator
 
Location: US

Join Date: Nov 2009
Posts: 437
Default

Quote:
Originally Posted by desaila View Post
So, asynchronous threads on a single box is the breakthrough here? If that's the case, GATK scales to the same levels as OpenGE does right now. Clearly that will change once this starts using CUDA.
I hope that future releases impress you more
adaptivegenome is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 11:38 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2018, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO