Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Biomart getBM and qmake parallelization

    Hi there,

    I'm using Biomart in my R script to retrieve gene symbols from ensembl IDs.
    Here is the part of my script making a call to Biomart.

    Code:
    resultTable <- DEXSeqResults(dxd)
    
    ensembl67=useMart(host="may2012.archive.ensembl.org",biomart="ENSEMBL_MART_ENSEMBL")
    mart <- useDataset("mmusculus_gene_ensembl",ensembl67)
    
    ensemblGeneId <- resultTable$groupID
    
    geneNames <-getBM(filters="ensembl_gene_id",
    attributes=c("ensembl_gene_id","external_gene_id"),
    values=ensemblGeneId,
    mart=mart)
    When I execute my script on a single computer, everything goes fine. But when I try to execute it in parallel on a cluster (running on SGE) with a qmake, some jobs work fine and some others produce the following error :

    Code:
    Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  :
      line 1 did not have 3 elements
    Calls: getBM -> read.table -> scan
    I then set the getBM verbose attribute to TRUE and I got the resulting response from the server for failed jobs :

    Code:
    Results from server:
    [1] "\n<html><!-- InstanceBegin template=\"/Templates/biomart_standalone.dwt\" codeOutsideHTMLIsLocked=\"false\" -->\t\n\t<head> \n\t\t<!-- InstanceBeginEditable name=\"doctitle\" -->\n<meta name=\"Description\" content=\"BioMart is a simple and robust data integration system for large scale data querying. It has been designed to provide researchers with an easy and interactive access to both the wealth of data available on the Internet and for in house data integration.\">\n<meta name=\"keywords\" content=\"BioMart, DATABASE, SEARCHES, protein\">\n\n<title>BioMart Webservice</title>\n<script language = \"javascript\">\nvar currentTab=\"Martservice\";\n</script>\n\n<!-- InstanceEndEditable -->\n<link rel=\"stylesheet\" href=\"/biomart/mview/martview.css\" type=\"text/css\" />\n <script language=\"javascript\" type=\"text/javascript\">\n\n\tfunction colourTab(){\n\t\tif((document.getElementById(currentTab))){\n\t\t\t(document.getElementById(currentTab)).className='menuh';\n\t\t}\n\t}\n\n\tfunction rollOver(overTab){\n\t\tif(currentTab!=overTab){\n\t\t\t(document.getElementById(overTab)).className='menuh';\n\t\t}\n\t}\n\tfunction rollOut(overTab){\n\t\tif(currentTab!=overTab){\n\t\t\t(document.getElementById(overTab)).className='menu';\n\t\t}\n\t}\n\n</script>\n\n</head>\n<body style=\"margin: 0px;\" onLoad=\"colourTab()\" >\n\n<div style=\" width:100%; height:15%;\">\n<table height=\"65%\" width=\"100%\" border=\"0\" cellspacing=\"0\" cellpadding=\"0\">\n\t<tr>\n\t\t\n    <td width=\"270\" align=\"right\">\n    <a href=\"index.html\">\n    \t\t<img style = \"margin: 15px 0px 5px 8px;\" height=\"33\" width=\"105\" src=\"/martview/images/biomart-logo.gif\" alt=\"EBI Home Page\" border=\"0\" />\n    </a>\n    </td>\n\t\t <td valign=\"top\" align=\"right\" width=\"100%\">\n\n  \t</td>\n\t</tr>\n</table>\n\n\n<table border=\"0\"  valign=\"bottom\" cellpadding=\"\" cellspacing=\"2\" height=\"35%\" width=\"1000\">\n        <tr>\n        \n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t          <div class=\"menu\" id=\"BioMart Home\" onClick=\"location='/martview/../index.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">HOME</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n          <div   class=\"menu\"  id=\"Documentation\" onClick=\"location='/martview/../install.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">DOCS</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t  <div   class=\"menu\"  id=\"News\" onClick=\"location='/martview/../news.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">NEWS</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n          <div   class=\"menu\"  id=\"Contact\" onClick=\"location='/martview/../contact.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">CONTACT</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div    class=\"menu\"  id=\"Credits\" onClick=\"location='/martview/../credits.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">CREDITS</div> \n\t\t\t</td>\n\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div     class=\"menuh\" id=\"Service\" onClick=\"location='/biomart/martservice'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">MARTSERVICE</div>\n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div    class=\"menu\"  id=\"Query\" onClick=\"location='/biomart/martview'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">MARTVIEW</div>    \n\t\t\t</td>\n\t\t\t\n\t\t</tr>\n</table>\n\n</div>\n\n<div class=\"mart_header\" style=\" width:100%; height:85%;\">\n<table bgcolor=\"black\" width=\"100%\" height=\"5\" border=\"0\" cellspacing=\"0\" cellpadding=\"0\">\n             <tr> \n                <td bgcolor=\"black\" width=\"100%\" height=\"20\" align=\"center\"><!-- InstanceBeginEditable name=\"topnav\" --><nobr></nobr><!-- InstanceEndEditable --></td>\n              </tr> \n</table>\n\n\n<h3>Martservices usage:</h3><br>\n(a) Querying BioMart<br><br>\nTo submit a query using our webservices generate an XML document conforming to our Query XML syntax. This can be achieved simply by building up your query using MartView and hitting the XML button. This XML should be posted to http://may2012.archive.ensembl.org/martservice attached to a single parameter of <b>query</b>. For example you could either:<br><br> \n- save your query as Query.xml and then POST this using the <a href=\"http://cvs.sanger.ac.uk/cgi-bin/viewcvs.cgi/*checkout*/biomart-perl/scripts/webExample.pl?root=BioMart&content-type=text%2Fplain\">webExample.pl</a> script in our biomart-perl/scripts installation.<br>\n- submit using wget: wget -O results.txt 'http://may2012.archive.ensembl.org/biomart/martservice?query=MY_XML' replacing MY_XML with the XML obtained above, first removing any new lines.<br>\n<br>\n(b) Retrieving Meta Data<br>\n<br>\n- to retrieve registry information: <a href=\"/biomart/martservice?type=registry\">http://may2012.archive.ensembl.org/biomart/martservice?type=registry</a><br>\n- to retrieve datasets available for a mart: <a href=\"/biomart/martservice?type=datasets&mart=ENSEMBL_MART_ENSEMBL\">http://may2012.archive.ensembl.org/biomart/martservice?type=datasets&mart=ENSEMBL_MART_ENSEMBL</a><br>\n- to retrieve attributes available for a dataset: <a href=\"/biomart/martservice?type=attributes&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=attributes&datatset=oanatinus_gene_ensembl</a><br>\n- to retrieve filters available for a dataset: <a href=\"/biomart/martservice?type=filters&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=filters&datatset=oanatinus_gene_ensembl</a><br>\n- to retrieve configuration for a dataset: <a href=\"/biomart/martservice?type=configuration&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=configuration&datatset=oanatinus_gene_ensembl</a><br>\n<br>\nFor more information see the webservices section of our <a href=\"http://www.biomart.org/user-docs.pdf\">documentation</a>.\n</div>\n</body>\n</html>\n"
    attr(,"Content-Type")
                      charset 
    "text/plain"      "utf-8"
    whereas in successfull jobs the response looks like that :

    Code:
    Results from server:
    [1] "ENSMUSG00000085299\tGm16627\nENSMUSG00000067978\tVmn2r-ps113\nENSMUSG00000020530\tGgnbp2\nENSMUSG00000074155\tKlk5\nENSMUSG00000087276\tGm11378\nENSMUSG00000048534\tAmica1 ......
    I can also mention that queries in both successfull and failed jobs look the same, and if I re-run a qmake the errors are not necessarly on the same data.

    I guess it's a connection problem but I would like to know if somebody already experienced it and solved it.

    Thanks

    PS : I'm using R-3.1.0 and biomaRt_2.20.0

  • #2
    **may be** the biomart server doesn't want to handle too many requests coming from the same IP at the same time ? As a test I would flag those specific targets with NOTPARALLEL ( http://www.gnu.org/software/make/man...pecial-Targets ) and add a 'sleep' to wait a few seconds before running another target.

    Comment


    • #3
      OHH! BTW, "BONJOUR Erwan" :-)

      Comment


      • #4
        Originally posted by lindenb View Post
        **may be** the biomart server doesn't want to handle too many requests coming from the same IP at the same time ?
        Yep that's what I think too, but I didn't find any informations about that in the documentation or on the web.

        As a test I would flag those specific targets with NOTPARALLEL ( http://www.gnu.org/software/make/man...pecial-Targets ) and add a 'sleep' to wait a few seconds before running another target.
        I will try that.

        Merci Pierre

        Comment

        Latest Articles

        Collapse

        • seqadmin
          Essential Discoveries and Tools in Epitranscriptomics
          by seqadmin


          The field of epigenetics has traditionally concentrated more on DNA and how changes like methylation and phosphorylation of histones impact gene expression and regulation. However, our increased understanding of RNA modifications and their importance in cellular processes has led to a rise in epitranscriptomics research. “Epitranscriptomics brings together the concepts of epigenetics and gene expression,” explained Adrien Leger, PhD, Principal Research Scientist on Modified Bases...
          Yesterday, 07:01 AM
        • seqadmin
          Current Approaches to Protein Sequencing
          by seqadmin


          Proteins are often described as the workhorses of the cell, and identifying their sequences is key to understanding their role in biological processes and disease. Currently, the most common technique used to determine protein sequences is mass spectrometry. While still a valuable tool, mass spectrometry faces several limitations and requires a highly experienced scientist familiar with the equipment to operate it. Additionally, other proteomic methods, like affinity assays, are constrained...
          04-04-2024, 04:25 PM

        ad_right_rmr

        Collapse

        News

        Collapse

        Topics Statistics Last Post
        Started by seqadmin, 04-11-2024, 12:08 PM
        0 responses
        39 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 04-10-2024, 10:19 PM
        0 responses
        41 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 04-10-2024, 09:21 AM
        0 responses
        35 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 04-04-2024, 09:00 AM
        0 responses
        55 views
        0 likes
        Last Post seqadmin  
        Working...
        X