Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Quick & dirty Galaxy installation in a virtual machine

    QUICK & DIRTY GALAXY INSTALLATION IN A VIRTUAL MACHINE

    (with basic tools, hg19 reference, ftp upload)


    ### DISCLAIMERS: ###
    1. This build is for someone who wants to just get galaxy up and running and be able to use it, with minimal Linux expertise, minimal time & cost, and on existing hardware.
    2. There is ZERO focus on security or data privacy in this build. See the title.
    3. I'm not in any way affiliated with the Galaxy team. I just find this style handy for basic setup of my Galaxy VMs and thought others might benefit.
    4. This build should work on any hardware capable of running virtualbox & 64-bit VMs (RAM & CPU permitting).
    5. The links WILL break & become outdated over time, so amend as necessary.
    6. Sorry about the mess.
    7. Enjoy =)


    **** RESOURCES ****
    http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup
    http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies
    http://www.agapow.net/science/bioinformatics/galaxy/installing-galaxy
    http://wiki.g2.bx.psu.edu/Admin/Config/Upload%20via%20FTP

    ### OS Setup ###
    [Install ubuntu 11.10 desktop 64 bit in virtualbox, vmware, or bare metal. Virtualbox is free and cross-platform.]
    - Give the VM's hard disk a bit of space, as the tools & data will eat it up. I'd say at least 100 GB.
    - root user: galaxy/galaxy
    [optional for vmware]: install vmware tools & reboot

    [Once OS is installed and at command line:]
    sudo apt-get update
    sudo apt-get install openssh-server
    sudo nano /etc/ssh/sshd_config
    [add the line:]
    UseDNS no
    [save & exit (ctrl-x, y, enter)]
    sudo /etc/init.d/ssh restart


    ### Static ip ###
    [optional] Give the server a static IP & set network settings appropriately
    - We'll use 10.71.10.26 here for simplicity, but you could equally set up a DNS name

    ### Galaxy setup ###
    mkdir ~/galaxy-python
    ln -s /usr/bin/python ~/galaxy-python/python
    export PATH=~/galaxy-python:$PATH
    sudo apt-get install mercurial
    hg clone https://bitbucket.org/galaxy/galaxy-dist/
    sudo apt-get install python-dev -y
    sudo apt-get install openjdk-6-jre -y
    sudo apt-get install build-essential -y
    cd ~/galaxy-dist
    [let the run script do its initial configuration:]
    sh run.sh

    [open a second terminal to continue]

    *** IF we get errors with building eggs, we can install them manually like so:
    python scripts/scramble.py -e Whoosh
    python scripts/scramble.py -e Cheetah
    python scripts/scramble.py -e twill
    python scripts/scramble.py -e WebHelpers
    python scripts/scramble.py -e docutils
    python scripts/scramble.py -e python_lzo
    python scripts/scramble.py -e ssh
    python scripts/scramble.py -e GeneTrack
    python scripts/scramble.py -e Paste
    python scripts/scramble.py -e PasteScript
    python scripts/scramble.py -e nose
    python scripts/scramble.py -e pycrypto
    ***

    [Assuming galaxy is up & running, we now need to kill it manually:]
    [find the PIDs of "sh run.sh" and "python ./scripts/paster.py serve universe_wsgi.ini", and use the kill command:]
    ps -au
    kill [PIDs]

    [open galaxy server to other machines on the network:]
    nano universe_wsgi.ini
    [add the line to [server:main] section:]
    host = 0.0.0.0
    [save & exit]
    sh run.sh

    [access from another machine's web browser, e.g.:]
    http://10.71.10.26:8080


    ### Configure Galaxy to run with postgresql ###
    [first, kill galaxy manually]
    sudo apt-get install postgresql
    sudo -u postgres psql postgres
    \password postgres
    [enter "postgres" as the password]
    ctrl-d
    psql -h localhost -U postgres
    CREATE DATABASE galaxy_prod;
    GRANT ALL ON DATABASE galaxy_prod TO postgres;
    ctrl-d
    nano ~/galaxy-dist/universe_wsgi.ini
    [add the following line to the [app:main] section of the config file:]
    database_connection = postgres://postgresostgres@localhost:5432/galaxy_prod
    [save & exit]
    cd ~/galaxy-dist
    python scripts/scramble.py -e psycopg2
    sh run.sh
    [tables will migrate to postgresql]
    ###############


    ### Configure FTP upload ###
    chmod 777 /home/galaxy/galaxy-dist/database/files
    nano ~/galaxy-dist/universe_wsgi.ini
    [add the lines to [app:main] section:]
    ftp_upload_dir = /home/galaxy/galaxy-dist/database/files/
    ftp_upload_site = 10.71.10.26
    [save & close]
    createuser -SDR galaxyftp -h localhost -U postgres
    [password "postgres"]
    psql -h localhost -U postgres -d galaxy_prod
    ALTER ROLE galaxyftp PASSWORD 'dbpassword';
    GRANT SELECT ON galaxy_user TO galaxyftp;
    ctrl-d
    sudo apt-get install proftpd-mod-pgsql -y
    [run proftpd: from standalone]
    sudo nano /etc/proftpd/proftpd.conf

    [Lines to modify:]
    ServerName "Galaxy FTP"
    Umask 000

    [Lines to add (AFTER "Include /etc/proftpd/modules.conf" line):]
    Include /etc/proftpd/sql.conf
    SyslogFacility DAEMON
    SyslogLevel debug
    DefaultRoot ~
    CreateHome on 777 dirmode 777
    AllowStoreRestart on
    <Limit SITE_CHMOD>
    DenyAll
    </Limit>
    <Limit RETR>
    DenyAll
    </Limit>
    AuthPAM off
    SQLPasswordEngine on
    SQLPasswordEncoding hex
    SQLEngine on
    SQLBackend postgres
    SQLConnectInfo galaxy_prod@localhost postgres postgres
    SQLAuthTypes SHA1
    SQLAuthenticate users
    SQLDefaultHomedir /var/opt/local/proftpd
    SQLUserInfo custom:/LookupGalaxyUser
    SQLNamedQuery LookupGalaxyUser SELECT "email,password,'512','512','/home/galaxy/galaxy-dist/database/files/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"

    [save & quit]
    sudo nano /etc/proftpd/modules.conf
    [uncomment the following:]
    LoadModule mod_sql.c
    LoadModule mod_sql_postgres.c
    LoadModule mod_sql_passwd.c

    [save & exit]
    sudo service proftpd start
    kill galaxy & restart it
    ###############


    ### Start Galaxy automatically at bootup ###
    sudo nano /etc/rc.local
    [add the lines:]
    sleep 10
    sh ~/galaxy-dist/run.sh --daemon
    [save & exit]
    ###############




    **** STARTING & STOPPING GALAXY MANUALLY ****
    [Once installed, galaxy can/should be started as a daemon:]
    sudo sh ~/galaxy-dist/run.sh --daemon

    [To stop galaxy when running as daemon:]
    sudo sh ~/galaxy-dist/run.sh --stop-daemon

    [To get galaxy status:]
    sudo sh ~/galaxy-dist/run.sh --status

    [Restart galaxy:]
    sudo sh ~/galaxy-dist/run.sh --monitor-restart
    *********************************************





    ########################
    configure external tools:
    ########################


    # Initial setup #
    sudo mkdir /usr/local/apps
    sudo chmod 755 /usr/local/apps
    sudo chown galaxy /usr/local/apps

    ### Blast ###
    sudo apt-get install ncbi-blast+
    ###############

    ### FastQC ###
    cd /usr/local/apps
    wget http://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.10.1.zip
    unzip fastqc_v0.10.1.zip
    cd FastQC
    chmod 755 fastqc
    sudo ln -s /usr/local/apps/FastQC/fastqc /usr/local/bin/fastqc
    cd ~/galaxy-dist/tool-data/shared/jars
    mkdir FastQC
    cd FastQC
    ln -s /usr/local/apps/FastQC/fastqc fastqc
    ###############

    ### samtools ###
    sudo apt-get install samtools
    ################

    ### fastx-toolkit ###
    sudo apt-get install fastx-toolkit
    ################

    ### bwa ###
    [NOTE: if bwa is updated, all reference indexes must be regenerated.]
    cd /usr/local/apps
    wget http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2
    tar -xjvf bwa-0.6.2.tar.bz2
    cd bwa-0.6.2
    sudo make
    sudo ln -s /usr/local/apps/bwa-0.6.2/bwa /usr/local/bin/
    sudo ln -s /usr/local/apps/bwa-0.6.2/*.pl /usr/local/bin/
    ################

    ### picard-tools ###
    cd /usr/local/apps/
    mkdir picard-tools
    cd picard-tools
    wget http://downloads.sourceforge.net/project/picard/picard-tools/1.74/picard-tools-1.74.zip
    unzip picard-tools-1.74.zip
    cd picard-tools-1.74.zip
    chmod +x *
    cd ~/galaxy-dist/tool-data/shared/jars
    mkdir picard
    cd picard
    ln -s /usr/local/apps/picard-tools/picard-tools-1.74/*.jar ./
    ################

    ### tophat & cufflinks ###
    cd /usr/local/apps
    wget http://tophat.cbcb.umd.edu/downloads/tophat-2.0.4.Linux_x86_64.tar.gz
    tar -xvf tophat-2.0.4.Linux_x86_64.tar.gz
    wget http://cufflinks.cbcb.umd.edu/downloads/cufflinks-2.0.2.Linux_x86_64.tar.gz
    tar -xvf cufflinks-2.0.2.Linux_x86_64.tar.gz
    [only for older versions of ubuntu < 11.10]:
    sudo apt-get install libboost*
    sudo ln -s /usr/local/apps/cufflinks-2.0.2.Linux_x86_64/* /usr/local/bin/
    sudo ln -s /usr/local/apps/tophat-2.0.4.Linux_x86_64/* /usr/local/bin/
    ###############

    ### bowtie ###
    cd /usr/local/apps
    wget http://downloads.sourceforge.net/project/bowtie-bio/bowtie/0.12.8/bowtie-0.12.8-linux-x86_64.zip
    unzip bowtie-0.12.8-linux-x86_64.zip
    sudo ln -s /usr/local/apps/bowtie-0.12.8/bowtie /usr/local/bin/bowtie
    sudo ln -s /usr/local/apps/bowtie-0.12.8/bowtie-build /usr/local/bin/bowtie-build
    sudo ln -s /usr/local/apps/bowtie-0.12.8/bowtie-inspect /usr/local/bin/bowtie-inspect
    ###############

    ### bowtie2 ###
    cd /usr/local/apps
    wget http://downloads.sourceforge.net/project/bowtie-bio/bowtie2/2.0.0-beta7/bowtie2-2.0.0-beta7-linux-x86_64.zip
    unzip bowtie2-2.0.0-beta7-linux-x86_64.zip
    sudo ln -s /usr/local/apps/bowtie2-2.0.0-beta7/bowtie2 /usr/local/bin/bowtie2
    sudo ln -s /usr/local/apps/bowtie2-2.0.0-beta7/bowtie2-align /usr/local/bin/bowtie2-align
    sudo ln -s /usr/local/apps/bowtie2-2.0.0-beta7/bowtie2-build /usr/local/bin/bowtie2-build
    sudo ln -s /usr/local/apps/bowtie2-2.0.0-beta7/bowtie2-inspect /usr/local/bin/bowtie2-inspect
    ###############





    ########################
    Data & reference genomes:
    ########################

    2bit conversion program:
    ###############
    mkdir /usr/local/data/twoBitToFa
    cd /usr/local/data/twoBitToFa
    wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitToFa
    chmod 755 twoBitToFa
    sudo ln -s /usr/local/apps/twoBitToFa /usr/local/bin/
    ###############

    adding a reference genome (fasta) & samtools index:
    ###############
    sudo mkdir /usr/local/data
    sudo chown galaxy /usr/local/data
    cd /usr/local/data
    mkdir samtools
    mkdir samtools/hg19/
    cd samtools/hg19/
    wget http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/hg19.2bit
    twoBitToFa hg19.2bit hg19.fa
    samtools faidx hg19.fa
    nano ~/galaxy-dist/tool-data/sam_fa_indices.loc
    [add the line:]
    index hg19 /usr/local/data/samtools/hg19/hg19.fa
    [save & close]
    nano ~/galaxy-dist/tool-data/faseq.loc
    [add the line:]
    hg19 /usr/local/data/samtools/hg19
    [save & close]
    ###############

    bowtie reference indexes:
    ###############
    [make sure ~8-10 GB of RAM is available before building indexes, or it may take forever]
    cd /usr/local/data/samtools/hg19
    bowtie-build -f hg19.fa hg19
    mkdir /usr/local/data/bowtie
    mkdir /usr/local/data/bowtie/hg19
    mv *.ebwt /usr/local/data/bowtie/hg19/
    nano ~/galaxy-dist/tool-data/bowtie_indices.loc
    [add the line:]
    hg19 hg19 hg19 /usr/local/data/bowtie/hg19/hg19
    [close & save]
    [restart galaxy]
    ###############


    bowtie2 reference indexes:
    ###############
    mkdir /usr/local/data/bowtie2
    mkdir /usr/local/data/bowtie2/hg19
    cd /usr/local/data/bowtie2/hg19
    wget ftp://ftp.cbcb.umd.edu/pub/data/bowtie2_indexes/incl/hg19.zip
    unzip hg19.zip
    nano ~/galaxy-dist/tool-data/bowtie2_indices.loc
    [add the line:]
    hg19 hg19 hg19 /usr/local/data/bowtie2/hg19/hg19
    [close & save]
    [restart galaxy & check the reference genome dropdown list in bowtie2]
    ###############

    BWA reference index:
    ###############
    [NOTE: if bwa is updated, all reference indexes must be regenerated.]
    mkdir /usr/local/data/bwa
    mkdir /usr/local/data/bwa/hg19
    cd /usr/local/data/samtools/hg19
    bwa index -a bwtsw hg19.fa
    mv hg19.fa.* /usr/local/data/bwa/hg19
    mv /usr/local/data/bwa/hg19/hg19.fa.fai /usr/local/data/samtools/hg19

    [MAKE SURE hg19.fa.fai STAYS IN samtools/hg19 DIR]

    nano ~/galaxy-dist/tool-data/bwa_index.loc
    [add the line:]
    hg19 hg19 hg19 /usr/local/data/bwa/hg19/hg19.fa
    [close & save]
    [restart galaxy & check the reference genome dropdown list in bowtie2]
    ###############

  • #2
    Hey,
    Thanks for tutorial. It made my work easy.
    Few of the links and steps are changed in above, I will try to put it, as soon as I complete it.

    My question is How to upload data using FTP ?
    as per your tutorial, what is user name and password ?
    how should i connect it ?

    Waiting for positive reply.
    Thank You.

    Comment

    Latest Articles

    Collapse

    • seqadmin
      Strategies for Sequencing Challenging Samples
      by seqadmin


      Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
      03-22-2024, 06:39 AM
    • seqadmin
      Techniques and Challenges in Conservation Genomics
      by seqadmin



      The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

      Avian Conservation
      Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
      03-08-2024, 10:41 AM

    ad_right_rmr

    Collapse

    News

    Collapse

    Topics Statistics Last Post
    Started by seqadmin, 03-27-2024, 06:37 PM
    0 responses
    12 views
    0 likes
    Last Post seqadmin  
    Started by seqadmin, 03-27-2024, 06:07 PM
    0 responses
    11 views
    0 likes
    Last Post seqadmin  
    Started by seqadmin, 03-22-2024, 10:03 AM
    0 responses
    53 views
    0 likes
    Last Post seqadmin  
    Started by seqadmin, 03-21-2024, 07:32 AM
    0 responses
    68 views
    0 likes
    Last Post seqadmin  
    Working...
    X