Issues with AGeS install

[pag]

I'm having issues with the installation of AGeS on ubuntu

I solved a lot of my compilation problems finally (most were with the kent/src compilation flags).

I did a force install on AGeS because several of the tests failed, but those appeared to be associated with bad scripts on remote computers and the local scripts that interface with those.

I have AGeS, pipa and diya installed along with
blast glimmer MUMmer RepeatFinder tRNAscan-SE
hmmer infernal rnammer
mod_ipr Pipeman
iprscan pftools phobius psort

However, when I load up the AGeS local webpage under "view annotations" (in firefox) the lower frame says "BHSAI

Error 404
Internal Error!

There is an I/O exception on the system. "

I don't know if this is an issue with apache installation, jetty or gbrowse2

Anyone know where I would look for an error log? Anyone have any experience with AGeS installation? Basically, the installation was such a C.F. of installations of packages through apt-get, cpan and manual downloads, configures and makes that I have no idea where to start. Nor do I want to start from scratch, as it took me several days to get to this point.

Also, it says under the annotation tab: started for a local database of mine. The "refresh status" button doesn't appear to be a button and has been saying "started" for the past 2-3 days (admittedly before I had worked out the kinks with gbrowse, diya or PIPA)

Edit: I notice that in the ages tarball that I downloaded, the INSTALL plain text file says that there's a file called ages_run that is referenced in the files in lib/org. This file is not in the tarball as specified. Is this the source of my problem?

[pag]

edit: the summary of this is, I believe, that ages and gbrowse are both using the cgi-bin...ages is treating /cgi-bin/gbrowse as a directory. gbrowse (or some part of ages) wants a gbrowse script file there.

The /cgi-bin/gbrowse/ "directory" should be /usr/bin/genome/ages/ages/web/webapps/vis/databases/. Moving the script files to a subdirectory of cgi-bin and making gbrowse a symlink to the databases directory takes away the error page and replaces it with a blank page. Re-examining /usr/bin/genome/ages/ages/web/webapps/vis/cgi-bin reveals that SOMETHING {edit: something in the Jetty server startup "rungui.sh" which calls java -cp $CLASSPATH org.bhsai.ages.AgesMain} wiped out my symlink and inserted files with names corresponding to the old contents of cgi-bin. However these files are empty (0 bytes) unlike their old status.

More details:
I see the following in my ages.log
[ WARN] 2012-07-18 09:21:47,182
/cgi-bin/gbrowse/AGeS_Demo_23182_01: java.io.IOException: Cannot run program "/usr/bin/genome/ages/ages/web/webapps/vis/cgi-bin/gbrowse" (in directory "/usr/bin/genome/ages/ages/web/webapps/vis/cgi-bin"): java.io.IOException: error=2, No such file or directory

I'm not sure which paths are relative to the webroot and which are relative to my system. I THINK port 9000 (ages) is using a different webroot than 9001 (gbrowse), given that localhost:9001 gives a gbrowse landing page, and 9000 comes up with a blank page.

In any case, I have the following directory structure by default

/usr/bin/genome/ages/ages/web/webapps$ ls
ages ages23182_localhost ROOT vis
(nothing of consequence in 23182 or ROOT)
/usr/bin/genome/ages/ages/web/webapps/ages
ages.jspx css error images inc index.jsp upload WEB-INF xmlhttp
(no cgi-bin under this directory...a zillion other things)
/usr/bin/genome/ages/ages/web/webapps/vis/gbrowse
annotation_help.html gbrowse gmod_cog.jpeg README.berkeleydb titlebar.css
contrib gbrowse.css i README.fedora titlebar-default.css
css gbrowse_karyotype_help.html images README.gff3 titlebar-safari.css
DISCLAIMER gbrowse_syn_help.html index.html README.MacOSX tmp
docs general_help.html js README.sessions tracks.css
example_scripts general_help.html.zh README README.Ubuntu tutorial
(the GBrowse.conf is below)
/usr/bin/genome/ages/ages/web/webapps/vis/cgi-bin
gbgff gbrowse_details gbrowse_est.tmpl gbrowse_img.tmpl gbrowse_moby gbrowse_syn.tmpl moby_server.tmpl
das gbgff.tmpl gbrowse_details.tmpl gbrowse_gmap gbrowse_key_img gbrowse_moby.tmpl gbrowse.tmpl
das.tmpl gbrowse gbrowse_est gbrowse_img gbrowse_login gbrowse_syn moby_server
/usr/bin/genome/ages/ages/web/webapps/vis/databases
admin_uploads AGeS_Demo_23182_02 AGeS_Demo_23182_04 gbrowse-mine ideograms yeast_chr1+2
AGeS_Demo_23182_01 AGeS_Demo_23182_03 gbrowse gbrowse_syn pop_demo yeast_scaffolds


# This is the global configuration for gbrowse
# It contains setting common to all data sources as well
# as the various constants formerly scattered amongst scripts and libraries

[GENERAL]
config_base = /usr/bin/genome/ages/ages/web/webapps/vis # overridden by environment variable GBROWSE_CONF
htdocs_base = /usr/bin/genome/ages/ages/web/webapps/vis
url_base = /gbrowse2
tmp_base = /var/tmp/gbrowse2
persistent_base = /var/lib/gbrowse2
userdata_base = /var/lib/gbrowse2/userdata
db_base = /var/lib/gbrowse2/databases

# These paths are relative to the url base
buttons = images/buttons
balloons = images/balloons
openid = images/openid
gbrowse_help = .
js = js

# These paths are relative to the config base
plugin_path = plugins
language_path = languages
templates_path = templates
moby_path = MobyServices

# session settings
session lock type = default

# If no session driver is set, then GBrowse will pick one for you.
# It will use db_file for the driver and storable for the serializer
# if available; otherwise falling back to the file driver and default serializer.
# Override driver guessing by setting these options
# The safest, but slowest session driver...
#session driver = driver:file;serializer:default
#session args = Directory /var/lib/gbrowse2/sessions

# to use the berkeley DB driver comment out the previous
33,1 Top

# line and uncomment these two
#session driver = driver:db_file;serializer:default
#session args = FileName /var/lib/gbrowse2/sessions.db

# DBI backend to use for uploaded userdata.
# The SQLite option is the easiest to use and the best tested.
# if this option is commented out, then GBrowse will
# try 'DBI::SQLite', 'berkeleydb', 'DBI::mysql' and finally the 'memory'
# backend.

# NOTICE the double semicolon! This is a DBI Perl module, NOT a DBI connection string.

# For the DBI::mysql adaptor to work, you must give the web user
# permission to create databases named userdata_% using the following
# mysql command:
# mysql> grant all privileges on `userdata\_%`.* to 'www-data'@localhost identified by 'foobar';
# Note the backquotes around the database name, and do be sure to replace "foobar" with
# a more secure password!

# for SQLite
#upload_db_adaptor = DBI::SQLite

# for Berkeleydb
#upload_db_adaptor = berkeleydb

# for mysql
#upload_db_adaptor = DBI::mysql
#upload_db_host = localhost
#upload_db_user = www-data
#upload_db_pass = foobar

# Debug settings
debug = 1
debug_external = 1
debug_plugins = 1
# Performance settings
renderfarm = 1
slave_timeout = 45
global_timeout = 60
search_timeout = 15
max_render_processes = 4 # try double number of CPU/cores

# Clean up settings (used by the gbrowse_clean script)
expire session = 1M # expire unused sessions after a month
expire cache = 2h # expire cached data if unmodified for >2 hours
expire uploads = 6w # expire uploaded data if unused for >6 weeks

# Appearance settings
truecolor = 1 # better appearance at the expense of larger image files

# The #include line following this one defines a transparent theme.
# Replace "transparent_colors" with "solid_gray_colors"
# or "warm_colors" for different themes.

#include "themes/warm_colors"
# #include "themes/transparent_colors"
# #include "themes/solid_gray_colors"

balloon tips = 1
titles are balloons = 1
plugins = FastaDumper RestrictionAnnotator SequenceDumper TrackDumper
overview grid = 0
region grid = 0
detail grid = 1
image widths = 450 640 800 1024
default width = 800
pad_left = 60
pad_right = 30
too many landmarks = 100
track listing style = categories # either "categories" or "facets"
# Loads more details image data than can fit on the screen. This lets the user drag and drop the details
# tracks, without loading more data from the server. A value of 1 is default (no drag and drop). A value
# of 3 loads one full width on each side.
details multiplier = 3

# where to link to when user clicks in detailed view
link = AUTO

# HTML to insert inside the <head></head> section
head =

# At the top of the HTML...
header =

# At the footer
footer = <hr />
<p style="font-size:small">The Generic Genome Browser. For questions about the data
at this site, please contact its webmaster. For support of the
browser software <i>only</i>, send email to
<a href="mailto:[email protected]">[email protected]</a>
or visit the <a href="http://www.gmod.org">GMOD Project</a> web pages.
</p>

# Various places where you can insert your own HTML -- see configuration docs
html1 =
html2 =
html3 =
html4 =
html5 =
html6 =
# Limits on genomic regions (can be overridden in datasource config files)
region segment = 200000
max segment = 5000000
default segment = 5000
zoom levels = 100 200 1000 2000 5000 10000 20000 50000 100000 200000 500000 1000000
region sizes = 1000 5000 10000 20000
default region = 5000
fine zoom = 10%

# keyword search maxima
max keyword results = 1000

###### Authorization ######
# uncomment this to use the PAM authentication plugin
# authentication plugin = PamAuthenticate


####### User Account Registration Database ######
# If no authentication plugin is defined, and
# "user_accounts" is true, then GBrowse
# will attempt to use its internal user accounts database
# to authenticate and/or register users.
user_accounts = 1
user_accounts_registration = 1
user_accounts_openid = 1

# Path to the database -- you will need to create this database and grant all
# privileges on it to the indicated user.

user_account_db = DBI:SQLite:/var/lib/gbrowse2/databases/users.sqlite

# For SQLite
# user_account_db = DBI:SQLite:/var/lib/gbrowse2/databases/users.sqlite

# For MySQL
# user_account_db = DBI:mysql:gbrowse_login;user=gbrowse;password=gbrowse
# The number of public files to display
public_files = 10

# What email gateway to use for outgoing registration confirmation messages.
# The full format is
# <smtp.server.com>:<port>:<encryption>:<username>:<password>
# Only the first field, the server name, is required.
# The port is assumed to be 25 unless ssl encryption is specified, in
# which case it defaults to 465.
# protocol is either "plain" or "ssl", "plain" assumed.
# username and password may be required by the gateway for authentication
#
# here are some common options
# smtp_gateway = localhost # localhost has properly configured outgoing gateway
# smtp_gateway = smtp.oicr.on.ca # indicated machine will forward email for you
# smtp_gateway = smtp.gmail.com:465:ssl:joe.user:secret # use gmail with account "joe.user" and password "secret"
# smtp_gateway = none # disable outgoing email
smtp_gateway = none # disable outgoing email

# These values are used in the login confirmation message sent during
# user registration. You may customize
application_name = GBrowse
application_name_long = The Generic Genome Browser
email_address = [email protected]

# name of the "superuser" who can add public tracks
admin_account = admin
admin_dbs = /var/lib/gbrowse2/databases/admin_uploads


######## DEFAULT DATASOURCE #########
default source = yeast

###############################################################################################
# Global settings for plugins (used for the PamAuthenticate plugin only at this point)
###############################################################################################
[PamAuthenticatelugin]
login hint = your UNIX account
login help = <span style="font-size:9pt">Please see your system administrator for help<br>if you have lost your password.</span>
pam service name = gbrowse

###############################################################################################
#
# DATASOURCE DEFINITIONS
# One stanza for each configured data source
#
###############################################################################################

[yeast]
description = Yeast chromosomes 1+2 (basic)
path = yeast_simple.conf

[yeast_advanced]
description = Yeast chromosomes 1+2 (advanced)
path = yeast_chr1+2.conf

[yeast_renderfarm]
description = Renderfarm demo (gbrowse_slave must be running!)
path = yeast_renderfarm.conf

[pop_demo]
description = Population Display Demo
path = pop_demo.conf

[volvox]
description = Tutorial database
path = volvox.conf

[pag]

~/Downloads/GBrowse-2.48$ sudo ./Build reconfig
NOTE: Run ./Build reconfig to change existing configuration.
Reuse previous configuration as defaults? [y ]y

**Beginning interactive configuration**
Directory for GBrowse's config and support files? [/usr/bin/genome/ages/ages/web/webapps/vis/gbrowse]
Directory for GBrowse's static images & HTML files? [/usr/bin/genome/ages/ages/web/webapps/vis/gbrowse]
Directory for GBrowse's temporary data [/tmp/gbrowse2]
Directory for GBrowse's sessions, uploaded tracks and other persistent data [/usr/bin/genome/ages/ages/web/webapps/vis/databases]
Directory for GBrowse's example databases [/usr/bin/genome/ages/ages/web/webapps/vis/databases]
Directory for GBrowse's CGI script executables? [/usr/bin/genome/ages/ages/web/webapps/vis/cgi-bin/]
Internet port to run demo web site on (for demo)? [8000]
Apache loadable module directory (for demo)? [/usr/lib/apache2/modules]
User account under which Apache daemon runs? [www-data]
Automatically update Apache config files to run GBrowse? [y] y
Automatically update system config files to run gbrowse-slave? [n]

**Interactive configuration done. Run './Build reconfig' to reconfigure**

**Paths reconfigured. Running "Build clean".
Cleaning up build files

[pag]

doing my symlink trick and removing permission for the www-data user to modify the gbrowse symlink (edit: AgesMain run by java during Jetty startup manages to overwrite this link anyway), I now get a "Resource not found!" rather than "Internal Server Error" at localhost:9001/cgi-bin/gbrowse/AGeS_Demo_23182_01
I get this error even with the following files in that directory (in addition to the fa and the gff):
index.html index.jsp index.php index.shtml index.xml

There does not appear to be any other files or directories with the name of simply AGeS_Demo_23182_01 anywhere within the web branch of AGeS.

As to the Jetty startup, running the following doesn't work either
/usr/bin/genome/ages/ages/web$ sudo chown xxx:www-data logs/ages.log
usr/bin/genome/ages/ages/web$ sudo -u www-data ./runserver.shjava.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:411)
at org.bhsai.ages.AgesMain.modifyGbrowseCgi(Unknown Source)
at org.bhsai.ages.AgesMain.initGbrowseCgi(Unknown Source)
at org.bhsai.ages.AgesMain.init(Unknown Source)
at org.bhsai.ages.AgesMain.run(Unknown Source)
at org.bhsai.ages.AgesMain.main(Unknown Source)

I don't know if this is because some environment variables need to be passed to the www-data user via sudo (made permissible in the /etc/sudoers)

the database directory and cgi-bin (and subs) seem to have the proper permissions for the www-data group to read/write

[hmdurand]

If in ages.log you are getting something like:

/cgi-bin/gbrowse/AGeS_Demo_10114_01: java.io.IOException: Cannot run program "/usr/local/src/ages/web/webapps/vis/cgi-bin/gbrowse" (in directory "/usr/local/src/ages/web/webapps/vis/cgi-bin"): java.io.IOException: error=2, No such file or directory

you should change the perl shebang (yes, still 2013 and devs are using shebangs!) in /usr/local/src/ages/web/webapps/vis/cgi-bin/ scripts to

#!/usr/bin/perl -w