imprting Raw reads into smrt Portal

[coldturkey]

Hello,

I am unable to import the Raw cell data (*.bas.h5 files) into smrt portal. The data is in the required format (as per the toubleshooting section of Pacbios github). However, when I scan these directories looking for the raw data I get "Scan completed. No new SMRT Cell data found.".

As I mentioned the raw data is there, in the correct format and with the required directory structure.

Has anyone encoutered this problem?

Regards

Brian

[GenoMax]

It sounds like someone else has already "imported" these SMRTcells into your instance SMRTPortal. In that case they are no longer recognized as "new" and you will see the message above.

If you do see the SMRTcells listed in the new job creation area (Design Job --> Create New) then you are good to go.

I am reasonably certain the following will work (going by my memory):

If the instance of SMRTportal you are using has user/group access controls implemented then you could try the following.

If you have access to the entire folder for that particular SMRTcells you are interested in then create a copy of the raw data folder(s) under a new file path. You can then use the SMRTportal to import the new copy in (Design Job --> Import and Manage --> Point to new path to scan).

[coldturkey]

Hi GenoMax,

The Cells have not been imported at all. This is the first time smrtportal will have been run. I have tried you alternative approach as well, numerous times.

Is it possible that the webserver still does not have permission to access the files and thus cannot see them?

currently the permissions are as below


drwxr-xr-x 2 brian brian 4096 Oct 29 19:58 Analysis_Results
-rw-r--r-- 1 brian brian 647311245 Oct 29 19:58 m121022_214434_42149_c100396442550000001523035711101217_s1_p0.mcd.h5
-rw-r--r-- 1 brian brian 3265 Oct 29 19:58 m121022_214434_42149_c100396442550000001523035711101217_s1_p0.metadata.xml
-rw-r--r-- 1 brian brian 3415 Oct 29 19:58 m121022_214434_42149_c100396442550000001523035711101217_s1_p0.xfer.xml

Brian

[GenoMax]

SMRTPortal is a strange beast to get going the first time.

Have you (or someone else) been able to import data from a test SMRTcell (lambda or other) and managed to get it analyzed in this instance of SMRTportal? I think there is a test dataset included in the smrtanalysis tarball.

Does user "smrtanalysis" have permissions to read the folders where these files are stored. Check for permissions all the way up the directory tree.

[coldturkey]

I cannot import the test data set either. I get the same error "no new cells found etc"

I think we are on to something regarding the user groups. I have no user "smrtanalysis" is this the issue.

[rhall]

If no SMRT Cells have been added yet, test that everything is set up correctly by adding

common/test

to the 'Import SMRT Cells' path list and scan it.

If this works and you still cannot import the other data can you ls the Analysis_Results directory and post the contents of the metadata.xml

[rhall]

Cross post.

Check which user is running tomcat

ps aux | grep tomcat

Then make sure that that user has access to the files.

[coldturkey]

Hi rhall,

That did not work the metadata.xml file contents are

<U+FEFF><?xml version="1.0" encoding="utf-8"?><Metadata xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://pacificbiosciences.com/PAP/Metadata.xsd"><InstCtrlVer>1.3.2.0.111587</InstCtrlVer><SigProcVer>[email protected]:8082, SwVer=1320.111587, HwVer=1.0</SigProcVer><Run><RunId>r000186_42149_121022</RunId><Name>ID1216_221012</Name><WhenCreated>2012-10-22T09:55:20</WhenCreated><WhenStarted>2012-10-22T14:02:40</WhenStarted></Run><Movie><WhenStarted>2012-10-22T20:05:42.691785+00:00</WhenStarted><DurationInSec>5400</DurationInSec><Number>0</Number></Movie><Sample><Name>NG-6380_10kb_0.5nM_Mag</Name><PlateId>ID1216_221012</PlateId><WellName>B01</WellName><Concentration>0</Concentration><SampleReuseEnabled>false</SampleReuseEnabled><UseCount>1</UseCount></Sample><InstrumentId>1</InstrumentId><InstrumentName>42149</InstrumentName><CollectionProtocol>MagBead Standard Seq v1</CollectionProtocol><CollectionNumber>3</CollectionNumber><CellIndex>6</CellIndex><SetNumber>1</SetNumber><EightPac><PartNumber>0015</PartNumber><LotNumber>230357</LotNumber><Barcode>10039644255000000152303571110121</Barcode><ExpirationDate>2012-11-10</ExpirationDate></EightPac><TemplatePrep><Name>DNA Template Prep Kit 2.0 (250bp - 3Kb)</Name><PartNumber>001540726</PartNumber><LotNumber>110116</LotNumber><Barcode>110116001540726071812</Barcode><ExpirationDate>2012-07-18</ExpirationDate><AdapterSequence>ATCTCTCTCttttcctcctcctccgttgttgttgttGAGAGAGAT</AdapterSequence><InsertSize>10000</InsertSize></TemplatePrep><BindingKit><Name>DNA/Polymerase Binding Kit 2.0 (24 Rxn)</Name><PartNumber>001672551</PartNumber><LotNumber>120126</LotNumber><Barcode>120126001672551071712</Barcode><ExpirationDate>2012-07-17</ExpirationDate><Control>Strobe_v1</Control><IsControlUsed>false</IsControlUsed></BindingKit><SequencingKit><Name>ReagentPlate0</Name><PartNumber>001558034</PartNumber><LotNumber>001320</LotNumber><Barcode>001320410001558034032213</Barcode><ExpirationDate>2013-03-22</ExpirationDate><Protocol>MagBeadReagentMixingProtocol_DWP</Protocol></SequencingKit><ReagentTube0><Name>ReagentTube0-0</Name><PartNumber>001028310</PartNumber><LotNumber>001224</LotNumber><Barcode>001224328001028310060914</Barcode><ExpirationDate>2014-06-09</ExpirationDate></ReagentTube0><Primary><Protocol>BasecallerV1</Protocol><ConfigFileName>1-3-0_Standard_C2.xml</ConfigFileName><ResultsFolder>Analysis_Results</ResultsFolder><CollectionPathUri>srs://192.168.30.15/mnt/san/pacbio/rawdata/active//ID1216_221012_186/B01_1/</CollectionPathUri><CollectionFileCopy>Fasta</CollectionFileCopy><CollectionFileCopy>Fastq</CollectionFileCopy></Primary><Secondary><ProtocolName /><CellCountInJob>0</CellCountInJob></Secondary><Custom><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_1" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_2" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_3" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_4" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_5" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_6" /></Custom></Metadata>
m121022_195644_42149_c100396442550000001523035711101216_s1_p0.metadata.xml (END)

the contents of the Analysis_Results directory are as follows

ls ../B01_1/Analysis_Results [ 8:50AM]
m121022_195644_42149_c100396442550000001523035711101216_s1_p0.bas.h5

[GenoMax]

@rhall: How is coldturkey running SMRTportal .. if not as user smrtanalysis?

@coldturkey: Are you the administrator for SMRTportal? Are you running this on a virtual server?

[coldturkey]

@ Genommax: so I am running it an administrator locally on a linux machine while I test it. Is it likely that I have set it up wrong. Could that be the problem?

@ rhall: the user running tomcat has full access to the files

[rhall]

@GenoMax: It is possible to run SMRT Portal as any user, but the install script should make sure everything under SEYMOUR_HOME belongs to that user.

The data all looks fine, we should work on getting the test data imported. Check the user for tomcat and that all files under SEYMOUR_HOME belong to that user.

[GenoMax]

@coldturkey: What is $SEYMOUR_HOME set to?

What linux version is this? SMRTportal is only supported on CentOS (5.4 or better but not 6.x) and Ubuntu (10.10).

[coldturkey]

All files under SEYMOUR_HOME belong to the tomcat user

for example

dr-xr-x--- 8 brian brian 4096 Jan 28 22:36 analysis
dr-xr-x--- 9 brian brian 4096 Jan 28 22:36 common
dr-xr-x--- 10 brian brian 4096 Jan 28 22:47 doc
dr-xr-x--- 3 brian brian 4096 Feb 15 15:44 etc
dr-xr-x--- 4 brian brian 4096 Jan 28 22:36 licenses
lrwxrwxrwx 1 brian brian 23 Jan 28 22:36 postinstall -> etc/scripts/postinstall
dr-xr-x--- 6 brian brian 4096 Jan 28 22:41 redist

[rhall]

Try giving the database a kick:

cd $SEYMOUR_HOME/etc/scripts && bash dbdata.sh

SMRT Portal will run without the database, but I would expect it to complain before cell input.

[coldturkey]

$SEYMOUR_HOME is set to /opt/smartanalysis

I am running Ubuntu 12.04