Using t1.micro instance can't import data into smrtPORTAL

[Fad2012]

Hello Guys,

I have received the subreads from the sequencing provider, and now trying to generate the consensus reads. I am using amazon web service to do so, i am using the free instance t1.micro and i followed the instructions in this link to set up the instance.

It is working now, and i can connect to smrtPORTAL and upload my data. However, when i try to import my data into smrtPORTAL, the software (SMRT Portal Interface) can't find any documents when i scan the folder "common/inputs_dropbox", it instead returns this messga "Scan completed. No new SMRT Cell data found". I tried to add the folder "common/test", as testing strategy that i got form this link, it worked, i was able to import what in "common/test".

I am not sure what is the reason, my data is composed of bas.h5 file,
three bax.h5 files, and metadata.xml ( in total around 6.5 GB). I am the administrator of the computer, and the Instance on Amazon, thus i have the permission to read, write and excute on all files ( so i would exclude the permission as a reason) Would it be that the instance can't work with data in such size. Do i have to buy an instance with better specifications. Or could it be the ubunto version:

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04.4 LTS
Release: 10.04
Codename: lucid

However, i am not sure that i have a control over the ubuntu version, as i am dealing with the image of smrtPORTAL which was set by PacBio and made available on Amazon Web Service within Ubuntu atmosphere, so in theory it should come with compatible version of Ubuntu ( Am i right by saying this??)

I would appreciate your help to sort this issue out!
Best regards

[rhall]

Hi,
To check off the first possible issue, are the files in the same directory structure as they where from the machine? https://github.com/PacificBioscience...new-SMRT-Cells

[AndyAqualab]

Similar problem using EC2 instance on amazon

Imported data onto the instance from S3 bucket using co from AWS CLI. Checked directory structure and permissions using ls and ls -l. All good. two folders (one per cell, E01_1 and G01_1) with permissions set recursively to 755. From smrtportal Add path /opt/smrtanalysis/userdata/inputs_dropbox the scan - no new smrt cells detected...

Andy

To original poster - have you tried ls -l to list permissions for each of the files in your directories?

Edit: Not sure whether it is better to start a new thread or hijack this one as problems are similar. I can't get the smrt portal on my amazon EC2 instance recognise smart cell data. Adding the common/test path works fine - 3 cells detected so the portal works. I think permissions and directory structure are good. As upload was difficult between Aus and US, I kept the transfer as small as possibly by only including the metadata xml along with the analysis results folder with only the bas.h5 and bax.h5 files (ie no fasta, fastq or csvs), but in the original cell folders (E01_1 and G01_1). Path added to smrt portal /opt/smrtanalysis/userdata/inputs_dropbox. Paths, structure and permissions below for 2 cells of data. smrt portal can't see either. Reboot tried and still not detecting:
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/E01_1$ ls -l
total 8
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 05:52 Analysis_results
-rwxr-xr-x 1 ubuntu ubuntu 3971 Jan 11 23:44 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.metadata.xml
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/E01_1$ cd Analysis_results/
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/E01_1/Analysis_results$ ls -l
total 4675868
-rwxr-xr-x 1 ubuntu ubuntu 1471197131 Jan 11 23:46 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.1.bax.h5
-rwxr-xr-x 1 ubuntu ubuntu 1608949249 Jan 11 23:51 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.2.bax.h5
-rwxr-xr-x 1 ubuntu ubuntu 1701930804 Jan 11 23:59 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.3.bax.h5
-rwxr-xr-x 1 ubuntu ubuntu 1318480 Jan 12 00:15 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.bas.h5
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/E01_1/Analysis_results$ pwd
/opt/smrtanalysis/userdata/inputs_dropbox/E01_1/Analysis_results
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/E01_1/Analysis_results$
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/E01_1/Analysis_results$ cd /opt/smrtanalysis/userdata/inputs_dropbox/G01_1
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/G01_1$ ls -l
total 690100
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 04:05 Analysis_results
-rwxr-xr-x 1 ubuntu ubuntu 8773 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.1.xfer.xml
-rwxr-xr-x 1 ubuntu ubuntu 5193 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.2.xfer.xml
-rwxr-xr-x 1 ubuntu ubuntu 5193 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.3.xfer.xml
-rwxr-xr-x 1 ubuntu ubuntu 705926294 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.mcd.h5
-rwxr-xr-x 1 ubuntu ubuntu 3986 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.metadata.xml
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/G01_1$ cd Analysis_results/
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/G01_1/Analysis_results$ ls -l
total 14747328
-rwxr-xr-x 1 ubuntu ubuntu 5092226724 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.1.bax.h5
-rwxr-xr-x 1 ubuntu ubuntu 4890734886 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.2.bax.h5
-rwxr-xr-x 1 ubuntu ubuntu 5015369751 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.3.bax.h5
-rwxr-xr-x 1 ubuntu ubuntu 1318480 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.bas.h5
-rwxr-xr-x 1 ubuntu ubuntu 86686931 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.sts.csv
-rwxr-xr-x 1 ubuntu ubuntu 124548 Jan 12 02:40 m141203_082318_42149_c100739672550000001823153306051536_s1_p0.sts.xml
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/G01_1/Analysis_results$ pwd
/opt/smrtanalysis/userdata/inputs_dropbox/G01_1/Analysis_results
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/userdata/inputs_dropbox/G01_1/Analysis_results$

[Fad2012]

Thanks Rhall,

It worked! I think now i am facing the problem of memory, the job on smartPORTAL stopped with the message: "Cannot allocate memory".

I think larger instance will do, is that right?

Many thanks

[Fad2012]

Hi AndyAqualab,

Make sure to have the Analysis_results contains the .h5 files, and the metadata.xml out of the folder, not in it.

[rhall]

Fad2012,
The size of instance will depend on exactly what you want to do, but the memory limitation on the t1.micro makes it almost useless for any processing. For very small analysis jobs, filtering a single cell, t2.medium would probably be sufficient. For larger analysis, resequencing, denovo assembly, base modification you will have to use something with more memory and cores.

[AndyAqualab]

Thanks Fad2012, I have the directory structure exactly like that. My feeling at present is there may be a version issue with ubuntu in my instance. I needed to migrate my data from an S3 bucket in Sydney as routing it from desktop using scp was much too slow (<30 kb per second). So I had to install AWS CLI on the EC2 instance and that required an upgrade of the ubuntu version running on EC2. I then use aws cp to copy my folders from S3 Sydney direct onto the EC2 instance via the AWS connection, which is very much faster. It worked, my files are there with original directory structure as far as i can see. But, smrt portal can't find them. Navigating through the folders on the EC2 instance everything is where it should be and the permissions look good. I also followed the filepath for the test data (which uploaded successfully) and my data seem to be correctly replicated in that filepath, so it all seems to be good (see below). BUt it stil won't import through smrtportal. Tried rebooting. tried stopping and then restarting. No dice. Any help from anywhere greatly appreciated.
here are paths with one of the smrt cells listed with permissions
ubuntu@ip-172-31-7-81:/opt$ cd smrtanalysis/install/smrtanalysis_2.3.0.140936/
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936$ ls
admin analysis bin common doc etc installerdeps licenses miscdeps misclibs postinstall redist smrtcmds support
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936$ cd common
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common$ ls -l
total 28
dr-xr-x--- 2 ubuntu ubuntu 4096 Oct 3 16:55 etc
lrwxrwxrwx 1 ubuntu ubuntu 23 Oct 3 16:49 inputs_dropbox -> userdata/inputs_dropbox
lrwxrwxrwx 1 ubuntu ubuntu 13 Oct 3 16:49 jobs -> userdata/jobs
lrwxrwxrwx 1 ubuntu ubuntu 21 Oct 3 16:49 jobs_archive -> userdata/jobs_archive
lrwxrwxrwx 1 ubuntu ubuntu 21 Oct 3 16:49 jobs_dropbox -> userdata/jobs_dropbox
dr-xr-x--- 4 ubuntu ubuntu 4096 Oct 1 05:36 lib
drwxr-x--- 4 ubuntu ubuntu 4096 Oct 3 17:02 log
drwxr-x--- 12 ubuntu ubuntu 4096 Oct 1 05:36 protocols
lrwxrwxrwx 1 ubuntu ubuntu 19 Oct 3 16:49 references -> userdata/references
lrwxrwxrwx 1 ubuntu ubuntu 27 Oct 3 16:49 references_dropbox -> userdata/references_dropbox
dr-xr-x--- 6 ubuntu ubuntu 4096 Oct 1 05:08 test
lrwxrwxrwx 1 ubuntu ubuntu 17 Oct 3 16:54 userdata -> ../../../userdata
dr-xr-x--- 8 ubuntu ubuntu 4096 Oct 1 05:24 userdata.d
dr-xr-x--- 3 ubuntu ubuntu 4096 Oct 1 05:24 www
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common$ cd inputs_dropbox
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common/inputs_dropbox$ ls -l
total 8
dr-xr-x--- 3 ubuntu ubuntu 4096 Jan 13 05:51 E01_1
dr-xr-x--- 3 ubuntu ubuntu 4096 Jan 13 04:05 G01_1
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common/inputs_dropbox$ cd E01_1
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common/inputs_dropbox/E01_1$ ls -l
total 8
dr-xr-x--- 2 ubuntu ubuntu 4096 Jan 13 05:52 Analysis_results
-r-xr-xr-- 1 ubuntu ubuntu 3971 Jan 11 23:44 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.metadata.xml
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common/inputs_dropbox/E01_1$ cd Analysis_results/
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common/inputs_dropbox/E01_1/Analysis_results$ ls -l
total 4675868
-r-xr-xr-- 1 ubuntu ubuntu 1471197131 Jan 11 23:46 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.1.bax.h5
-r-xr-xr-- 1 ubuntu ubuntu 1608949249 Jan 11 23:51 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.2.bax.h5
-r-xr-xr-- 1 ubuntu ubuntu 1701930804 Jan 11 23:59 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.3.bax.h5
-r-xr-xr-- 1 ubuntu ubuntu 1318480 Jan 12 00:15 m140717_033842_42149_c100664522550000001823130112201473_s1_p0.bas.h5
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common/inputs_dropbox/E01_1/Analysis_results$

Here is the metadata for that file:
<?xml version="1.0" encoding="utf-8"?><Metadata xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://pacificbiosciences.com/PAP/Metadata.xsd"><InstCtrlVer>2.1.0.0.127824</InstCtrlVer><SigProcVer>[email protected]:8082, SwVer=2100.127824, HwVer=1.0</SigProcVer><Run><RunId>r000334_42149_140716</RunId><Name>ID2616_16072014_TPa</Name><WhenCreated>2014-07-16T10:06:27</WhenCreated><WhenStarted>2014-07-16T08:55:04</WhenStarted></Run><Movie><WhenStarted>2014-07-17T03:50:29.507379+00:00</WhenStarted><DurationInSec>10800</DurationInSec><Number>0</Number></Movie><Sample><Name>46642_NG-7489_2</Name><PlateId>ID2616_16072014_TPa</PlateId><WellName>E01</WellName><Concentration>0</Concentration><SampleReuseEnabled>false</SampleReuseEnabled><StageHotstartEnabled>true</StageHotstartEnabled><SizeSelectionEnabled>true</SizeSelectionEnabled><UseCount>1</UseCount></Sample><InstrumentId>1</InstrumentId><InstrumentName>42149</InstrumentName><CollectionProtocol>MagBead Standard Seq v2</CollectionProtocol><CollectionNumber>6</CollectionNumber><CellIndex>3</CellIndex><SetNumber>1</SetNumber><EightPac><PartNumber>0018</PartNumber><LotNumber>231301</LotNumber><Barcode>10066452255000000182313011220147</Barcode><ExpirationDate>2014-12-20</ExpirationDate></EightPac><TemplatePrep><Name>DNA Template Prep Kit 2.0 (3Kb - 10Kb)</Name><PartNumber>001540835</PartNumber><LotNumber>002550</LotNumber><Barcode>002550001540835032514</Barcode><ExpirationDate>2014-03-25</ExpirationDate><AdapterSequence>ATCTCTCTCttttcctcctcctccgttgttgttgttGAGAGAGAT</AdapterSequence><InsertSize>20000</InsertSize></TemplatePrep><BindingKit><Name>DNA/Polymerase Binding Kit P4</Name><PartNumber>100236500</PartNumber><LotNumber>002567</LotNumber><Barcode>002567100236500030114</Barcode><ExpirationDate>2014-03-01</ExpirationDate><Control /><IsControlUsed>false</IsControlUsed></BindingKit><SequencingKit><Name>ReagentPlate0</Name><PartNumber>001558034</PartNumber><LotNumber>003751</LotNumber><Barcode>003751318001558034102914</Barcode><ExpirationDate>2014-10-29</ExpirationDate><Protocol>MagBeadReagentMixingProtocol_DWP</Protocol></SequencingKit><ReagentTube0><Name>ReagentTube0-0</Name><PartNumber>001028310</PartNumber><LotNumber>003672</LotNumber><Barcode>003672986001028310032316</Barcode><ExpirationDate>2016-03-23</ExpirationDate></ReagentTube0><ReagentTube1><Name>ReagentTube0-1</Name><PartNumber>100192000</PartNumber><LotNumber>003558</LotNumber><Barcode>003558857100192000103118</Barcode><ExpirationDate>2018-10-31</ExpirationDate></ReagentTube1><Primary><Protocol>BasecallerV1</Protocol><ConfigFileName>2-0-0_P4-C2.xml</ConfigFileName><ResultsFolder>Analysis_Results</ResultsFolder><CollectionPathUri>srs://smrt/mnt/nsa1/bioit/rawdata/active/pacbio//ID2616_16072014_TPa_334/E01_1/</CollectionPathUri><CollectionFileCopy>Fasta</CollectionFileCopy><CollectionFileCopy>Fastq</CollectionFileCopy></Primary><Secondary><ProtocolName /><CellCountInJob>0</CellCountInJob></Secondary><Custom><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_1" label="User Defined Field 1" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_2" label="User Defined Field 2" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_3" label="User Defined Field 3" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_4" label="User Defined Field 4" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_5" label="User Defined Field 5" /><KeyValue key="svc:/CentralDataSvc/#Display.Sample_Metadata.User_Defined_Field_6" label="User Defined Field 6" /></Custom><ExpirationData><EightPacPastExpiration>0</EightPacPastExpiration><ReagentKitPastExpiration>0</ReagentKitPastExpiration><ReagentTube0PastExpiration>0</ReagentTube0PastExpiration><ReagentTube1PastExpiration>0</ReagentTube1PastExpiration></ExpirationData></Metadata>

[rhall]

Could you check the SMRT portal log to see the specific error when you try to import the cells /opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common/log/smrtportal/smrtportal.0.log

Another thing to try is move the cell directories to a completely different location, that does not include soft links, add the directory as a search location and scan it. It's possible that the directory soft links are confusing things.

[AndyAqualab]

logfile tail

Firstly thank you so much for replying. I tailed the log file and scanned for cells again. The output is below.
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common/log/smrtportal$ tail -f smrtportal.0.log
Jan 14, 2015 10:28:31 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:37:46 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:37:46 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:37:46 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:37:47 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:04 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:04 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:08 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:08 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:08 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper collectMetadataXml
INFO: Scan path common/inputs_dropbox
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper collectMetadataXml
INFO: Scan path /opt/smrtanalysis/userdata/inputs_dropbox/E01_1
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper collectMetadataXml
INFO: Scan path /opt/smrtanalysis/userdata/inputs_dropbox/G01_1
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper getFirstMetadataFile
INFO: Parse movie metadata collection path: /opt/smrtanalysis/userdata/inputs_dropbox/E01_1
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputHelper loadMetadata
WARNING: Error parsing input from outputFilePath: /opt/smrtanalysis/userdata/inputs_dropbox/E01_1
java.lang.NullPointerException
at com.pacbio.secondary.smrtportal.util.FileUtils.listFiles(FileUtils.java:58)
at com.pacbio.secondary.smrtportal.common.helpers.SmrtpipeHelper.listFiles(SmrtpipeHelper.java:43)
at com.pacbio.secondary.smrtportal.common.helpers.AbstractSmrtpipeHelper.validateInput(AbstractSmrtpipeHelper.java:881)
at com.pacbio.secondary.smrtportal.common.helpers.InputHelper.loadMetadata(InputHelper.java:168)
at com.pacbio.secondary.smrtportal.common.helpers.InputHelper.scan(InputHelper.java:416)
at com.pacbio.secondary.smrtportal.ws.resources.InputsResource.scan(InputsResource.java:448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:165)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:833)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:806)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:767)
at com.pacbio.secondary.smrtportal.common.rest.Application.doFilter(Application.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:300)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper getFirstMetadataFile
INFO: Parse movie metadata collection path: /opt/smrtanalysis/userdata/inputs_dropbox/G01_1
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputHelper loadMetadata
WARNING: Error parsing input from outputFilePath: /opt/smrtanalysis/userdata/inputs_dropbox/G01_1
java.lang.NullPointerException
at com.pacbio.secondary.smrtportal.util.FileUtils.listFiles(FileUtils.java:58)
at com.pacbio.secondary.smrtportal.common.helpers.SmrtpipeHelper.listFiles(SmrtpipeHelper.java:43)
at com.pacbio.secondary.smrtportal.common.helpers.AbstractSmrtpipeHelper.validateInput(AbstractSmrtpipeHelper.java:881)
at com.pacbio.secondary.smrtportal.common.helpers.InputHelper.loadMetadata(InputHelper.java:168)
at com.pacbio.secondary.smrtportal.common.helpers.InputHelper.scan(InputHelper.java:416)
at com.pacbio.secondary.smrtportal.ws.resources.InputsResource.scan(InputsResource.java:448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:165)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:833)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:806)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:767)
at com.pacbio.secondary.smrtportal.common.rest.Application.doFilter(Application.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:300)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

[AndyAqualab]

Thanks for replying. Much appreciated. Apologies if this reply appears twice, first one never appeared. Logfile tail below. Included the few lines before and after the error. I also moved the original folders to a higher level directory, which was interesting as when I scanned that through portal it found a whole bunch of cells under commom/test/jobs/scerevisiae which threw the same error in the log.
ubuntu@ip-172-31-7-81:/opt/smrtanalysis/install/smrtanalysis_2.3.0.140936/common/log/smrtportal$ tail -f smrtportal.0.log
Jan 14, 2015 10:28:31 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:37:46 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:37:46 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:37:46 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:37:47 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:04 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:04 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:08 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:08 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:08 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.common.core.AnalysisContext reload
INFO: Reload analysis config
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper collectMetadataXml
INFO: Scan path common/inputs_dropbox
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper collectMetadataXml
INFO: Scan path /opt/smrtanalysis/userdata/inputs_dropbox/E01_1
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper collectMetadataXml
INFO: Scan path /opt/smrtanalysis/userdata/inputs_dropbox/G01_1
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper getFirstMetadataFile
INFO: Parse movie metadata collection path: /opt/smrtanalysis/userdata/inputs_dropbox/E01_1
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputHelper loadMetadata
WARNING: Error parsing input from outputFilePath: /opt/smrtanalysis/userdata/inputs_dropbox/E01_1
java.lang.NullPointerException
at com.pacbio.secondary.smrtportal.util.FileUtils.listFiles(FileUtils.java:58)
at com.pacbio.secondary.smrtportal.common.helpers.SmrtpipeHelper.listFiles(SmrtpipeHelper.java:43)
at com.pacbio.secondary.smrtportal.common.helpers.AbstractSmrtpipeHelper.validateInput(AbstractSmrtpipeHelper.java:881)
at com.pacbio.secondary.smrtportal.common.helpers.InputHelper.loadMetadata(InputHelper.java:168)
at com.pacbio.secondary.smrtportal.common.helpers.InputHelper.scan(InputHelper.java:416)
at com.pacbio.secondary.smrtportal.ws.resources.InputsResource.scan(InputsResource.java:448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:165)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:833)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:806)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:767)
at com.pacbio.secondary.smrtportal.common.rest.Application.doFilter(Application.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:300)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputDataHelper getFirstMetadataFile
INFO: Parse movie metadata collection path: /opt/smrtanalysis/userdata/inputs_dropbox/G01_1
Jan 14, 2015 10:38:17 PM com.pacbio.secondary.smrtportal.common.helpers.InputHelper loadMetadata
WARNING: Error parsing input from outputFilePath: /opt/smrtanalysis/userdata/inputs_dropbox/G01_1
java.lang.NullPointerException
at com.pacbio.secondary.smrtportal.util.FileUtils.listFiles(FileUtils.java:58)
at com.pacbio.secondary.smrtportal.common.helpers.SmrtpipeHelper.listFiles(SmrtpipeHelper.java:43)
at com.pacbio.secondary.smrtportal.common.helpers.AbstractSmrtpipeHelper.validateInput(AbstractSmrtpipeHelper.java:881)
at com.pacbio.secondary.smrtportal.common.helpers.InputHelper.loadMetadata(InputHelper.java:168)
at com.pacbio.secondary.smrtportal.common.helpers.InputHelper.scan(InputHelper.java:416)
at com.pacbio.secondary.smrtportal.ws.resources.InputsResource.scan(InputsResource.java:448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:165)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:833)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:806)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:767)
at com.pacbio.secondary.smrtportal.common.rest.Application.doFilter(Application.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:300)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

[gconcepcion]

Thanks for replying. Much appreciated. Apologies if this reply appears twice, first one never appeared. Logfile tail below. Included the few lines before and after the error. I also moved the original folders to a higher level directory, which was interesting as when I scanned that through portal it found a whole bunch of cells under commom/test/jobs/scerevisiae which threw the same error in the log.
...
<snip>
WARNING: Error parsing input from outputFilePath: /opt/smrtanalysis/userdata/inputs_dropbox/E01_1
java.lang.NullPointerException
...
<snip>

WARNING: Error parsing input from outputFilePath: /opt/smrtanalysis/userdata/inputs_dropbox/G01_1
java.lang.NullPointerException

This points to either malformed *.metadata.xml or corrupt bax.h5 files. Did you verify checksums after you downloaded from the provider?

What does the *.metadata.xml look like? Is it possible extra characters have snuck in?

[gconcepcion]

Sorry, just saw your metadata, it *looks* fine as long as there are no hidden characters.

My bet is that all of the copying you described in post #7 led to corrupt bax.h5 files.

[AndyAqualab]

Thanks all for your replies! It was a file problem. I deleted the previously transferred folders and uploaded to S3 then copied over the original compressed downloads from the sequencing provider using AWS CLI then unpacked the tarballs in situ on EC2. The resulting folders were detected by smrtportal and the assembly is running. Beers are on me if any of you make it to Bris

[gconcepcion]

Thanks all for your replies! It was a file problem. I deleted the previously transferred folders and uploaded to S3 then copied over the original compressed downloads from the sequencing provider using AWS CLI then unpacked the tarballs in situ on EC2. The resulting folders were detected by smrtportal and the assembly is running. Beers are on me if any of you make it to Bris

Ahhh glad it worked out and my intuition was correct.

Welcome to the world of long reads