[Tomi]

Hello,

I have a problem with the new update that I have no acess to the AMI, and the old one is also not acessible any more!

I would need help

Greetings,
Tomi

[pcs_murali]

Hi Tomi,

I am sorry for the late response. Please check now. It has been fixed last month.

Please let me know if you have any problem

Thanks
Chandra

[YW0712]

Hello,

Thanks for sharing all your helpful comments here. I'm new to PathSeq and am trying to finish the installation. Unfortunately, I'm getting this error message:

Created hadoop-0.19.0-x86
ec2-bundle-vol interrupted.
_64.part.81
Created hadoop-0.19.0-x86_64.part.82
Created hadoop-0.19.0-x86_64.part.83
Created hadoop-0.19.0-x86_64.part.84
Created hadoop-0.19.0-x86_64.part.85
Created hadoop-0.19.0-x86_64.part.86
Generating digests for each part...
Digests generated.
--manifest has invalid value '/mnt/hadoop-0.19.0-x86_64.manifest.xml': File does not exist or is not a file.
Try 'ec2-upload-bundle --help'
Done
Client.InvalidManifest: HTTP 403 (Forbidden) response for URL http://s3.amazonaws.com:80/ami-tmp/h....manifest.xml: check your S3 ACLs are correct.
Terminate with: ec2-terminate-instances i-0b0c2868
INSTANCE i-0b0c2868 running shutting-down
Creation completed

If you'd please share any insight, I'd really appreciate it! I'd be happy to provide any more info.

Thanks in advance!

[pcs_murali]

Hi,

Thanks for your interest.

did you download the ec2-api-tools?

Also, please recheck all the steps in installation manual.

Then, Please re-run the script to create the own AMI.

Thanks
Chandra

Hello,

Thanks for sharing all your helpful comments here. I'm new to PathSeq and am trying to finish the installation. Unfortunately, I'm getting this error message:

Created hadoop-0.19.0-x86
ec2-bundle-vol interrupted.
_64.part.81
Created hadoop-0.19.0-x86_64.part.82
Created hadoop-0.19.0-x86_64.part.83
Created hadoop-0.19.0-x86_64.part.84
Created hadoop-0.19.0-x86_64.part.85
Created hadoop-0.19.0-x86_64.part.86
Generating digests for each part...
Digests generated.
--manifest has invalid value '/mnt/hadoop-0.19.0-x86_64.manifest.xml': File does not exist or is not a file.
Try 'ec2-upload-bundle --help'
Done
Client.InvalidManifest: HTTP 403 (Forbidden) response for URL http://s3.amazonaws.com:80/ami-tmp/h....manifest.xml: check your S3 ACLs are correct.
Terminate with: ec2-terminate-instances i-0b0c2868
INSTANCE i-0b0c2868 running shutting-down
Creation completed

If you'd please share any insight, I'd really appreciate it! I'd be happy to provide any more info.

Thanks in advance!

[YW0712]

Hi Chandra,

Thanks for the prompt response. I did download .ec2-api-tools and have double-checked the steps and can't seem find what I'm missing. The first error messages I'm still getting are:

REP AVAILABLE
/xchip/pasteur/chandra/Amazon/hadoop-0.20.2/src/contrib/ec2/bin/test: No such file or directory
distrib.tar.Z: No such file or directory
install_rlib 100% 103 0.1KB/s 00:00
/bin/tar: /usr/local/RepeatMasker/repeatmaskerlibraries-20090604.tar.gz: Cannot open: No such file or directory
/bin/tar: Error is not recoverable: exiting now
/bin/tar: Child returned status 2
/bin/tar: Error exit delayed from previous errors
install_cross 100% 208 0.2KB/s 00:00
/bin/tar: /usr/local/RepeatMasker/crossmatch/distrib.tar.Z: Cannot open: No such file or directory
/bin/tar: Error is not recoverable: exiting now

I've double-checked the directories for RepeatMasker and Crossmatch specified in my cluster.config file (which is not under /usr/local/). Is there another place I should've changed the directories? Any help would be greatly appreciated!

[pcs_murali]

Hi,

Repeat masker libraries and Crossmatch are not packaged with Pathseq because it needs licenses from respective people. If you are academic institute / non-profit organization you may get it for free.

Even without these repeatmasker libraries and crossmatch the createownAMI should work?

I will try to simulate it at my computer and see what is happening.

Please check the account number and make sure the number is without "-".

Thanks
Chandra

Hi Chandra,

Thanks for the prompt response. I did download .ec2-api-tools and have double-checked the steps and can't seem find what I'm missing. The first error messages I'm still getting are:

REP AVAILABLE
/xchip/pasteur/chandra/Amazon/hadoop-0.20.2/src/contrib/ec2/bin/test: No such file or directory
distrib.tar.Z: No such file or directory
install_rlib 100% 103 0.1KB/s 00:00
/bin/tar: /usr/local/RepeatMasker/repeatmaskerlibraries-20090604.tar.gz: Cannot open: No such file or directory
/bin/tar: Error is not recoverable: exiting now
/bin/tar: Child returned status 2
/bin/tar: Error exit delayed from previous errors
install_cross 100% 208 0.2KB/s 00:00
/bin/tar: /usr/local/RepeatMasker/crossmatch/distrib.tar.Z: Cannot open: No such file or directory
/bin/tar: Error is not recoverable: exiting now

I've double-checked the directories for RepeatMasker and Crossmatch specified in my cluster.config file (which is not under /usr/local/). Is there another place I should've changed the directories? Any help would be greatly appreciated!

[YW0712]

Thanks Chandra.

I've downloaded both and have correctly linked to where they are saved in the cluster.config file. For some reason, the create_own_AIM execution is not looking in the correct directories for them.

Also, I'm assuming the two lines below are switched?

# Cross match location (sample: /local/directory/repeatmasker/repeatmaskerlibraries-20090604.tar.gz)
REPBASELOC=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Repbase location (sample: /local/directory/repeatmasker/distrib.tar.Z)
CROSSMATCHLOC=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Thanks again for the help!

[pcs_murali]

Did you replace the xxx with the correct location?

Thanks
Chandra

Thanks Chandra.

I've downloaded both and have correctly linked to where they are saved in the cluster.config file. For some reason, the create_own_AIM execution is not looking in the correct directories for them.

Also, I'm assuming the two lines below are switched?

# Cross match location (sample: /local/directory/repeatmasker/repeatmaskerlibraries-20090604.tar.gz)
REPBASELOC=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Repbase location (sample: /local/directory/repeatmasker/distrib.tar.Z)
CROSSMATCHLOC=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Thanks again for the help!

[YW0712]

Yes. Thanks - turns out I accidentally changed a couple lines in the create-h-image file. Now that's fixed. However, I'm still getting the error message at the very end:

---
Unable to read instance meta-data for product-codes
Creating bundle manifest...
ec2-bundle-vol complete.
ERROR: Error talking to S3: Server.AccessDenied(403): Access Denied
Done
Client.InvalidManifest: HTTP 403 (Forbidden) response for URL http://s3.amazonaws.com:80/ami-tmp/h....manifest.xml: check your S3 ACLs are correct.
Terminate with: ec2-terminate-instances i-1d2a147e
INSTANCE i-1d2a147e running shutting-down
Creation completed
---

Maybe there's something wrong with my S3 configuration?

Access Key: (correctly set)
Secret Key: (correctly set)
Encryption password: (correctly set)
Path to GPG program: /usr/local/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/Users/LOCAL/.s3cfg'


Thanks again for your time!

[YW0712]

Hi Chandra,

Just saw the solution in the FAQ section to my problem described above: "Please rename the "IMAGEBUCKET" in cluster.config file."

I've tried a couple different names that still didn't work. Is there any specifications that I should be aware of?

Thanks again, and sorry about so many questions!

[rts]

cluster.config IMAGEBUCKET variable

Hi All,
I am learning Amazon Cloud for the PathSeq pipeline.
From the installation manual and reading I have gotten fairly far.
But, I am struggling the the IMAGEBUCKET variable in cluster.config file.
I have a S3 bucket named redwoodpath and have an instance w/ an AMI-ID but I do not get how to give an S3 bucket an "ami-name" (where the hadoop AMI is stored) as shown in the installation manual.

# The Amazon S3 bucket where the Hadoop AMI is stored.(Edit)
# so you can store it in a bucket you own.
IMAGEBUCKET=ami-pathseqimageXXXXXXXXXXXXXX

I discussed this with another PathSeq user who is struggling and they too were unsure and said "I'm actually having the same problem with naming the IMAGEBUCKET. I just put random letters for the "X's" assuming that it just needs to be unique."

Any help would be much appreciated.

thanks,
rts

[pcs_murali]

Hi,

could you please download one more latest version and then re-configure the config scripts?

Then run the create_ownAMI.com script and let me know what happens.

Normally, i use ami-patsheqid1000 as my IMAGEBUCKET.

Please let me know what happens.

Thanks
Chandra

Hi Chandra,

Just saw the solution in the FAQ section to my problem described above: "Please rename the "IMAGEBUCKET" in cluster.config file."

I've tried a couple different names that still didn't work. Is there any specifications that I should be aware of?

Thanks again, and sorry about so many questions!

[pcs_murali]

Hi

Thank you very much for your interest.

You don't have the create the IMAGEBUCKET manually, the createOwn_AMI.com create the s3 bucket for you.

Please try to make the IMAGEBUCKET name as unique as possible.

Please let me know if you have trouble with the Pathseq installation / runs.

Thanks

Hi All,
I am learning Amazon Cloud for the PathSeq pipeline.
From the installation manual and reading I have gotten fairly far.
But, I am struggling the the IMAGEBUCKET variable in cluster.config file.
I have a S3 bucket named redwoodpath and have an instance w/ an AMI-ID but I do not get how to give an S3 bucket an "ami-name" (where the hadoop AMI is stored) as shown in the installation manual.

# The Amazon S3 bucket where the Hadoop AMI is stored.(Edit)
# so you can store it in a bucket you own.
IMAGEBUCKET=ami-pathseqimageXXXXXXXXXXXXXX

I discussed this with another PathSeq user who is struggling and they too were unsure and said "I'm actually having the same problem with naming the IMAGEBUCKET. I just put random letters for the "X's" assuming that it just needs to be unique."

Any help would be much appreciated.

thanks,
rts

[rts]

Almost there w/ installation create_ownAMI

Hi,
Getting close....!
But an error I can't seem to figure out....following create_ownAMI command.
Error out out context:
bash: /mnt/create-h-i-remote: Permission denied
Client.InvalidManifest: HTTP 404 (Not Found) response for URL http://s3.amazonaws.com:80/ami-paths....manifest.xml: check your manifest path is correct and in the correct region.

I have tried changing permission on /mnt (chmod 777), I have tried changing permissions on ./Build_OwnAMI/create-h-i-remote (chmod 600).
No files are in the /mnt folder.
I am trying to install w/o cross_match and repeatmasker for now

Here is the full output....

shared@redwoodpath:~/PathSeq/Pathseq_cloud_V5.2$ ./create_ownAMI.com
Creation of your Amazon Machine Image
/home/shared/PathSeq/Pathseq_cloud_V5.2/Build_OwnAMI
/home/shared/PathSeq/Pathseq_cloud_V5.2/Build_OwnAMI
ami-5fb67036
Starting a AMI with ID ami-5fb67036.

Instance is i-b3f0f9d0.
Polling server status (ec2-describe-instances i-b3f0f9d0)
...........................The server is available at ec2-204-236-213-0.compute-1.amazonaws.com.
Warning: Permanently added 'ec2-204-236-213-0.compute-1.amazonaws.com,204.236.213.0' (RSA) to the list of known hosts.
Copying scripts.
cluster.config 100% 1979 1.9KB/s 00:00
env.sh 100% 3983 3.9KB/s 00:00
create-h-i-remote 100% 3531 3.5KB/s 00:00
ec2-run-user-data 100% 1763 1.7KB/s 00:00
N
pk-HURMYZGNG74DWNEPWDVDFAJTWSF7O575.pem 100% 924 0.9KB/s 00:00
cert-HURMYZGNG74DWNEPWDVDFAJTWSF7O575.pem 100% 916 0.9KB/s 00:00
bash: /mnt/create-h-i-remote: Permission denied
Client.InvalidManifest: HTTP 404 (Not Found) response for URL http://s3.amazonaws.com:80/ami-paths....manifest.xml: check your manifest path is correct and in the correct region.
Terminate with: ec2-terminate-instances i-b3f0f9d0
INSTANCE i-b3f0f9d0 running shutting-down
Creation completed
rm: cannot remove `install_cross': No such file or directory
rm: cannot remove `install_rlib': No such file or directory

when I try w/ sudo...i get error of EC2_HOME not set in ec2-run-instances.

shared@redwoodpath:~/PathSeq/Pathseq_cloud_V5.2$ sudo ./create_ownAMI.comCreation of your Amazon Machine Image
/home/shared/PathSeq/Pathseq_cloud_V5.2/Build_OwnAMI
/home/shared/PathSeq/Pathseq_cloud_V5.2/Build_OwnAMI
ami-5fb67036
Starting a AMI with ID ami-5fb67036.

/home/shared/ec2-api-tools-1.5.0.0/bin/ec2-run-instances: line 9: EC2_HOME: EC2_HOME is not set
Instance is .
Polling server status (ec2-describe-instances )
./home/shared/ec2-api-tools-1.5.0.0/bin/ec2-describe-instances: line 9: EC2_HOME: EC2_HOME is not set
./home/shared/ec2-api-tools-1.5.0.0/bin/ec2-describe-instances: line 9: EC2_HOME: EC2_HOME is not set
^X./home/shared/ec2-api-tools-1.5.0.0/bin/ec2-describe-instances: line 9: EC2_HOME: EC2_HOME is not set

I followed instructions for my.bashrc and have tried setting EC2_HOME variable manually too w/ export EC2_HOME=/home/shared/ec2-api-tools-1.5.0.0 prior to running sudo create_ownAMI.com

Thanks for any guidance you may have.
rts

[pcs_murali]

Hi,

I am trying to simulate the problem in my machine. I didn't see any problem.

Could you please recheck whether you followed all the points listed in Installation manual?

If you changed any of the scripts, please download the Pathseq once again. Then, run it without changing the scripts.

/mnt is in the cloud not in your local machine. One thing i will suggest is, when you download the Pathseq and extract..please give permission for R+W for all scripts and subfolders in the directory of Pathseq.

Last part of log printed out:
Uploaded hadoop-0.19.0-x86_64.part.82
Uploaded hadoop-0.19.0-x86_64.part.83
Uploaded hadoop-0.19.0-x86_64.part.84
Uploaded hadoop-0.19.0-x86_64.part.85
Uploaded hadoop-0.19.0-x86_64.part.86
Uploading manifest ...
Uploaded manifest.
Bundle upload completed.
Done
IMAGE ami-298c4640
Terminate with: ec2-terminate-instances i-35737856
INSTANCE i-35737856 running shutting-down
Creation completed
rm: cannot remove `install_cross': No such file or directory
rm: cannot remove `install_rlib': No such file or directory

Thanks
Chandra

Hi,
Getting close....!
But an error I can't seem to figure out....following create_ownAMI command.
Error out out context:
bash: /mnt/create-h-i-remote: Permission denied
Client.InvalidManifest: HTTP 404 (Not Found) response for URL http://s3.amazonaws.com:80/ami-paths....manifest.xml: check your manifest path is correct and in the correct region.

I have tried changing permission on /mnt (chmod 777), I have tried changing permissions on ./Build_OwnAMI/create-h-i-remote (chmod 600).
No files are in the /mnt folder.
I am trying to install w/o cross_match and repeatmasker for now

Here is the full output....

shared@redwoodpath:~/PathSeq/Pathseq_cloud_V5.2$ ./create_ownAMI.com
Creation of your Amazon Machine Image
/home/shared/PathSeq/Pathseq_cloud_V5.2/Build_OwnAMI
/home/shared/PathSeq/Pathseq_cloud_V5.2/Build_OwnAMI
ami-5fb67036
Starting a AMI with ID ami-5fb67036.

Instance is i-b3f0f9d0.
Polling server status (ec2-describe-instances i-b3f0f9d0)
...........................The server is available at ec2-204-236-213-0.compute-1.amazonaws.com.
Warning: Permanently added 'ec2-204-236-213-0.compute-1.amazonaws.com,204.236.213.0' (RSA) to the list of known hosts.
Copying scripts.
cluster.config 100% 1979 1.9KB/s 00:00
env.sh 100% 3983 3.9KB/s 00:00
create-h-i-remote 100% 3531 3.5KB/s 00:00
ec2-run-user-data 100% 1763 1.7KB/s 00:00
N
pk-HURMYZGNG74DWNEPWDVDFAJTWSF7O575.pem 100% 924 0.9KB/s 00:00
cert-HURMYZGNG74DWNEPWDVDFAJTWSF7O575.pem 100% 916 0.9KB/s 00:00
bash: /mnt/create-h-i-remote: Permission denied
Client.InvalidManifest: HTTP 404 (Not Found) response for URL http://s3.amazonaws.com:80/ami-paths....manifest.xml: check your manifest path is correct and in the correct region.
Terminate with: ec2-terminate-instances i-b3f0f9d0
INSTANCE i-b3f0f9d0 running shutting-down
Creation completed
rm: cannot remove `install_cross': No such file or directory
rm: cannot remove `install_rlib': No such file or directory

when I try w/ sudo...i get error of EC2_HOME not set in ec2-run-instances.

shared@redwoodpath:~/PathSeq/Pathseq_cloud_V5.2$ sudo ./create_ownAMI.comCreation of your Amazon Machine Image
/home/shared/PathSeq/Pathseq_cloud_V5.2/Build_OwnAMI
/home/shared/PathSeq/Pathseq_cloud_V5.2/Build_OwnAMI
ami-5fb67036
Starting a AMI with ID ami-5fb67036.

/home/shared/ec2-api-tools-1.5.0.0/bin/ec2-run-instances: line 9: EC2_HOME: EC2_HOME is not set
Instance is .
Polling server status (ec2-describe-instances )
./home/shared/ec2-api-tools-1.5.0.0/bin/ec2-describe-instances: line 9: EC2_HOME: EC2_HOME is not set
./home/shared/ec2-api-tools-1.5.0.0/bin/ec2-describe-instances: line 9: EC2_HOME: EC2_HOME is not set
^X./home/shared/ec2-api-tools-1.5.0.0/bin/ec2-describe-instances: line 9: EC2_HOME: EC2_HOME is not set

I followed instructions for my.bashrc and have tried setting EC2_HOME variable manually too w/ export EC2_HOME=/home/shared/ec2-api-tools-1.5.0.0 prior to running sudo create_ownAMI.com

Thanks for any guidance you may have.
rts