Does IGV support non amazon s3 buckets? #1636
Comments
|
I don't have enough information to help you debug your problem. For starters what version of IGV are you using? |
|
And what is running that S3 server - MinIO or something similar? We have some on-prem S3-compatible object stores that we could test against if needed. |
|
@mrvollger If you could setup a test instance and give me credentials I will look further, but first verify you are using the latest IGV. There were S3 issues with some versions of 2.18.x |
|
Thanks for the reply! And sorry, I wanted first to establish that this should work before troubleshooting, but it sounds like it should. I downloaded the latest version of IGV MacOS Apple yesterday during my test, 2.19.1. I am not sure about the details of what is running the s3 server. It is a service provided by UW, so I will inquire about it with them and get back to you. They also administer credentials, so I will request those to establish a test case. Thanks, |
|
If you're able to create test credentials you can email them privately to [email protected]. I don't know what I will be able to determine but I will look. There might also be information in your igv log file (usually named igv0.log in /igv. |
|
Hi Jim, I just sent an email with the credentials! This is an example error log I get when I try doing this: Finally, our IT team shared that this s3 endpoint is hosted by a "Ceph RADOS Gateway"; hopefully, that is useful information, but I can always ask for more details if this does not help. I really appreciate the help. Thanks, |
|
AFAICT, currently non-Amazon buckets shouldn't work since there's no provision for so-called "custom endpoints" in the current The change could potentially be relatively straightforward by (conditionally via BUT, as you can see in the issue above, this usecase doesn't seem to be fully supported by the Java2 AWS SDK, so caution must be exercised to not hit the wrong buttons (i.e avoiding parsing non-S3 URIs)... and/or find suitable workarounds. I'll leave Jim putting those bits together, but let me know if you need assistance, @jrobinso ;) |
|
Thanks @brainstorm, but they are not configuring oAuth. They are using .aws/credentials. |
Oh, I didn't know the S3Client code paths became independent (imho they shouldn't), I will revisit the implementation... anyway, here's something that could help you as well, Jim: |
|
@mrvollger Thanks for the test credentials and file. No success yet. The link @brainstorm includes above seemed somewhat promising, I tried setting the endpoint in the "config" file as suggested there but no luck, I got an exception that "region" was not set. I'm not sure what the region should be for a non-AWS endpoint, but setting it to us-east-1 did not work either (not surprisingly). There are probably IGV bugs as well, or implicit assumptions that the provider is AWS. So this is going to take more time than I have this evening, or even the next couple of days, but keep the issue open. I'm hopeful there will be a solution, but not certain. If you could ask the powers-that-be to leave those test credentials in place for a while that would be helpful. @brainstorm I'm not sure what you are referring to wrt "S3Client code paths became independent" but its probably not relevant at the moment given other issues I've found. |
|
@mrvollger One more question, it might be relevant or at least helpful to know what s3 compatible server you are running (e.e. minio). Putting this here as possibly relevant, for myself mostly when I return to this. https://stackoverflow.com/questions/76780500/how-to-using-aws-s3-java-v2-sdk-to-talk-to-s3-compatible-storage-minio |
|
Thanks so much for helping! We are really glad that you are willing to put time into this. We will keep the credentials valid as long as you need them, and if something happens, we can get another set up. I think when using a custom endpoint, the region doesn't matter or isn't used. But I am only guessing that based of this AWS CLI test: Last time, they told us the underlying vendor was Ceph RADOS Gateway, but I will ask more specifically what server is running the s3 instance and get back to you. Thanks again! |
|
CCing @sjneph. |
|
@mrvollger A workaround for the interim, if its allowed there, would be to create signed URLs with the AWS command line tools. The signed URLs should work. |
|
Thanks for the suggestion! We have been doing that for things in a real-time crunch, but we have a couple of applications where we cannot do that since the URLs are posted publicly. (I also hate when the links in my painfully constructed IGV sessions (XMLs) die). We are still waiting for our IT to share details about what runs the s3 server. |
|
Another potentially relevant stackoverflow but it does look like stuff covered already by the other two linked here. |
|
@mrvollger Thanks, that looks like potentially new info, and from a vendor (Cloudfare). Amazon is not interested in fixing bugs on this or making this easy, understandably. I'm in the middle of a block of work I must finish before concentrating on this, hopefully next week. Feel free to continue posting links here. |
|
BTW it would be really helpful, possibly essential, to know what backend you are running. Cloudfare has some instructions here but I won't waste my time with this if you are running something else. https://developers.cloudflare.com/r2/examples/aws/aws-sdk-java/ |
|
We have a Ceph RADOS Gateway and I believe that is built on a service called librados (https://docs.ceph.com/en/latest/radosgw/) but I am still waiting to get official confirmation... If I am right about this, I just found some docs that look relevant: UW also has some docs on programmatically accessing data on kopah (our s3 server), but it is focused on an API called boto3: I will send a reminder email to make sure they follow up with me. Thanks for the help! |
Hi Jim,
I have been trying to determine whether IGV can work with non-AWS S3 buckets. For example, the University of Washington hosts its own s3 endpoint (endpoint_url = https://s3.kopah.orci.washington.edu), on which we host a lot of data we want to view with IGV.
I can go into my .aws/credentials and .aws/config and set my defaults such that I can see into these buckets by default with the aws cli (or any other s3 cli):
But when I try opening up IGV and viewing these buckets, nothing happens when I click the "Load from S3 bucket" menu item.
Any advice would be much appreciated! And if IGV doesn't support this it would be awesome if it could be added. I would think the change should be small, though I don't know much about the API for s3.
Thanks,
Mitchell
The text was updated successfully, but these errors were encountered: