Store the resources in S3 buckets (data file and revisions offset files)

[JohannesLichtenberger]

As an alternative backend or a combination of a local store and an S3 bucket store we could add a new storage type using JClouds blob store for instance (as in Treetank, the former project from which this project was forked).

[chatterjeesudip]

Hi @JohannesLichtenberger I would like to work on this issue.

[JohannesLichtenberger]

@sudip-unb did you make any advances or do you need help?

[chatterjeesudip]

Hi @JohannesLichtenberger please give me a little bit time. I am currently busy for my exam. I already explored the link you have provided me. I can start development from this weekend.

[JohannesLichtenberger]

Sure, just trying to ping assigned people because usually it's likely that they do not have time at all in my experience ;-) but glad that you're starting to work on it afterward. Take your time -- as said, I just wanted to make sure that you're still up for the task :-)

[JohannesLichtenberger]

@sudip-unb any news? :)

[Yashendr]

Hi can I help out with this issue?

[JohannesLichtenberger]

@Yashendr you can have a look at Treetank (https://github.com/sebastiangraf/treetank) where Sebastian already implemented such a backend (also a combined storage)...

[JohannesLichtenberger]

https://github.com/sebastiangraf/treetank/tree/master/coremodules/core/src/main/java/org/treetank/io/jclouds

[sband]

Hi @JohannesLichtenberger ,

I went through the org.sirix.io package to understand the storage types that SirixDB has currently.
From what I have understood, FileStorage is already supported, however the above issue speaks about the combination of local file storage and S3 bucket store. Could you please elaborate more on this ask ? Or is it just that a support for S3 storage is needed as per the issue ?
Once I understand this, I'd be happy to contribute my bit to this project.
However, for starters here are my approaches of how I could implement this:

1. Using the FileWriter write data and revision files on the filesystem and copy them to S3. (I may be completely wrong about this given that I have not looked at the logic of how the page references are written.
2. Extend the current logic to write on to S3 object store directly
   a. Externalize the configuration to be able to provide AWS keys and S3 bucket details
   b. Create a connection to S3 bucket using the above configuration (in point "a")
   c. Using AWS SDK APIs, use the current logic to write the page references on to S3 object directly.

In approach 2 I could refer/use the jclouds package you have mentioned

Let me know what you think about these approaches.

[JohannesLichtenberger]

I'd probably see it as a kind of automatic backup. A local file based store and an async store via JClouds. IIRC the pure S3 storage was way too slow. So, in short I prefer your second option. To combine the storage approaches we can implement something as simple as this combined storage: https://github.com/sebastiangraf/treetank/tree/master/coremodules/core/src/main/java/org/treetank/io/combined

BTW: If you dig a bit deeper into the storage mechanism (simply store word aligned page fragments instead of same sized full pages), it would also be interesting to find out, why the iouring based backend currently is slower than the simple file channel based solution and the memory mapped backend (I think somehow because of the event loop)...