Allow non-AWS endpoints

This is useful for local Ceph S3 deployments.  Fixes #10.  Fixes #65.

Author: Andrew Gaul

CHANGELOG.md

@@ -1,3 +1,6 @@
+## 3.2.1
+  - Support for non-AWS endpoints
+
 ## 3.2.0
   - Move to the new concurrency model `:single`
   - use correct license identifier #99

CONTRIBUTORS

@@ -10,6 +10,7 @@ Contributors:
 * Nick Ethier (nickethier)
 * Pier-Hugues Pellerin (ph)
 * Richard Pijnenburg (electrical)
+* Andrew Gaul (andrewgaul)
 
 Note: If you've sent us patches, bug reports, or otherwise contributed to
 Logstash, and you aren't on the list above and want to be, please let us know

lib/logstash/outputs/s3.rb

@@ -8,6 +8,7 @@
 require "thread"
 require "tmpdir"
 require "fileutils"
+require "uri"
 
 
 # INFORMATION:
@@ -59,9 +60,10 @@
 # This is an example of logstash config:
 # [source,ruby]
 # output {
-#    s3{
+#    s3 {
 #      access_key_id => "crazy_key"             (required)
 #      secret_access_key => "monkey_access_key" (required)
+#      endpoint => "http://127.0.0.1:8080"      (optional, used for non-AWS endpoints, default = "")
 #      region => "eu-west-1"                    (optional, default = "us-east-1")
 #      bucket => "boss_please_open_your_bucket" (required)
 #      size_file => 2048                        (optional) - Bytes
@@ -84,6 +86,10 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
   # S3 bucket
   config :bucket, :validate => :string
 
+  # Specify a custom endpoint for use with non-AWS S3 implementations, e.g.,
+  # Ceph.  Provide a URL in the format http://127.0.0.1:8080/
+  config :endpoint, :validate => :string
+
   # Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file.
   # If you have tags then it will generate a specific size file for every tags
   ##NOTE: define size of file is the better thing, because generate a local temporary file on disk and then put it in bucket.
@@ -147,6 +153,7 @@ def aws_s3_config
 
   def full_options
     aws_options_hash.merge(signature_options)
+                    .merge(endpoint_options)
   end
 
   def signature_options
@@ -157,6 +164,20 @@ def signature_options
     end
   end
 
+  def endpoint_options
+    if @endpoint
+      uri = URI(@endpoint)
+      {
+        :s3_force_path_style => true,
+        :s3_endpoint => uri.host,
+        :s3_port => uri.port,
+        :use_ssl => uri.scheme == "https",
+      }
+    else
+      {}
+    end
+  end
+
   def aws_service_endpoint(region)
     return {
       :s3_endpoint => region == 'us-east-1' ? 's3.amazonaws.com' : "s3-#{region}.amazonaws.com"