Skip to content

Commit ee52059

Browse files
Andrew Gaulgaul
authored andcommitted
Allow non-AWS endpoints
This is useful for local Ceph S3 deployments. Fixes #10. Fixes #65.
1 parent 632db70 commit ee52059

File tree

3 files changed

+24
-1
lines changed

3 files changed

+24
-1
lines changed

CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
## 4.0.3
2+
- Support for non-AWS endpoints
3+
14
## 4.0.2
25
- Fixed AWS authentication when using instance profile credentials.
36

CONTRIBUTORS

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Contributors:
1111
* Nick Ethier (nickethier)
1212
* Pier-Hugues Pellerin (ph)
1313
* Richard Pijnenburg (electrical)
14+
* Andrew Gaul (andrewgaul)
1415

1516
Note: If you've sent us patches, bug reports, or otherwise contributed to
1617
Logstash, and you aren't on the list above and want to be, please let us know

lib/logstash/outputs/s3.rb

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
require "pathname"
1414
require "aws-sdk"
1515
require "logstash/outputs/s3/patch"
16+
require "uri"
1617

1718
Aws.eager_autoload!
1819

@@ -64,9 +65,10 @@
6465
# This is an example of logstash config:
6566
# [source,ruby]
6667
# output {
67-
# s3{
68+
# s3 {
6869
# access_key_id => "crazy_key" (required)
6970
# secret_access_key => "monkey_access_key" (required)
71+
# endpoint => "http://127.0.0.1:8080" (optional, used for non-AWS endpoints, default = "")
7072
# region => "eu-west-1" (optional, default = "us-east-1")
7173
# bucket => "your_bucket" (required)
7274
# size_file => 2048 (optional) - Bytes
@@ -106,6 +108,10 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
106108
# S3 bucket
107109
config :bucket, :validate => :string, :required => true
108110

111+
# Specify a custom endpoint for use with non-AWS S3 implementations, e.g.,
112+
# Ceph. Provide a URL in the format http://127.0.0.1:8080/
113+
config :endpoint, :validate => :string
114+
109115
# Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file.
110116
# If you have tags then it will generate a specific size file for every tags
111117
##NOTE: define size of file is the better thing, because generate a local temporary file on disk and then put it in bucket.
@@ -256,6 +262,7 @@ def full_options
256262
options = Hash.new
257263
options[:s3_signature_version] = @signature_version if @signature_version
258264
options.merge(aws_options_hash)
265+
.merge(endpoint_options)
259266
end
260267

261268
def normalize_key(prefix_key)
@@ -288,6 +295,18 @@ def aws_service_endpoint(region)
288295
{ :s3_endpoint => region == 'us-east-1' ? 's3.amazonaws.com' : "s3-#{region}.amazonaws.com"}
289296
end
290297

298+
def endpoint_options
299+
if @endpoint
300+
uri = URI(@endpoint)
301+
{
302+
:endpoint => @endpoint,
303+
:force_path_style => true,
304+
}
305+
else
306+
{}
307+
end
308+
end
309+
291310
def upload_options
292311
{
293312
:acl => @cannel_acl,

0 commit comments

Comments
 (0)