Mas 3.0.16 preview (#1138)

* Update deps for 3.0.16 test

* Update deps

* Prepare for release

Now with riak_core claim algorithm update

* Add addendum to 3.0.12 release notes to highlight snappy issue

* Update RELEASE-NOTES.md

* Update to tagged repos

Author: martinsumner

RELEASE-NOTES.md

@@ -1,10 +1,22 @@
+# Riak KV 3.0.16 Release Notes
+
+This release includes the following updates:
+
+- A significant new [riak_core cluster claim algorithm](https://github.com/basho/riak_core/pull/1008) has been added - `choose_claim_v4`.  The default cluster claim algorithm (`choose_claim_v2`) is unchanged, but the `choose_claim_fun` configuration option in riak.conf can be used to enable the new algorithm.  The new algorithm offers significant improvements in the claim process, especially when location awareness is enabled, with the algorithm now more likely to find valid solutions and with fewer required transfers.  The trade-off when compared to `choose_claim_v2` is that in some situations computational overhead of claim may increase by multiple orders of magnitude, leading to long cluster plan times.
+
+- The leveled backend has been updated to reduce the memory required for stores with a large number of keys, and also reduce volatility in the memory demanded, by [optimising the memory footprint of the SST process working heap](https://github.com/martinsumner/leveled/pull/403).
+
+- A collection of fixes have been added.  A fix to timeouts in [leveled hot backups](https://github.com/martinsumner/leveled/pull/406).  A fix to reap and erase queries to [prevent a backlog from overloading riak_kv_eraser or riak_kv_reaper mailboxes](https://github.com/basho/riak_kv/pull/1862) (creating rapidly increasing memory footprint).  A fix to an issue with [nextgen replication when node_confirms is enabled](https://github.com/basho/riak_kv/pull/1858).  The [R/W value used by nextgen replication is now configurable](https://github.com/basho/riak_kv/pull/1860), and this may be useful for preventing a backlog of replication events from overloading a cluster.
+
+As a result of the memory management improvements made in 3.0.16, the recommendation to consider altering the eheap single-block carrier threshold made in [Riak 3.0.12](#riak-kv-3012-release-notes) has been deprecated.  With the leveled backend, memory management should now be efficient on default settings, although with a small overhead on the latency of HEAD operations in that backend.
+
 # Riak KV 3.0.15 Release Notes
 
 Fix to an issue introduced with the `auto_check` feature for TictacAAE full-sync in [Riak KV 3.0.10](#riak-kv-3010-release-notes).
 
 # Riak KV 3.0.14 Release Notes
 
-This release [fixes an issue](https://github.com/martinsumner/leveled/issues/393) whereby a failure to signal and handle back-pressure correctly by the leveled backend can cause a backlog within the store.  In particular this can be triggered by handoffs (e.g. due to cluster admin operations), and lead to partition transfers stalling almost completely.  The issue existed in previous releases, by may have been exacerbated by refactoring in [Riak KV 3.0.13](#riak-kv-3013-release-notes).
+This release [fixes an issue](https://github.com/martinsumner/leveled/issues/393) whereby a failure to signal and handle back-pressure correctly by the leveled backend can cause a backlog within the store.  In particular this can be triggered by handoffs (e.g. due to cluster admin operations), and lead to partition transfers stalling almost completely.  The issue existed in previous releases, but may have been exacerbated by refactoring in [Riak KV 3.0.13](#riak-kv-3013-release-notes).
 
 An additional [minor improvement has been made to handoffs](https://github.com/basho/riak_kv/pull/1851). Previously requests to reap tombstones after deletions (where the delete_mode is not `keep`), would not be forwarded during handoffs.  These tombstones would then need to be corrected by AAE (which may result in a permanent tombstone).  There is now a configuration option `handoff_deletes` which can be enabled to ensure these reap requests are forwarded, reducing the AAE work required on handoff completion.
 
@@ -50,6 +62,12 @@ This is a general release of changes and fixes:
 
 As part of this release, further testing of the new memory configuration options added in Riak 3.0.10 has been undertaken.  It is now recommended when using the leveled backend, that if memory growth in the Riak process is a signifcant concern, then the following configuration option may be tested: `erlang.eheap_memory.sbct = 128`.  This has been shown to reduce the memory footprint of Riak, with a small performance overhead.
 
+## 3.0.12 Addendum
+
+Although not originally advertised in the release, the version of eleveldb used in 3.0.12 includes an upgrade of the [`snappy` compression algorithm used within eleveldb](https://github.com/basho/eleveldb/pull/267).  This version of snappy is incompatible with the previous version used, and so any update to 3.0.12 and beyond where the eleveldb backend is deployed with `snappy` compression [will fail](https://github.com/basho/eleveldb/issues/273).
+
+To update to 3.0.12 in this case, the change should be rolled into the cluster using the ` riak admin cluster replace` command.
+
 # Riak KV 3.0.11 Release Notes
 
 A simple change to [release a bottleneck](https://github.com/martinsumner/leveled/issues/379) in 2i queries with the leveled backend.  Should only be relevant to those using leveled, and attempting o(1000) 2i queries per second.

rebar.config

@@ -11,13 +11,13 @@
 {deps, [
         {lager_syslog, {git, "https://github.com/basho/lager_syslog.git", {tag, "3.1.1"}}},
         {cluster_info, {git, "https://github.com/basho/cluster_info.git", {tag, "2.1.0"}}},
-        {riak_kv, {git, "https://github.com/basho/riak_kv.git", {tag, "riak_kv-3.0.15"}}},
+        {riak_kv, {git, "https://github.com/basho/riak_kv.git", {tag, "riak_kv-3.0.16"}}},
         {riak_auth_mods, {git, "https://github.com/basho/riak_auth_mods.git", {tag, "riak_kv-3.0.10"}}},
         {riaknostic, {git, "https://github.com/basho/riaknostic.git", {tag, "riak_kv-3.0.10"}}},
 
         % {yokozuna, {git, "https://github.com/basho/yokozuna.git", {branch, "develop-3.0"}}},
 
-        {riak_repl, {git, "https://github.com/basho/riak_repl.git", {tag, "riak_kv-3.0.15"}}}
+        {riak_repl, {git, "https://github.com/basho/riak_repl.git", {tag, "riak_kv-3.0.16"}}}
       ]}.
 
 {project_plugins, [
@@ -46,7 +46,7 @@
     ]}
 ]}.
 
-{relx, [{release, {riak, "3.0.15"},
+{relx, [{release, {riak, "3.0.16"},
     [kernel,
      stdlib,
      lager,

rebar.lock

@@ -50,7 +50,7 @@
   2},
  {<<"kv_index_tictactree">>,
   {git,"https://github.com/martinsumner/kv_index_tictactree.git",
-       {ref,"cd192703c2464bc0f0da12177b077c000c12b392"}},
+       {ref,"50302987d33689fb2066c831ec7376f11d4304c9"}},
   1},
  {<<"lager">>,
   {git,"https://github.com/erlang-lager/lager.git",
@@ -62,7 +62,7 @@
   0},
  {<<"leveled">>,
   {git,"https://github.com/martinsumner/leveled",
-       {ref,"bd6ec2fdb591aa199e18b1d4abeb3fe8e037926c"}},
+       {ref,"4142914e3fc32090747a4a7ea63087e18a74a042"}},
   2},
  {<<"lz4">>,
   {git,"https://github.com/martinsumner/erlang-lz4",
@@ -103,15 +103,15 @@
   1},
  {<<"riak_api">>,
   {git,"https://github.com/basho/riak_api.git",
-       {ref,"d38c4c33b2d803ba1488017e5942c240ffa46c9b"}},
+       {ref,"1ccf72103a154493e952e7a544ea3429cb342b20"}},
   1},
  {<<"riak_auth_mods">>,
   {git,"https://github.com/basho/riak_auth_mods.git",
        {ref,"5e4a240791f6bbadc9b3bce4653ba08cb9aeb0ce"}},
   0},
  {<<"riak_core">>,
   {git,"https://github.com/basho/riak_core.git",
-       {ref,"10cad320852a66d9d3c81a1f3745e7e6a5286cc2"}},
+       {ref,"7b04e2c4731c16ffb9cfbe1ac085cb4056cb0422"}},
   1},
  {<<"riak_dt">>,
   {git,"https://github.com/basho/riak_dt.git",
@@ -123,19 +123,19 @@
   2},
  {<<"riak_kv">>,
   {git,"https://github.com/basho/riak_kv.git",
-       {ref,"ed93f69c1e345620534c36282ba7fc08f8d9ed26"}},
+       {ref,"e80a0699ffc5bc004b93e06c09197b79109d6844"}},
   0},
  {<<"riak_pb">>,
   {git,"https://github.com/basho/riak_pb.git",
        {ref,"04e8cc7daa081c23be137b1eff2a6a3fb827f8e4"}},
   2},
  {<<"riak_pipe">>,
   {git,"https://github.com/basho/riak_pipe.git",
-       {ref,"0fc3833b0dfb1166753b67af460eb8af6b8af1bf"}},
+       {ref,"abedf61218d4ed61a3494a12e95bd4de0277468d"}},
   1},
  {<<"riak_repl">>,
   {git,"https://github.com/basho/riak_repl.git",
-       {ref,"1b3f602861a5e8045e13fcd8b1f4f5efae5cfefa"}},
+       {ref,"dab42494f35b5102925138b28e8a1b0492947287"}},
   0},
  {<<"riak_sysmon">>,
   {git,"https://github.com/basho/riak_sysmon.git",