Update documents (#384)

* WIP: Doc

* more docs

* Update documents

* More doc update

Author: greensky00

README.md

@@ -30,7 +30,8 @@ Features
 * [Custom/separate quorum size for commit & leader election](docs/custom_quorum_size.md)
 * [Asynchronous replication](docs/async_replication.md)
 * [SSL/TLS support](docs/enabling_ssl.md)
-
+* [Parallel Log Appending](docs/parallel_log_appending.md)
+* [Custom Commit Policy](docs/custom_commit_policy.md)
 
 How to Build
 ------------
@@ -111,9 +112,9 @@ Please refer to [tests/bench](./tests/bench).
 
 Supported Platforms
 -------------------
-* Ubuntu (tested on 14.04, 16.04, and 18.04)
+* Ubuntu (tested on 14.04 -- 20.04)
 * Centos (tested on 7)
-* OSX (tested on 10.13 and 10.14)
+* OSX (tested on 10.13 -- 12.3)
 * Windows (built using MSVC 2019, not thoroughly tested)
 
 
@@ -130,7 +131,7 @@ Contact
 
 License Information
 --------------------
-Copyright 2017-2019 eBay Inc.
+Copyright 2017-present eBay Inc.
 
 Author/Developer: Jung-Sang Ahn
 

docs/async_replication.md

@@ -1,13 +1,13 @@
 Asynchronous Replication
 ------------------------
 
-The original Raft's commit happens after reaching quorum, which means that network communication is always involved in users' operation path.
+The original Raft's commit happens after reaching the quorum, which means that network communication is always involved in users' operation paths.
 
-However, there are some loosened cases that we want to achieve low latency by sacrificing consistency and resolving conflicts manually. Then waiting for the acknowledges from a majority of servers is a waste of time.
+However, there are some loosened cases where we want to achieve low latency by sacrificing consistency and resolving conflicts manually. Then waiting for the acknowledgment from a majority of servers is a waste of time.
 
 To support such cases, we provide `async_replication_` flag in [`cluster_config`](../include/libnuraft/cluster_config.hxx). If that flag is set, `append_entries()` API immediately returns with the result of `state_machine::pre_commit()`, and replication is done in background later.
 
-Below diagram shows the overall flow. You can compare it with [original sequence](replication_flow.md):
+The below diagram shows the overall flow. You can compare it with [original sequence](replication_flow.md):
 ```
 User    Leader  Follower(s)
 |       |       |
@@ -32,8 +32,8 @@ To enable asynchronous replication, `state_machine::pre_commit()` function shoul
 In synchronous replication mode, we provide another option: `async_handler` in [`raft_params`](../include/libnuraft/raft_params.hxx). Here are the differences between asynchronous replication mode and synchronous replication with `async_handler` mode:
 
 * Synchronous replication with `async_handler` mode:
-    * The actual execution in state machine happens after reaching consensus.
-    * Although `append_entries()` API returns immediately, the given data is not committed yet. The result of `commit()` will be set later, by invoking user-defined handler as a notification.
+    * The actual execution in the state machine happens after reaching consensus.
+    * Although `append_entries()` API returns immediately, the given data is not committed yet. The result of `commit()` will be set later by invoking a user-defined handler as a notification.
 * Asynchronous replication mode:
-    * The actual execution in state machine happens before replication.
-    * `append_entries()` API returns immediately, which already contains the result of state machine execution. There is no later notification.
+    * The actual execution in the state machine happens before replication.
+    * `append_entries()` API returns immediately, which already contains the result of state machine execution. Hence, there will be no later notification.

docs/basic_operations.md

@@ -5,13 +5,13 @@ Basic Operations
 
 Initializing Raft Server
 ------------------------
-First of all, you need to define your own [log store](../include/libnuraft/log_store.hxx), [state machine](../include/libnuraft/state_machine.hxx), and [state manager](../include/libnuraft/state_mgr.hxx) (optionally [debugging logger](../include/libnuraft/logger.hxx)):
+First of all, you should implement your own [log store](../include/libnuraft/log_store.hxx), [state machine](../include/libnuraft/state_machine.hxx), and [state manager](../include/libnuraft/state_mgr.hxx) (optionally [debugging logger](../include/libnuraft/logger.hxx)):
 ```C++
 ptr<logger> my_logger;
 ptr<state_machine> my_state_machine;
 ptr<state_mgr> my_state_manager;
 ```
-Log store will not be passed at initialization time, but will be loaded through `load_log_store()` API in state manager later. So you need properly implement that function.
+Log store will not be passed at the initialization time but will be loaded by `load_log_store()` API in the state manager later. So you need properly implement that function.
 
 After that, set your [Asio](../include/libnuraft/asio_service_options.hxx) and [Raft](../include/libnuraft/raft_params.hxx) options:
 ```C++
@@ -37,17 +37,17 @@ if (server->is_initialized()) {
 
 ### What Is Happening on Raft Initialization? ###
 
-Once you initialize Raft server, it will invoke below APIs from your custom modules:
+Once you initialize the Raft server, it will invoke the below APIs from your custom modules:
 
 * `state_mgr::load_log_store()`
     * This function should return your log store instance.
 * `state_mgr::load_config()`
-    * This function should return the last committed Raft cluster config, that contains the membership info.
-    * At the very first launch, you can return a cluster config that contains the server itself only. After adding server the cluster config will change, and you should make it durable (if necessary).
+    * This function should return the last committed Raft cluster config containing the membership info.
+    * At the very first launch, you can return a cluster config containing the server itself only. After adding servers, the cluster config will change, and you should make it durable (if necessary).
 * `state_mgr::read_state()`
     * This function should return the last [server state](../include/libnuraft/srv_state.hxx), that contains term and voting info.
 * `state_machine::last_commit_index()`
-    * You should make the last committed log number durable (if necessary), and return it here. Otherwise, Raft server attempts to do catch-up from the beginning.
+    * You should make the last committed log number durable (if necessary), and return it here. Otherwise, the Raft server attempts to do the log replaying from the beginning.
 * `state_machine::last_snapshot()`
     * You should make the last snapshot durable (if necessary), and return the handle of it here.
 
@@ -58,7 +58,7 @@ You can simply use [Launcher](../include/libnuraft/launcher.hxx)'s shutdown API:
 ```C++
 bool success = launcher.shutdown();
 ```
-This API is a blocking call, so that the server termination is guaranteed once the function returns `true`.
+This API is a blocking call, so the server termination is guaranteed once the function returns `true`.
 
 Adding Server -- Organizing a Group
 ---
@@ -73,8 +73,8 @@ server->add_srv( server_to_add );
 ```
 Note:
 * `add_srv()` API is an asynchronous task, thus need to check the result using `get_srv_config()` API.
-* The server to be added should be running and empty at the time you add server.
-* If the log of leader has been compacted (i.e., the smallest log number is greater than 1), leader will transfer snapshot first. Before receiving snapshot is done, the server is officially not the member of Raft group. In the meantime, you also cannot add other servers concurrently.
+* The server to be added should be running at the time you add the server.
+* If the leader's logs have already been compacted (i.e., the smallest log number is greater than 1), the leader will transfer a snapshot first. Before receiving the snapshot is done, the server is officially not a member of the Raft group. In the meantime, you also cannot add other servers concurrently.
 
 Removing Server
 ---
@@ -85,7 +85,7 @@ server->remove_srv( server_id_to_remove );
 ```
 The same as `add_srv()` API, `remove_srv()` is also an asynchronous task so that you need to check the result by using `get_srv_config()` API.
 
-The server to be removed should be running at the time you remove server. Otherwise, the leader will attempt to communicate with it a few times, and then force remove it.
+The server to be removed should be running at the time you remove the server. Otherwise, the leader will attempt to communicate with it a few times and then force remove it.
 
 Appending Log -- Replication
 ---

docs/custom_commit_policy.md

@@ -0,0 +1,28 @@
+Custom Commit Policy
+====================
+
+In addition to the basic quorum-based consensus, NuRaft provides 1) full consensus mode and 2) selective quorum.
+
+
+Full Consensus Mode
+-------------------
+The leader commits a log only when all healthy members have the log, in order to achieve strong consistency; once a log is committed, the latest data can be read from any members.
+
+However, stronger consistency worsens availability, and full consensus mode is the extreme case. If there is at least one unreachable member, the entire protocol stops. To avoid such a bad availability, full consensus mode dynamically excludes *unhealthy* members from the quorum. If there is any node not responding longer than `response_limit_` (in [`raft_server::limits`](../include/libnuraft/raft_server.hxx)) multiplied by heartbeat period, NuRaft automatically regards it as *unhealthy*. The full consensus is achieved among healthy members only. Unhealthy members become healthy as soon as they respond to the leader's message.
+
+If the number of unhealthy members becomes a majority, then the leader will not be able to commit logs, the same as the basic quorum-based consensus.
+
+
+Selective Quorum
+----------------
+[State machine](../include/libnuraft/state_machine.hxx) interface provides `adjust_commit_index` API for selective quorum. This API is called for each commit decision, with `adjust_commit_index_params`. This parameter contains the list of <peer ID, its last log index> pairs, along with the current commit index and the new commit index determined by NuRaft. This API will return the new log index to commit.
+
+With the given information, we can pin some servers in the quorum so as to make them always have the latest committed log. For example, let's assume we have 5 servers, and their ID and last log index are as follows:
+```
+{{1, 10}, {2, 9}, {3, 10}, {4, 10}, {5, 8}}
+```
+In such a case, with the original Raft algorithm, the commit index should be `10`, as `{1, 3, 4}` can form a quorum.
+
+However, if we want to make server 2 always have the latest committed log, server 2 should be pinned in the quorum for commit. So quorum can be either `{1, 2, 3}` or `{1, 2, 4}` in this example, and the commit index number should be `9`. We can inform such a decision to NuRaft by letting `adjust_commit_index` return `9`. In this case, the `append_entries` request for log `10` will be pending until log `10` is committed, i.e., until server 2 receives log `10`.
+
+Similar to full consensus mode, pinning a fixed set of servers in the quorum will sacrifice availability. Users who implement the selective quorum are responsible for tuning and its trade-off.

docs/custom_metadata.md

@@ -0,0 +1,43 @@
+Custom Metadata for Each Message
+--------------------------------
+
+NuRaft provides an ability to ship custom metadata for each message and response that can be used for your own verification. When you set [`asio_options`](../include/libnuraft/asio_service_options.hxx), there are a few options for it:
+```C++
+asio_service::options asio_opt;
+asio_opt.write_req_meta_  = my_write_req_meta;
+asio_opt.read_req_meta_   = my_read_req_meta;
+asio_opt.write_resp_meta_ = my_write_resp_meta;
+asio_opt.read_resp_meta_  = my_read_resp_meta;
+```
+
+* `write_req_meta_`: return custom metadata to be shipped with the Raft message.
+```c++
+std::string my_write_req_meta(const asio_service::meta_cb_params& params) {
+    return "my req metadata";
+}
+```
+
+* `read_req_meta_`: read the custom metadata shipped with each Raft message, and return a boolean flag to decide to accept the message.
+```c++
+bool my_read_req_meta(const asio_service::meta_cb_params& params,
+                      const std::string& meta) {
+    return (meta == "my req metadata");
+}
+```
+
+* `write_resp_meta_`: return custom metadata to be shipped with the Raft message response.
+```c++
+std::string my_write_resp_meta(const asio_service::meta_cb_params& params) {
+    return "my resp metadata";
+}
+```
+
+* `read_resp_meta_`: read the custom metadata shipped with the Raft message response, and return a boolean flag to decide to accept the response.
+```c++
+bool my_read_resp_meta(const asio_service::meta_cb_params& params,
+                       const std::string& meta) {
+    return (meta == "my resp metadata");
+}
+```
+
+If `read_req_meta_` or `read_resp_meta_` returns `false`, the message or response is discarded immediately; thus it behaves the same as if it is not received.

docs/custom_quorum_size.md

@@ -5,7 +5,7 @@ The motivation comes from [Flexible Paxos](https://fpaxos.github.io/) paper:
 
 * Howard et al., [Flexible Paxos: Quorum Intersection Revisited](https://arxiv.org/pdf/1608.06696v1.pdf), 2016.
 
-The basic idea is that as long as there is at least one overlapping node between the quorum for commit and the quorum for leader election, the entire group is safe. For example, let's say Qc and Qe represent the size of quorum for commit and leader election, respectively. If we have 5 servers, the set of {Qc, Qe} pairs {1, 5}, {2, 4}, {3, 3} (original algorithm), {4, 2}, and {5, 1} provides the same level of safety. Note that availability will be sacrificed as the value of |Qc - Qe| increases.
+The basic idea is that as long as there is at least one overlapping node between the quorum for commit and the quorum for leader election, the entire group is safe. For example, let's say Qc and Qe represent the size of quorum for the commit and leader election, respectively. If we have 5 servers, the set of {Qc, Qe} pairs {1, 5}, {2, 4}, {3, 3} (original algorithm), {4, 2}, and {5, 1} provides the same level of safety. Note that availability will be sacrificed as the value of |Qc - Qe| increases.
 
 For custom quorum size, we provide two parameters: `custom_commit_quorum_size_` and `custom_election_quorum_size_` in [`raft_params`](../include/libnuraft/raft_params.hxx).
 
@@ -17,7 +17,7 @@ params.custom_election_quorum_size_ = 4;
 
 The default value of both parameters is 0, which follows the original algorithm.
 
-Those parameters are dynamically adjustable; you can change the size of quorum without shutting down Raft server:
+Those parameters are dynamically adjustable; you can change the size of quorums without shutting down the Raft server:
 
 ```C++
 raft_params params = server->get_current_params();
@@ -26,4 +26,4 @@ params.custom_election_quorum_size_ = 4;
 server->update_params(params);
 ```
 
-Note that it is also possible to set those quorum sizes without intersection: {2, 2} out of 5 servers for example. In such case, data loss or log diverging is inevitable and resolving those problems is your responsibility.
+Note that it is also possible to set those quorum sizes without intersection: {2, 2} out of 5 servers, for example. In such cases, data loss or log diverging is inevitable, and resolving those problems is your responsibility.

docs/custom_resolver.md

@@ -0,0 +1,32 @@
+Custom Resolver
+---------------
+
+You can attach your custom resolver that can be used when establishing new connections. When you set [`asio_options`](../include/libnuraft/asio_service_options.hxx), there is an option named `custom_resolver_`:
+```C++
+std::function< void( const std::string&,
+                     const std::string&,
+                     asio_service_custom_resolver_response ) > custom_resolver_;
+```
+The first parameter contains the host name to resolve, and the second has the port number. The third parameter is a callback function to be invoked when your resolver finishes its job.
+
+The type of the callback function is as follows:
+```c++
+using asio_service_custom_resolver_response =
+    std::function< void(const std::string&, const std::string&, std::error_code) >;
+```
+You need to pass the resolved IP address (or the host name that the default resolver can understand) to the first parameter and the port number to the second one. The third parameter should contain an error code if resolving the host name fails.
+
+This is a very simple example of resolving `"my_custom_localhost"`:
+```c++
+asio_service::options asio_opt;
+asio_opt.custom_resolver_ =
+    [](const std::string& host,
+       const std::string& port,
+       asio_service_custom_resolver_response when_done) {
+        if (host == "my_custom_localhost") {
+            when_done("127.0.0.1", port, std::error_code());
+        }
+        // Otherwise: pass through. The default resolver will do the rest.
+        when_done(host, port, std::error_code());
+    };
+```

docs/dealing_with_buffer.md

@@ -1,33 +1,33 @@
 Dealing with Buffer
 ---
 
-[`buffer`](../include/libnuraft/buffer.hxx) class instance just points to a raw memory blob. At the beginning of each memory blob, a few bytes are reserved for metadata: 4 bytes if buffer size is less than 32KB and 8 bytes otherwise. User data section starts right after that, thus you should write your data starting from there.
+[`buffer`](../include/libnuraft/buffer.hxx) class instance just points to a raw memory blob. At the beginning of each memory blob, a few bytes are reserved for metadata: 4 bytes if the buffer size is less than 32KB and 8 bytes otherwise. The user data section starts right after that; thus you should write your data starting from there.
 
 You can use `alloc()` API to allocate memory:
 ```C++
 ptr<buffer> b = buffer::alloc( size_to_allocate );
 ```
 
-You can get the starting point of user section by using `data()` API:
+You can get the starting point of the user section by using `data()` API:
 ```C++
 void* ptr = (void*)b->data();
 ```
 
-Each buffer instance has a cursor pointing to its current internal position (initially 0), you can get or set the position:
+Each buffer instance has a cursor pointing to its current internal position (initially 0); you can get or set the position:
 ```C++
 size_t current_position = b->pos();
 ...
 b->pos( new_position );
 ```
 
-If you want to get the starting point of user section regardless of the current position, use `data_begin()` API:
+If you want to get the starting point of the user section regardless of the current position, use `data_begin()` API:
 ```C++
 void* ptr_begin = (void*)b->data_begin();
 ```
 
 Buffer Serializer
 ---
-`buffer` itself has a few APIs to get and put data, but we do not recommend using those APIs, since they change the internal position of the buffer, which blocks concurrent reads and also is mistake-prone if you forget to reset the position.
+`buffer` itself has a few APIs to get and put data, but we do not recommend using those APIs since they change the internal position of the buffer, which blocks concurrent reads and also is mistake-prone if you forget to reset the position.
 
 Instead, you can use [`buffer_serializer`](../include/libnuraft/buffer_serializer.hxx):
 ```C++
@@ -37,4 +37,4 @@ buffer_serializer s(b);
 ```
 `buffer_serializer` allows concurrent access to the same `buffer` instance.
 
-We provide endianness options, and it is little endian by default.
+We provide endianness options, and it is a little endian by default.