12 sqlite

Cargo.toml

@@ -20,3 +20,12 @@ serde = { version = "1.0.201", features = ["derive"] }
 serde_json = "1.0"
 sha2 = "0.10"
 log = "0.4.22"
+zeromq = "0.4.1"
+tokio = "1.40.0"
+bitcoincore-rpc = "0.19.0"
+
+# features= bundled
+#   This causes rusqlite to compile its own private libsqlite3 and link it with your Rust code, instead of using /usr/lib/x86_64-linux-gnu/libsqlite3.so 
+rusqlite = { version = "0.32.1", features = ["bundled"] }
+r2d2_sqlite = "0.25.0"
+r2d2 = "0.8.10"

README.md

@@ -1,89 +1,258 @@
-# gabriel
+gabriel
 
+- [1. Introduction](#1-introduction)
+- [2. Setup](#2-setup)
+  - [2.1. Pre-reqs](#21-pre-reqs)
+    - [2.1.1. Hardware](#211-hardware)
+    - [2.1.2. Software](#212-software)
+      - [2.1.2.1. Rust](#2121-rust)
+      - [2.1.2.2. bitcoind](#2122-bitcoind)
+      - [2.1.2.3. SQLite client](#2123-sqlite-client)
+  - [2.2. Clone](#22-clone)
+  - [2.3. Build](#23-build)
+  - [2.4. Execute tests](#24-execute-tests)
+- [3. Run Gabriel](#3-run-gabriel)
+  - [3.1. analyze all block data files](#31-analyze-all-block-data-files)
+  - [3.2. analyze single block data file](#32-analyze-single-block-data-file)
+    - [3.2.1. Optional:  debug via VSCode:](#321-optional--debug-via-vscode)
+  - [3.3. consume and analyze new raw block events](#33-consume-and-analyze-new-raw-block-events)
+    - [3.3.1. Fund a tx w/ a P2PK output on reg-test](#331-fund-a-tx-w-a-p2pk-output-on-reg-test)
+    - [3.3.2. Generate block:](#332-generate-block)
+    - [3.3.3. Optional: dDebug in VSCode:](#333-optional-ddebug-in-vscode)
+- [4. Inspect P2PK Analysis data](#4-inspect-p2pk-analysis-data)
+
+
+## 1. Introduction
 Measures how many unspent public key addresses there are, and how many coins are in them over time. Early Satoshi-era coins that are just sitting with exposed public keys. If we see lots of coins move... That's a potential sign that quantum computers have silently broken bitcoin.
 
-## Execution
+## 2. Setup
 
-### Pre-reqs
+### 2.1. Pre-reqs
 
-```
-$ bitcoind \
-    -conf=$GITEA_HOME/blockchain/bitcoin/admin/bitcoind/bitcoin.conf \
-    -daemon=0
-```
+#### 2.1.1. Hardware
+
+Gabriel requires a fully synced Bitcoin Core daemon to be running.
+Your hardware requirements will vary depending on the bitcoin network(ie: main, testnet4, regtest, etc) you choose.
 
-#### Hardware
+If running in _regtest_ (ie: for dev / test purposes) then use of a modern laptop will be plenty sufficient.
 
-#### Software
-##### Rust
+#### 2.1.2. Software
+##### 2.1.2.1. Rust
 The best way to install Rust is to use [rustup](https://rustup.rs).
 
-##### bitcoind
+##### 2.1.2.2. bitcoind
+
+Gabriel requires a fully synced Bitcoin Core daemon to be running.
+For testing and development purposes, running Bitcoin Core on _regtest_ is sufficient.
 
 If on bitcoind v28.0, ensure the following flag is set prior to initial block download:  `-blocksxor=0`
 
-#### Environment Variables
+1. Start Bitcoin Core:
+   The following example starts Bitcoin Core in _regtest_ mode.
+
+
+        $ bitcoind \
+          -regtest \
+          -server -daemon \
+          -fallbackfee=0.0002 \
+          -rpcuser=admin -rpcpassword=pass \
+          -rpcallowip=127.0.0.1/0 -rpcbind=127.0.0.1 \
+          -blockfilterindex=1 -peerblockfilters=1 \
+          -zmqpubrawblock=unix:/tmp/zmqpubrawblock.unix \
+          -blocksxor=0
+
+   NOTE: Gabriel includes functionality that consumes block events from Bitcoin Core via its _zmqpubrawblock_ ZeroMQ interface.
+   The example above specifies a Unix domain socket.
+   Alternatively, you could choose to specify a tcp socket and port similar to the following:  `-zmqpubrawblock=tcp://127.0.0.1:29001`
+
+2. Define a shell alias to `bitcoin-cli`, for example:
+
+                $ `alias b-reg=bitcoin-cli -rpcuser=admin -rpcpassword=pass -rpcport=18443`
+
+3. Create (or load) a default wallet, for example:
+
+                $ `b-reg createwallet <wallet-name>`
+
+4. Mine some blocks, for example:
+
+                $ `b-reg generatetoaddress 110 $(b-reg getnewaddress)`
+
+
+##### 2.1.2.3. SQLite client
+
+Gabriel persists P2PK utxo analysis to a local SQLite database.
+If you would like to view that data, you'll want to download and install the  [SQLite client](https://sqlite.org/download.html) for your operating system.
 
-### Clone code
+### 2.2. Clone
+
+You'll need the Gabriel source code:
 
 ```
 $ git clone https://github.com/SurmountSystems/gabriel.git
 $ git checkout HB/gabriel-v2
 ```
 
-### Build
+### 2.3. Build
 
 * execute:
 
         $ cargo build
 
-* view gabriel command line options:
+* view Gabriel's command line options:
 
 
         $ ./target/debug/gabriel
 
-### Execute tests
+### 2.4. Execute tests
 
 ```
 $ cargo test
 ```
 
-### Run Gabriel
+## 3. Run Gabriel
+
+### 3.1. analyze all block data files
+
+Gabriel can be used to identify P2PK utxos across all transactions.
+
+Execute the following if analyzing the entire (previously downloaded) Bitcoin blockchain:
+
+        $ export BITCOIND_DATA_DIR=/path/to/bitcoind/data/dir
+        $ ./target/debug/gabriel index \
+            --input $BITCOIND_DATA_DIR/blocks \
+            --output /tmp/gabriel-testnet4.csv
+
+### 3.2. analyze single block data file
 
-* execute indexer on a specific bitcoin block data file :
+Alternatively, you can have (likely for testing purposes) Gabriel analyze a single Bitcoin Core block data file.
+
+Execute as follows:
 
         $ export BITCOIND_DATA_DIR=/path/to/bitcoind/data/dir
         $ export BITCOIND_BLOCK_DATA_FILE=xxx.dat
+        $ export SQLITE_ABSOLUTE_PATH=/tmp/gabriel_p2pk.db
 
         $ ./target/debug/gabriel block-file-eval \
             -b $BITCOIND_DATA_DIR/blocks/$BITCOIND_BLOCK_DATA_FILE \
             -o /tmp/$BITCOIND_BLOCK_DATA_FILE.csv
 
+#### 3.2.1. Optional:  debug via VSCode:
 
-* execute indexer across all bitcoin block data files :
+Modify the following as appropriate and add to your vscode `launch.json`:
+
+        {
+          "version": "0.2.0",
+          "configurations": [
+            {
+                "type": "lldb",
+                "request": "launch",
+                "name": "gabriel local: 'block-file-eval'",
+                "args": ["block-file-eval", "-b=/tmp/<changeme>.dat", "-o=/tmp/<changeme>.dat.csv"],
+                "cwd": "${workspaceFolder}",
+                "program": "./target/debug/gabriel",
+                "sourceLanguages": ["rust"]
+            }
+          ]
+        }
 
-        $ export BITCOIND_DATA_DIR=/path/to/bitcoind/data/dir
-        $ ./target/debug/gabriel index \
-            --input $BITCOIND_DATA_DIR/blocks \
-            --output /tmp/gabriel-testnet4.csv
+### 3.3. consume and analyze new raw block events
+
+After identifying P2PK utxos from an Initial Block Download (IBD), Gabriel can asynchronously consume new block events as generated by your Bitcoin Core node.
+
+Execute as follows:
+```
+$ export SQLITE_ABSOLUTE_PATH=/tmp/gabriel_p2pk.db
+
+$ ./target/debug/gabriel block-async-eval \
+        --zmqpubrawblock-socket-url ipc:///tmp/zmqpubrawblock.unix \
+        --output /tmp/async_blocks.txt
+```
+
+NOTE: The following example configures Gabriel to consume block events using the same ZeroMQ Unix domain socket that Bitcoin Core was previously configured to produce to.
+If your Bitcoin Core daemon is configured to use TCP for its ZeroMQ interfaces, then you will want Gabriel to use a TCP consumer as well:
+
+```
+--zmqpubrawblock-socket-url=tcp://127.0.0.1:29001
+```
+
+
+#### 3.3.1. Fund a tx w/ a P2PK output on reg-test
+
+If interested in testing Gabriel's ability to consume and process a block with a P2PK utxo, you can use the following in a new terminal:
+
+1. Get extended private key from bitcoind:\
+
+   NOTE:  for the following command, you'll already need to have unlocked your wallet via the bitcoin cli.
+
+        $ XPRV=$( b-reg gethdkeys '{"active_only":true, "private":true}' \
+            | jq -r .[].xprv ) && echo $XPRV
+
+2. Create a tx w/ P2PK output:
+
+        $ export URL=http://127.0.0.1:18443 \
+            && export COOKIE=/path/to/bitcoind/datadir/regtest/.cookie
+
+        $ SIGNED_P2PK_RAW_TX=$( ./target/debug/gabriel \
+                generate-p2pk-tx \
+                -e $XPRV ) \
+                && echo $SIGNED_P2PK_RAW_TX
+
+3. View decoded tx:
+
+        $ b-reg decoderawtransaction $SIGNED_P2PK_RAW_TX
 
-#### Debug in VSCode:
+4.  Send tx:
+
+        $ b-reg sendrawtransaction $SIGNED_P2PK_RAW_TX
+
+#### 3.3.2. Generate block:
+
+        $ b-reg -generate 1
+
+        NOTE:  You should now see a new record in Gabriel's output file indicating the new P2PK utxo.
+
+
+#### 3.3.3. Optional: dDebug in VSCode:
 
 Add and edit the following to $PROJECT_HOME/.vscode/launch.json:
 
-`````
-{
-    "version": "0.2.0",
-    "configurations": [
         {
-            "type": "lldb",
-            "request": "launch",
-            "name": "Debug gabriel local: 'block-file-eval'",
-            "args": ["block-file-eval", "-b=/u04/bitcoin/datadir/blocks/blk00000.dat", "-o=/tmp/blk00000.dat.csv"],
-            "cwd": "${workspaceFolder}",
-            "program": "./target/debug/gabriel",
-            "sourceLanguages": ["rust"]
+          "version": "0.2.0",
+          "configurations": [
+            {
+                "type": "lldb",
+                "request": "launch",
+                "name": "gabriel local: 'generate-p2pk-trnx'",
+                "args": ["generate-p2pk-trnx", "-e=$XPRV-CHANGEME"],
+                "cwd": "${workspaceFolder}",
+                "program": "./target/debug/gabriel",
+                "sourceLanguages": ["rust"]
+            }
+          ]
         }
-    ]
-}
-`````
+
+## 4. Inspect P2PK Analysis data
+Gabriel will persist analysis of P2PK utxos in a SQLite database.
+
+The path of the SQLite database is the value of the SQLITE_ABSOLUTE_PATH environment variable.
+
+At the command line, you can inspect the data in SQLite database similar to the following:
+
+```
+$ sqlite3 $SQLITE_ABSOLUTE_PATH
+
+# list tables;
+sqlite> .tables
+
+# view the schema of the   p2pk_utxo_block_aggregates table:
+sqlite> .schema p2pk_utxo_block_aggregates
+
+# identify number of records in p2pk_utxo_block_aggregates table
+sqlite> select count(block_height) from p2pk_utxo_block_aggregates;
+
+# delete all records
+sqlite> delete from p2pk_utxo_block_aggregates;
+
+# quit sqlite command line:  press  <ctrl> d
+
+```

src/bitcoind_rpc.rs

@@ -0,0 +1,79 @@
+use std::env;
+
+use anyhow::{anyhow, Result};
+use bitcoin::{Address, Amount, BlockHash};
+
+use bitcoin::hashes::sha256d::Hash;
+use bitcoincore_rpc::json::{self, GetAddressInfoResult, ListUnspentResultEntry};
+use bitcoincore_rpc::{Auth, Client, RpcApi};
+
+/// Represents a connection to a Bitcoin Core RPC interface
+#[derive(Debug)]
+pub struct BitcoindRpcInfo {
+    rpc_client: Client,
+}
+
+impl BitcoindRpcInfo {
+    pub fn new() -> Result<Self> {
+        let url = env::var("URL").map_err(|e| anyhow!("Missing URL environment variable: {}", e))?;
+
+        let auth = match env::var("COOKIE") {
+            Ok(cookiefile) => Auth::CookieFile(cookiefile.into()),
+            Err(_) => {
+                let user = env::var("USER")
+                    .map_err(|e| anyhow!("Missing USER environment variable: {}", e))?;
+                let pass = env::var("PASS")
+                    .map_err(|e| anyhow!("Missing PASS environment variable: {}", e))?;
+                Auth::UserPass(user, pass)
+            }
+        };
+
+        Client::new(&url, auth)
+            .map(|rpc_client| BitcoindRpcInfo { rpc_client })
+            .map_err(|e| anyhow!("Failed to create RPC client: {}", e))
+    }
+
+    pub fn get_bitcoind_info_for_test_p2pk(
+        &self,
+        output_amount_btc: f64,
+    ) -> Result<(ListUnspentResultEntry, GetAddressInfoResult, Address, Amount)> {
+        // Get network relay fee
+        let network_relay_fee = self.rpc_client.get_network_info()
+            .map_err(|e| anyhow!("Failed to get network info: {}", e))?
+            .relay_fee;
+        let output_tx_total = network_relay_fee.to_btc() + output_amount_btc;
+
+        // Find suitable UTXO
+        let unspent_vec = self.rpc_client
+            .list_unspent(Some(3), None, None, None, None)
+            .map_err(|e| anyhow!("Failed to list unspent transactions: {}", e))?;
+
+        let unspent_tx = unspent_vec
+            .into_iter()
+            .find(|utxo| utxo.amount.to_btc() > output_tx_total)
+            .ok_or_else(|| anyhow!("No unspent txs have sufficient funds: {}", output_tx_total))?;
+
+        // Get input UTXO address info
+        let input_utxo_address = unspent_tx.address.clone()
+            .ok_or_else(|| anyhow!("UTXO has no address"))?
+            .assume_checked();
+
+        let input_utxo_address_info = self.rpc_client
+            .get_address_info(&input_utxo_address)
+            .map_err(|e| anyhow!("Failed to get address info: {}", e))?;
+
+        // Get change address
+        let change_addr = self.rpc_client
+            .get_raw_change_address(Some(json::AddressType::Bech32))
+            .map_err(|e| anyhow!("Failed to get change address: {}", e))?
+            .assume_checked();
+
+        Ok((unspent_tx, input_utxo_address_info, change_addr, network_relay_fee))
+    }
+
+    pub fn get_block_height(&self, sha256d_hash: &Hash) -> Result<usize> {
+        let hash = BlockHash::from_raw_hash(*sha256d_hash);
+        let block_header = self.rpc_client.get_block_header_info(&hash)?;
+        Ok(block_header.height)
+    }
+}