Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .github/workflows/codspeed.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: CodSpeed

on:
push:
branches:
- "master"
pull_request:
workflow_dispatch:

permissions:
contents: read
id-token: write

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
benchmarks:
name: Run benchmarks
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v6
with:
persist-credentials: false

- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: "3.12"
cache: pip
cache-dependency-path: |
pyproject.toml
poetry.lock

- name: Load cached Poetry installation
id: cached-poetry
uses: actions/cache@v5
with:
path: ~/.local
key: poetry-${{ runner.os }}-3.12-0

- name: Install Poetry
if: steps.cached-poetry.outputs.cache-hit != 'true'
uses: snok/install-poetry@v1

- name: Load cached venv
id: cached-poetry-dependencies
uses: actions/cache@v5
with:
path: .venv
key: venv-${{ runner.os }}-3.12-${{ hashFiles('**/poetry.lock') }}-codspeed-0

- name: Install Dependencies
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
run: poetry install --no-interaction
shell: bash

- name: Upgrade Setuptools
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
run: poetry run python -m pip install -U "setuptools<81"
shell: bash

- name: Setup brownie networks
run: poetry run brownie networks modify mainnet host=${{ secrets.TENDERLY_PROVIDER }}
shell: bash
continue-on-error: true

- name: Run benchmarks
uses: CodSpeedHQ/action@v4
with:
mode: simulation
run: poetry run pytest tests/benchmarks/ --codspeed
env:
PYTEST_NETWORK: mainnet
ETHERSCAN_TOKEN: ${{ secrets.ETHERSCAN_TOKEN }}
MAX_RETRIES: 2
MIN_SLEEP_TIME: 1
MAX_SLEEP_TIME: 3
Binary file modified 8a4082216e5922611c62__mypyc.cp310-win_amd64.pyd
Binary file not shown.
Binary file modified 8a4082216e5922611c62__mypyc.cp311-win_amd64.pyd
Binary file not shown.
Binary file modified 8a4082216e5922611c62__mypyc.cp312-win_amd64.pyd
Binary file not shown.
Binary file modified 8a4082216e5922611c62__mypyc.cp313-win_amd64.pyd
Binary file not shown.
Binary file modified 8a4082216e5922611c62__mypyc.cp314-win_amd64.pyd
Binary file not shown.
183 changes: 92 additions & 91 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,91 +1,92 @@
# Dank Mids

[![PyPI](https://img.shields.io/pypi/v/dank-mids.svg?logo=Python&logoColor=white)](https://pypi.org/project/dank-mids)
[![Monthly Downloads](https://img.shields.io/pypi/dm/dank-mids)](https://pypistats.org/packages/dank-mids)

Dank Mids is a EVM RPC batching library that helps reduce the number of HTTP requests to a node, saving time and resources. It automatically collects eth_call calls into [multicalls](https://github.com/makerdao/multicall#multicall-) and bundles all RPC calls together in [jsonrpc](https://www.jsonrpc.org/specification#batch) [batch](https://geth.ethereum.org/docs/interacting-with-geth/rpc/batch) calls.

##### tl;dr: its fast as fuck.

![image](https://github.com/BobTheBuidler/dank_mids/assets/70677534/3ecb46aa-f33a-41bd-85fb-c6d2433c7154)

The goal of this tool is to reduce the workload on RPC nodes and allow users to make calls to their preferred node more efficiently. This optimization is especially useful for developers writing scripts that perform large-scale blockchain analysis, as it can save development time and resources.

![](https://i.imgur.com/o9FUmAn.jpg)

### Why is Dank so fast?

There are a number of optimizations that went into making Dank the fastest way to pull rpc data to Python.
1. Implemented (mostly) in C.
2. Bypasses the default formatters in [web3.py](https://github.com/ethereum/web3.py)
3. JSON encoding and decoding is handled by [msgspec](https://jcristharif.com/msgspec/). All responses are decoded to specialized [msgspec.Struct](https://jcristharif.com/msgspec/structs.html) objects defined in the [evmspec](https://github.com/BobTheBuidler/evmspec) library.
4. We use my C-compiled [faster-eth-abi](https://github.com/BobTheBuidler/faster-eth-abi/tree/master) and [faster-eth-utils](https://github.com/BobTheBuidler/faster-eth-utils/tree/master) instead of the original python implementations [eth-abi](https://github.com/ethereum/eth-abi) and [eth-utils](https://github.com/ethereum/eth-utils).
5. Responses are decoded on a JIT (just-in-time) basis, meaning individual task cancellation works as expected even when response data is received as part of a larger batch.
6. more stuff I'll write down later...

### Installation

To install Dank Mids, use pip:

`pip install dank-mids`

### Benchmark

We've included a [benchmark script](./examples/benchmark.py) that compares the time it takes to fetch the pool tokens (token0 and token1) for each pool on Sushiswap on Ethereum mainnet. To run it, first install the repo with `poetry install` and then run the benchmark with `brownie run examples/benchmark`.

```
Running 'examples/benchmark.py::main'...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4213/4213 [08:50<00:00, 7.95it/s]
brownie sync end: 2025-04-14 21:21:35.531099
brownie sync took: 0:08:50.212665
brownie 4 threads start: 2025-04-14 21:21:35.548373
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4213/4213 [08:31<00:00, 8.23it/s]
brownie 4 threads end: 2025-04-14 21:30:08.065397
brownie 4 threads took: 0:08:32.517024
brownie 16 threads start: 2025-04-14 21:30:08.086342
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4213/4213 [08:26<00:00, 8.32it/s]
brownie 16 threads end: 2025-04-14 21:38:38.141635
brownie 16 threads took: 0:08:30.055293
dank start: 2025-04-14 21:38:38.161024
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4213/4213 [00:55<00:00, 75.49it/s]
dank end: 2025-04-14 21:39:33.982835
dank took: 0:00:55.821811
```

As you can see, dank_mids allowed us to save 7 minutes and 34 seconds vs brownie with 16 threads. That's an 89% reduction in runtime, or about 9x as fast as brownie!

### Usage with web3.py

The primary function you need to use Dank Mids is `setup_dank_w3_from_sync`. This function takes a sync Web3 instance and wraps it for async use. If using dank_mids with eth-brownie, you can just import the premade dank_web3 object as well

Example usage of Dank Mids with web3py:

```python
from dank_mids.helpers import setup_dank_w3_from_sync
dank_web3 = setup_dank_w3_from_sync(w3)
# OR
from dank_mids import dank_web3

# Then:
random_block = await dank_web3.eth.get_block(123)
```

### Usage with eth-brownie

- [Dank Brownie Example Commented Code](./examples/dank_brownie_example.py)

### Usage with ape

- COMING SOON: Dank Mids will also work with [ape](https://github.com/ApeWorX/ape).

### Testimonials

[Yearn](https://yearn.finance) big brain [Tonkers Kuma](https://github.com/tonkers-kuma) had this to say:

![image](https://user-images.githubusercontent.com/70677534/211255488-e76e641c-a0fe-461c-a4e5-27c45a3fea5b.png)

### Notes

You can also set `DANK_MIDS_DEMO_MODE=True` to see a visual representation of the batching in real time on your console.


# Dank Mids

[![PyPI](https://img.shields.io/pypi/v/dank-mids.svg?logo=Python&logoColor=white)](https://pypi.org/project/dank-mids)
[![Monthly Downloads](https://img.shields.io/pypi/dm/dank-mids)](https://pypistats.org/packages/dank-mids)
[![CodSpeed](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/BobTheBuidler/dank_mids?utm_source=badge)

Dank Mids is a EVM RPC batching library that helps reduce the number of HTTP requests to a node, saving time and resources. It automatically collects eth_call calls into [multicalls](https://github.com/makerdao/multicall#multicall-) and bundles all RPC calls together in [jsonrpc](https://www.jsonrpc.org/specification#batch) [batch](https://geth.ethereum.org/docs/interacting-with-geth/rpc/batch) calls.

##### tl;dr: its fast as fuck.

![image](https://github.com/BobTheBuidler/dank_mids/assets/70677534/3ecb46aa-f33a-41bd-85fb-c6d2433c7154)

The goal of this tool is to reduce the workload on RPC nodes and allow users to make calls to their preferred node more efficiently. This optimization is especially useful for developers writing scripts that perform large-scale blockchain analysis, as it can save development time and resources.

![](https://i.imgur.com/o9FUmAn.jpg)

### Why is Dank so fast?

There are a number of optimizations that went into making Dank the fastest way to pull rpc data to Python.
1. Implemented (mostly) in C.
2. Bypasses the default formatters in [web3.py](https://github.com/ethereum/web3.py)
3. JSON encoding and decoding is handled by [msgspec](https://jcristharif.com/msgspec/). All responses are decoded to specialized [msgspec.Struct](https://jcristharif.com/msgspec/structs.html) objects defined in the [evmspec](https://github.com/BobTheBuidler/evmspec) library.
4. We use my C-compiled [faster-eth-abi](https://github.com/BobTheBuidler/faster-eth-abi/tree/master) and [faster-eth-utils](https://github.com/BobTheBuidler/faster-eth-utils/tree/master) instead of the original python implementations [eth-abi](https://github.com/ethereum/eth-abi) and [eth-utils](https://github.com/ethereum/eth-utils).
5. Responses are decoded on a JIT (just-in-time) basis, meaning individual task cancellation works as expected even when response data is received as part of a larger batch.
6. more stuff I'll write down later...

### Installation

To install Dank Mids, use pip:

`pip install dank-mids`

### Benchmark

We've included a [benchmark script](./examples/benchmark.py) that compares the time it takes to fetch the pool tokens (token0 and token1) for each pool on Sushiswap on Ethereum mainnet. To run it, first install the repo with `poetry install` and then run the benchmark with `brownie run examples/benchmark`.

```
Running 'examples/benchmark.py::main'...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4213/4213 [08:50<00:00, 7.95it/s]
brownie sync end: 2025-04-14 21:21:35.531099
brownie sync took: 0:08:50.212665
brownie 4 threads start: 2025-04-14 21:21:35.548373
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4213/4213 [08:31<00:00, 8.23it/s]
brownie 4 threads end: 2025-04-14 21:30:08.065397
brownie 4 threads took: 0:08:32.517024
brownie 16 threads start: 2025-04-14 21:30:08.086342
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4213/4213 [08:26<00:00, 8.32it/s]
brownie 16 threads end: 2025-04-14 21:38:38.141635
brownie 16 threads took: 0:08:30.055293
dank start: 2025-04-14 21:38:38.161024
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4213/4213 [00:55<00:00, 75.49it/s]
dank end: 2025-04-14 21:39:33.982835
dank took: 0:00:55.821811
```

As you can see, dank_mids allowed us to save 7 minutes and 34 seconds vs brownie with 16 threads. That's an 89% reduction in runtime, or about 9x as fast as brownie!

### Usage with web3.py

The primary function you need to use Dank Mids is `setup_dank_w3_from_sync`. This function takes a sync Web3 instance and wraps it for async use. If using dank_mids with eth-brownie, you can just import the premade dank_web3 object as well

Example usage of Dank Mids with web3py:

```python
from dank_mids.helpers import setup_dank_w3_from_sync
dank_web3 = setup_dank_w3_from_sync(w3)
# OR
from dank_mids import dank_web3

# Then:
random_block = await dank_web3.eth.get_block(123)
```

### Usage with eth-brownie

- [Dank Brownie Example Commented Code](./examples/dank_brownie_example.py)

### Usage with ape

- COMING SOON: Dank Mids will also work with [ape](https://github.com/ApeWorX/ape).

### Testimonials

[Yearn](https://yearn.finance) big brain [Tonkers Kuma](https://github.com/tonkers-kuma) had this to say:

![image](https://user-images.githubusercontent.com/70677534/211255488-e76e641c-a0fe-461c-a4e5-27c45a3fea5b.png)

### Notes

You can also set `DANK_MIDS_DEMO_MODE=True` to see a visual representation of the batching in real time on your console.


Loading