Implement Bitswap in py-libp2p

[sumanjeet0012]

What was wrong?

• py-libp2p lacked Bitswap protocol support for IPFS block exchange
• Could not participate in decentralized file sharing networks
• Missing feature parity with Go, Rust, and JavaScript implementations
• Limited use cases for distributed storage and content delivery

How was it fixed?

Core Module Implementation

Added new libp2p.bitswap package:

Protocol Layer:
• client.py - Bitswap protocol client with block request/response
• block_store.py - Abstract storage interface + in-memory implementation
• bitswap_message.py - Message serialization/deserialization
• exceptions.py - Custom exception types

Merkle DAG:
• dag.py - File chunking and DAG construction
• dag_pb.py - DAG-PB encoding/decoding
• unixfs.py - UnixFS metadata support
• utils.py - CID computation utilities

Protocol Buffers:
• pb/bitswap.proto, dag_pb.proto, unixfs.proto - Wire format definitions
• Generated .py and .pyi files with mypy stubs

Key Features

• Bitswap Protocol: Bidirectional block exchange, want-list management, priority requests, block deduplication, multistream negotiation
• Merkle DAG: File chunking, DAG construction/traversal, CID computation, raw + DAG-PB encoding
• UnixFS: File metadata, directory structure, IPFS UnixFS v1 compatible
• Block Storage: Pluggable BlockStore interface, in-memory implementation, async/await, type-safe CIDs
• CLI Tool: Provider/client modes for file sharing with examples

Protocol Compliance

• Bitswap: Compatible with go-ipfs and js-ipfs
• DAG-PB: IPLD Merkle DAG encoding
• UnixFS v1: IPFS file system abstraction
• Multicodec/Multihash: Standard content addressing
• Protobuf definitions integrated in Makefile with mypy stubs

Examples

CLI Usage:

# Provider: Share a file
python examples/bitswap/bitswap.py --mode provider --file <file_path>

# Client: Download by CID
python examples/bitswap/bitswap.py --mode client --provider <multiaddr> --cid <root_cid> --output <output_path>

Benefits

• Feature Parity: Matches Go/Rust/JavaScript implementations, enables IPFS network participation
• Developer Experience: Type-safe API, async/await, documented examples, modular design
• Performance: Efficient chunking, minimal memory footprint, proper async cleanup
• Interoperability: Compatible with go-ipfs/js-ipfs, standard multiaddr/CID addressing

To-Do

• Clean up commit history (rebase/squash if needed)
• Add performance benchmarks (optional)
• Consider adding more storage backend implementations (optional)

Cute Animal Picture

[sumanjeet0012]

@seetadev @pacrob Ready for Review.

[pacrob]

It doesn't appear that the tests here are being run in CI. If you look in tox.ini:

[testenv]
usedevelop=True
commands=
    core: pytest -n auto --timeout=1200 {posargs:tests/core}
    interop: pytest -n auto --timeout=1200 {posargs:tests/interop}
    docs: make check-docs-ci
    demos: pytest -n auto --timeout=1200 {posargs:tests/core/examples/test_examples.py}

only files in tests/core and tests/interop are being run. We should probably move these to tests/core/bitswap.

When I do run the tests directly, I'm getting a number of them failing - at least one in test_chunker.py, one in test_client.py, and a few in test_dag.py. Part of it may be that MerkleDag.fetch_file returns a tuple of (bytes, str | None), while some tests only expect bytes.

[sumanjeet0012]

@pacrob Thanks for pointing out the test files issue,
I have moved test files to tests/core/bitswap and fixed failing tests.

[seetadev]

@sumanjeet0012 : Thanks Sumanjeet. Appreciate your efforts.

Please create a discussion page reflecting the details on the implementation of the PR. Also, please add the nice screencasts that you had recorded on bitswap implementation.

We should also add an example for this PR in the examples section. This would enable other developers to start using bitswap in your projects.

I'll ask @lla-dane and @Hany-Almnaem to use bitswap in their projects too -

1. https://github.com/lla-dane/P2P-Federated-Learning
2. https://github.com/Hany-Almnaem/libp2p-privacy-poc

[seetadev]

@sumanjeet0012 : Thank you for resolving the issue shared by @pacrob. This PR is indeed ready for final review + merge. Reviewing it and will share feedback soon.