aiomegfile - Asyncio implementation of megfile

• Docs: http://megvii-research.github.io/aiomegfile

aiomegfile is asyncio implementation of megfile.

Support Protocols

• fs(local filesystem)
• s3

Support more in the future.

Installation

pip3 install aiomegfile

Quick Start

Smart methods and SmartPath are the same as megfile, but all methods are async. You can use aiomegfile to do file operations in an async way.

Functional Interface Example

import asyncio
from aiomegfile import smart_open

async def main():
    async with smart_open('s3://bucket/key', 'r') as f:
        content = await f.read()
        print(content)

if __name__ == '__main__':
    asyncio.run(main())

SmartPath Interface

SmartPath has a similar interface with pathlib.Path, but async.

import asyncio
from aiomegfile import SmartPath

async def main():
    path = SmartPath('s3://bucket/key')
    content = await path.read_text()
    print(content)

if __name__ == '__main__':
    asyncio.run(main())

Quick Start

Smart methods and SmartPath are the same as megfile, but all methods are async. You can use aiomegfile to do file operations in an async way.

Functional Interface Example

import asyncio
from aiomegfile import smart_open

async def main():
    async with smart_open('s3://bucket/key', 'r') as f:
        content = await f.read()
        print(content)

if __name__ == '__main__':
    asyncio.run(main())

SmartPath Interface

SmartPath has a similar interface with pathlib.Path, but async.

import asyncio
from aiomegfile import SmartPath

async def main():
    path = SmartPath('s3://bucket/key')
    content = await path.read_text()
    print(content)

if __name__ == '__main__':
    asyncio.run(main())

CLI

Install the CLI extras:

pip3 install "aiomegfile[cli]"

Example usage:

amf ls ./data
amf ls s3://bucket/prefix -l
amf cp -r ./data s3://bucket/backup
amf sync ./data s3://bucket/backup --progress-bar

For a wider protocol matrix and advanced configuration, see the megfile docs: http://megvii-research.github.io/megfile

How to Contribute

• We welcome everyone to contribute code to the aiomegfile project, but the contributed code needs to meet the following conditions as much as possible: You can submit code even if the code doesn't meet conditions. The project members will evaluate and assist you in making code changes

  • Code format: Your code needs to pass code format check. aiomegfile uses ruff as lint tool
  • Static check: Your code needs complete type hint. aiomegfile uses pytype as static check tool. If pytype failed in static check, use # pytype: disable=XXX to disable the error and please tell us why you disable it.
  • Test: Your code needs complete unit test coverage. aiomegfile uses pyfakefs and moto as local file system and s3 virtual environment in unit tests. The newly added code should have a complete unit test to ensure the correctness

• You can help to improve aiomegfile in many ways:

  • Write code.
  • Improve documentation.
  • Report or investigate bugs and issues.
  • If you find any problem or have any improving suggestion, submit a new issue as well. We will reply as soon as possible and evaluate whether to adopt.
  • Review pull requests.
  • Star aiomegfile repo.
  • Recommend aiomegfile to your friends.
  • Any other form of contribution is welcomed.