Duplicate Images Finder / 重复图像查找工具

Based on EfficientIR.

A tool to find duplicate image pairs or the most similar images of target image in your file system. A optimized version of EfficientNet B2 model is used to generate the eigenvalues of images behind the scenes.

Why Write This?

Like a hamster, I love browsing sites like Pixiv, X, etc., liking and saving the pictures that give me aesthetic pleasure. The problem is that some pictures are uploaded multiple times, and I only want to keep one copy.

There are also multiple softwares that already meet my needs like EfficientIR. As a front-end engineer, I want to re-implement it using modern front-end technologies while improving my native coding skills.

That's why I write this software, hope it helps save your problems, too! Pull requests are always welcomed!

Support Platforms

• Windows

Usage

Download and install the tool.

1. Firstly, in Indexes page, add paths that contain images. Subdirectories will be included.
2. Secondly, in Indexes page, click button to generate the eigenvalues of images. This might take a long time due to the number of images and the performance of your computer.
3. Finally, in Search or Search Target page, start a search progress using generated eigenvalues.
   • In Search page, you can search duplicate image pairs.
   • In Search Target page, you can search the most similar images of selected image.

Performance

Duplicate Images Finder supports indexing with multiple work processes, default to 4.

CPU	Image Size	Time Consuming	Branch Tag
Inter i5-12600KF	approximately 50,000 images (≈ 170GB)	90min (with 1 work process)	dev
AMD Ryzen 5 9600X	approximately 100,000 images (≈ 380GB)	55min (with 4 work processes)	v1.3.0

Develop

You need to build the EfficientIR binary before running the frontend. The binary is located at EfficientIR/dist/EfficientIR.exe after building.

Backend

Python environment is required. Binary is built successfully on python >= 3.12. For Windows, Microsoft Visual C++ >= 14.0 is required.

# Initialize submodule
cd EfficientIR
git submodule update --init
# Create virtual environment and install dependencies
python -m venv .venv
.venv/Scripts/Activate.ps1 # For Linux or macOS, use `source .venv/bin/activate`
pip install -r requirements.txt
# Build binary
pyinstaller build.spec

Frontend

If you need to run index or search actions, prepare backend binary first.

Install Dependencies

yarn

Development

yarn dev

Lint

yarn lint

# fix resolvable lint errors
yarn lint --fix

Production

yarn build