feat: Add NTFS MFT Fast Scan - #702
Draft
rankaiyx wants to merge 20 commits into
Draft
Conversation
- Add fsearch_privilege.c/h: Polkit integration for root permission request - Add NTFS config persistence (ntfs_fast_scan_enabled, ntfs_auto_polkit) - Wire up Polkit authorization on NTFS preferences widget toggle - Add Polkit policy file for io.github.cboxdoerfer.FSearch.ntfs-scan - Use org.freedesktop.policykit.exec as action (custom action pending) - Add polkit-gobject-1 as a build dependency
- Track authorization state at application level (privilege_is_authorized) - Skip Polkit request if already authorized (fixes repeated dialogs) - Request authorization on startup with 500ms delay (dialog appears on top) - Initialize widget status from application-level state - Protect against NULL callback in on_authorize_finish
- Added fsearch_filesystem_detect.c/h: parses /etc/fstab and /proc/mounts to detect ntfs-3g mounted partitions (strict fstype matching) - Added partition list population in NTFS preferences widget - Fixed double-free issue with g_ptr_array_new_with_free_func
Add fsearch_database_scan_ntfs.c/h for direct MFT scanning via libntfs-3g.
This bypasses the FUSE layer to index NTFS volumes significantly faster.
Two-phase scan approach:
Phase 1: Walk all MFT records, extract FILE_NAME_ATTR (name, parent_mft,
data_size, last_data_change_time, file_attributes)
Phase 2: Create FsearchDatabaseEntry objects and build parent tree
- Skip NTFS system files (mft_no < 12 and $Extend children)
- Support exclude manager filtering
- Support GCancellable for cancellation
- Add libntfs-3g as a required build dependency
Replace ntfs_mft_record_read() with ntfs_mft_records_read() to fetch 1024 records at a time (~1MB batch). This reduces system call overhead and improves sequential I/O on mechanical HDDs where the MFT is typically stored contiguously. Extract process_mft_record() as a standalone helper function.
Like Everything, index all files during MFT scan and apply exclusions at search time rather than during indexing. This simplifies the scan module and avoids O(n*depth) path building for exclusion checks.
Add NTFS partition configuration persistence to FsearchConfig: - FsearchNtfsPartitionConfig struct (mountpoint, include, monitor) - Save/load NTFS partitions in config file [NTFS] section - fsearch_ntfs_get_partition_config() lookup helper Integrate MFT scan into fsearch_database_index_scan: - Check if include path matches an NTFS partition with include=true - Call db_scan_ntfs() for MFT scanning, fall back to readdir - fsearch_database_index_set_ntfs_partitions() to pass config Add fs_path_is_on_ntfs_mount() to resolve mount point for any path.
Pass FsearchConfig.ntfs_partitions from the application layer down to each FsearchDatabaseIndex, enabling MFT fast scan to actually trigger during index creation and rescan. Changes: - FsearchDatabaseIndexStore: add ntfs_partitions field, pass to each index in start() and create_index_for_rescan() - FsearchDatabase: add ntfs_partitions field, pass to all store creation sites (scan, rescan, rescan_sync, load) - fsearch_database_file_load: accept ntfs_partitions parameter - fsearch_database_rescan_blocking: accept ntfs_partitions parameter - fsearch.c: pass config->ntfs_partitions to database creation - CLI mode (database_scan_in_local_instance): load config and pass ntfs_partitions for command-line database updates
NTFS partitions with include==true are now scanned automatically without needing to be added to the Database include folder list. This matches Everything's behavior: NTFS whole-disk scanning is a first-class feature independent of manual folder selection. Implementation: fsearch_database_index_store_start() now iterates ntfs_partitions after regular includes, creating indices for those with include==true that aren't already in the include list.
The root entry was added to the folders array and referenced by child entries as their parent, but ntfs_scan_context_free() also freed it. This caused use-after-free when the sort comparison function traversed parent pointers. Fix: clear ctx->root_entry before calling ntfs_scan_context_free() so the entry remains valid for the caller's DynamicArray.
…lignment sizeof(MFT_RECORD) is 48 bytes (header only) but actual record size is vol->mft_record_size (typically 1024 bytes). ntfs_mft_records_read writes records at 1024-byte boundaries, but batch[i] indexed at 48-byte strides, causing all records except the first in each batch to read from wrong offsets. Also move system file skip (MFT 0-11) before used_records counting and remove redundant is_ntfs_system_file call.
- Add fsearch_ntfs_partition_configs_equal() to compare NTFS configs - Add ntfs_partitions comparison to config_cmp() so NTFS changes trigger database_config_changed - Add fsearch_database_set_ntfs_partitions() to update database's ntfs_partitions pointer without touching the current store - Add fsearch_database_index_store_set_ntfs_partitions() setter - Update database_scan skip check to also compare ntfs_partitions, preventing skipped scans when only NTFS config changed - Set ntfs_partitions in on_preferences_dialog_response before queuing scan work, so live config changes take effect immediately - Remove per-file debug logs from NTFS MFT scanner
…FS detection - Remove parse_fstab() and fstab dependency - Add verify_ntfs_with_blkid() to confirm NTFS volume type - fs_detect_ntfs_partitions() now scans /proc/mounts for fuseblk entries, then verifies each with blkid TYPE=ntfs - Supports removable NTFS devices not configured in fstab - Update header comments to reflect new detection approach
- Remove NTFS awareness from FsearchDatabaseIndex (pure readdir path) - Delete ntfs_partitions member and set_ntfs_partitions() - Remove MFT scan branch from fsearch_database_index_scan() - Elevate NTFS MFT scan to FsearchDatabaseIndexStore - Store layer iterates NTFS partition configs independently - MFT scan results injected via fsearch_database_index_new_with_content() - Remove "already_included" check (overlap is user responsibility) - Two paths now fully independent: - NTFS MFT: direct block device read, no exclude filtering - readdir: include folder recursion with real-time exclude filtering - Results merged at Store level into unified database
rankaiyx
force-pushed
the
feat/ntfs-fast-scan
branch
from
June 25, 2026 05:21
5d60c1e to
e1d3363
Compare
The index store held a non-owning pointer to ntfs_partitions, which became dangling when the old config was freed (e.g., on preferences dialog close). The database worker thread could then read freed memory in fsearch_ntfs_partition_configs_equal, causing heap-use-after-free. Fix: make the store own its ntfs_partitions reference via g_ptr_array_ref/unref, matching the pattern used for include_manager and exclude_manager.
…tion pkexec clears environment variables for security, including DISPLAY and XAUTHORITY. DISPLAY was restored via --x-display CLI argument, but XAUTHORITY (the X11 authentication cookie file path) was missing, causing 'Authorization required' errors when the elevated process tried to connect to the X server. Add --x-authority CLI argument to pass the original user's XAUTHORITY path to the elevated process, falling back to ~/.Xauthority if the environment variable is not set.
When FSearch is started via sudo or as root directly, the NTFS MFT scan would proceed regardless of the ntfs_fast_scan_enabled config flag, since the privilege module skips elevation checks for root. Add ntfs_fast_scan_enabled to FsearchDatabase and use a helper to return NULL for ntfs_partitions when the feature is disabled, ensuring the store never scans NTFS partitions against the user's configuration.
|
Is this still being worked on? |
Owner
|
I didn't have any time yet to look at the code, but there are some issues with the design which need to be fixed.
|
Author
|
Sorry, I got distracted by other fun stuff lately—but I’m back to working on this now. About “This needs to work with Wayland compositors too—no X11 dependency”: About “The whole app shouldn’t run as root if root is only needed to read one file”:
Still, if you’d rather avoid giving the whole app root access, we can switch to the helper approach—or find a better middle ground. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR adds NTFS MFT (Master File Table) fast scan support to fsearch. It reads NTFS file systems directly via libntfs-3g API, bypassing slow readdir traversal through FUSE layer.
Goal: Match Everything's indexing speed on Linux. Everything achieves instant search on Windows by reading NTFS MFT directly. This PR brings the same approach to fsearch.
What This Does
NTFS MFT Fast Scan
ntfs_mount()andntfs_mft_record_read()from libntfs-3g/dev/sda1)Partition Detection
Function
fs_detect_ntfs_partitions()detects NTFS partitions by:/proc/mountsforfuseblkentries (ntfs-3g appears as fuseblk)blkid -s TYPEto confirm it is NTFS volumeAlso provides
fs_path_is_on_ntfs_mount()to check if a given path is on an NTFS mount point.UI Changes
New NTFS tab in Preferences → Search → Database Preferences:
ntfs_status_libntfsandntfs_status_rootlabelsPrivilege Architecture
gtk_init():privilege_request_if_needed()checks config for NTFS fast scan settingexeclp("pkexec", exe_path, "--privileged", "--x-display", DISPLAY, NULL)--privilegedflag prevents infinite loop--x-displaypasses DISPLAY variable through pkexecprivilege_restore_xdg_paths(): After elevation, detects original user viaPKEXEC_UID/SUDO_UID, restores XDG_CONFIG_HOME/XDG_DATA_HOME to original user's pathsstrip_privileged_flag()inmain.c: Removes--privilegedand--x-displayfrom argv so GTK does not complainprivilege_get_status_text()):Configuration
Files Changed
New Files (7)
src/fsearch_privilege.c/h— Privilege module (pkexec re-exec + XDG path restore + status text)src/fsearch_filesystem_detect.c/h— File system type detectionsrc/fsearch_database_scan_ntfs.c/h— NTFS MFT scanner (ported from everything-demo)src/fsearch_database_preferences_widget.ui— NTFS tab UI (187 lines added)Modified Files (17)
src/main.c— Addedprivilege_request_if_needed(),privilege_restore_xdg_paths(),strip_privileged_flag()src/fsearch_config.c/h— Addedntfs_fast_scan_enabled,FsearchNtfsPartitionConfig, partition config load/save/compare functionssrc/fsearch_database.c/h— Added NTFS partition config sync, scan skip checksrc/fsearch_database_index.c— Addedntfs_partitionsfield, MFT scan integration infsearch_database_index_scan()src/fsearch_database_index_store.c/h— Addedntfs_partitionsthreading, settersrc/fsearch_database_preferences_widget.c/h— Added NTFS page init, partition list, config get/set, status updatesrc/fsearch_database_file.c/h— Added NTFS partition save/loadsrc/fsearch_database.h— Added NTFS partition config comparison inconfig_cmp()src/fsearch_preferences_dialog.c— Added NTFS config sync on applysrc/meson.build— Addeddependency('libntfs-3g', required: true)meson.build— Project-level changes.gitignore— Added local project filesNot Yet Done
1. Periodic Full MFT Scan
Scheduled full MFT scan to correct index drift over time. Currently only scans at startup or when config changes.
2. NTFS Change Monitoring
Use root privilege to set up fanotify/inotify watch tree directly on block device or mount point, for real-time NTFS change detection. Currently relies on FUSE layer monitoring only.
3. libntfs-3g Runtime Dynamic Loading (
dlopen)Currently meson requires libntfs-3g at compile time (
required: true). Plan to change todlopenruntime loading:required: falseGModuleto load libntfs-3g at runtimeTesting