Skip to content

Enable MySQL "LOAD DATA LOCAL" with url option#4970

Merged
weiznich merged 1 commit intodiesel-rs:mainfrom
TheLQ:main
Mar 20, 2026
Merged

Enable MySQL "LOAD DATA LOCAL" with url option#4970
weiznich merged 1 commit intodiesel-rs:mainfrom
TheLQ:main

Conversation

@TheLQ
Copy link
Contributor

@TheLQ TheLQ commented Feb 11, 2026

Enable LOAD DATA LOCAL in batch_execute. mysqlclient handles all the IO. I'm inserting 30 million rows significantly faster now.

With native Diesel support, the mysql and postgres versions get complicated quickly. Also LOAD DATA isn't supported in transactions. This option is enough for me to do it when rarely needed.

Example

fn main() {
    let url = "mysql://u:p@localhost/db?local_infile";
    let conn = &mut MysqlConnection::establish(url).unwrap();
    conn.batch_execute(
        "CREATE TEMPORARY TABLE fast_table (\
        id INT NOT NULL PRIMARY KEY,\
        value VARCHAR(20) NOT NULL\
        )",
    )
        .unwrap();

    let load_path = Path::new("mysql-rows.dat");
    write_rows_file(&load_path);
    load_rows_file(conn, &load_path);
}

const ROW_SEP: u8 = 0x1e;
const COL_SEP: u8 = 0x1f;
fn write_rows_file(path: &Path) -> () {
    let mut out_file = File::create(path).unwrap();
    for row in 0..10 {
        let something = "something";
        write!(out_file, "{row}{COL_SEP}{something}{ROW_SEP}").unwrap();
    }
}

fn load_rows_file(conn: &mut MysqlConnection, path: &Path) {
    conn.batch_execute(&format!(
        "LOAD DATA LOCAL INFILE '{}' \
        INTO TABLE `fast_table` \
        FIELDS TERMINATED BY '{COL_SEP}' \
        LINES TERMINATED BY '{ROW_SEP}' \
        (id, value)",
        path.display()
    ))
        .unwrap()
}

@weiznich weiznich requested a review from a team February 11, 2026 12:08
Copy link
Member

@JohnTitor JohnTitor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I'd tweak ConnectionOptions and add tests.

// this is not present in the database_url, using a default value
let client_flags = CapabilityFlags::CLIENT_FOUND_ROWS;
let mut client_flags = CapabilityFlags::CLIENT_FOUND_ROWS;
if query_pairs.contains_key("local_infile") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

local_infile should take binary value so we need handle it as well: https://dev.mysql.com/doc/refman/8.4/en/mysql-command-options.html#option_mysql_local-infile

@TheLQ TheLQ force-pushed the main branch 3 times, most recently from cf7a150 to a82c10d Compare February 12, 2026 22:46
@TheLQ
Copy link
Contributor Author

TheLQ commented Feb 12, 2026

Initially used mysqlclient's default compiled flag (usually disabled) or explicit enable.

To have explicit enable and disable we need to use mysql_option instead.
Docs for MYSQL_OPT_LOCAL_INFILE https://dev.mysql.com/doc/c-api/9.5/en/mysql-options.html

Added parsing unit test. Example above was successful too.

@TheLQ
Copy link
Contributor Author

TheLQ commented Mar 8, 2026

Rebased after #4972 so tests pass. Also fixed the doc comment.

@weiznich weiznich added this pull request to the merge queue Mar 20, 2026
Merged via the queue into diesel-rs:main with commit f34d3b6 Mar 20, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants