3.11

• New sqlite-utils memory data.csv --schema option, for outputting the schema of the in-memory database generated from one or more files. See --schema, --dump and --save. (#288)
• Added installation instructions. (#286)

3.10

This release introduces the sqlite-utils memory command, which can be used to load CSV or JSON data into a temporary in-memory database and run SQL queries (including joins across multiple files) directly against that data.

Also new: sqlite-utils insert --detect-types, sqlite-utils dump, table.use_rowid plus some smaller fixes.

sqlite-utils memory

This example of sqlite-utils memory retrieves information about the all of the repositories in the Dogsheep organization on GitHub using this JSON API, sorts them by their number of stars and outputs a table of the top five (using -t):

$ curl -s 'https://api.github.com/users/dogsheep/repos'\
  | sqlite-utils memory - '
      select full_name, forks_count, stargazers_count
      from stdin order by stargazers_count desc limit 5
    ' -t
full_name                            forks_count    stargazers_count
---------------------------------  -------------  ------------------
dogsheep/twitter-to-sqlite                    12                 225
dogsheep/github-to-sqlite                     14                 139
dogsheep/dogsheep-photos                       5                 116
dogsheep/dogsheep.github.io                    7                  90
dogsheep/healthkit-to-sqlite                   4                  85

The tool works against files on disk as well. This example joins data from two CSV files:

$ cat creatures.csv
species_id,name
1,Cleo
2,Bants
2,Dori
2,Azi
$ cat species.csv
id,species_name
1,Dog
2,Chicken
$ sqlite-utils memory species.csv creatures.csv '
  select * from creatures join species on creatures.species_id = species.id
'
[{"species_id": 1, "name": "Cleo", "id": 1, "species_name": "Dog"},
 {"species_id": 2, "name": "Bants", "id": 2, "species_name": "Chicken"},
 {"species_id": 2, "name": "Dori", "id": 2, "species_name": "Chicken"},
 {"species_id": 2, "name": "Azi", "id": 2, "species_name": "Chicken"}]

Here the species.csv file becomes the species table, the creatures.csv file becomes the creatures table and the output is JSON, the default output format.

You can also use the --attach option to attach existing SQLite database files to the in-memory database, in order to join data from CSV or JSON directly against your existing tables.

Full documentation of this new feature is available in Querying data directly using an in-memory database. (#272)

sqlite-utils insert --detect-types

The sqlite-utils insert command can be used to insert data from JSON, CSV or TSV files into a SQLite database file. The new --detect-types option (shortcut -d), when used in conjunction with a CSV or TSV import, will automatically detect if columns in the file are integers or floating point numbers as opposed to treating everything as a text column and create the new table with the corresponding schema. See Inserting CSV or TSV data for details. (#282)

Other changes

• Bug fix: table.transform(), when run against a table without explicit primary keys, would incorrectly create a new version of the table with an explicit primary key column called rowid. (#284)
• New table.use_rowid introspection property, see .use_rowid. (#285)
• The new sqlite-utils dump file.db command outputs a SQL dump that can be used to recreate a database. (#274)
• -h now works as a shortcut for --help, thanks Loren McIntyre. (#276)
• Now using pytest-cov and Codecov to track test coverage - currently at 96%. (#275)
• SQL errors that occur when using sqlite-utils query are now displayed as CLI errors.

3.9.1

• Fixed bug when using table.upsert_all() to create a table with only a single column that is treated as the primary key. (#271)

3.9

• New sqlite-utils schema command showing the full SQL schema for a database, see Showing the schema (CLI). (#268)
• db.schema introspection property exposing the same feature to the Python library, see Showing the schema (Python library).

3.8

• New sqlite-utils indexes command to list indexes in a database, see Listing indexes. (#263)
• table.xindexes introspection property returning more details about that table's indexes, see .xindexes. (#261)

3.7

• New table.pks_and_rows_where() method returning (primary_key, row_dictionary) tuples - see Listing rows with their primary keys. (#240)
• Fixed bug with table.add_foreign_key() against columns containing spaces. (#238)
• table_or_view.drop(ignore=True) option for avoiding errors if the table or view does not exist. (#237)
• sqlite-utils drop-view --ignore and sqlite-utils drop-table --ignore options. (#237)
• Fixed a bug with inserts of nested JSON containing non-ascii strings - thanks, Dylan Wu. (#257)
• Suggest --alter if an error occurs caused by a missing column. (#259)
• Support creating indexes with columns in descending order, see API documentation and CLI documentation. (#260)
• Correctly handle CSV files that start with a UTF-8 BOM. (#250)

3.6

This release adds the ability to execute queries joining data from more than one database file - similar to the cross database querying feature introduced in Datasette 0.55.

• The db.attach(alias, filepath) Python method can be used to attach extra databases to the same connection, see db.attach() in the Python API documentation. (#113)
• The --attach option attaches extra aliased databases to run SQL queries against directly on the command-line, see attaching additional databases in the CLI documentation. (#236)

3.5

• sqlite-utils insert --sniff option for detecting the delimiter and quote character used by a CSV file, see Alternative delimiters and quote characters. (#230)
• The table.rows_where(), table.search() and table.search_sql() methods all now take optional offset= and limit= arguments. (#231)
• New --no-headers option for sqlite-utils insert --csv to handle CSV files that are missing the header row, see CSV files without a header row. (#228)
• Fixed bug where inserting data with extra columns in subsequent chunks would throw an error. Thanks @nieuwenhoven for the fix. (#234)
• Fixed bug importing CSV files with columns containing more than 128KB of data. (#229)
• Test suite now runs in CI against Ubuntu, macOS and Windows. Thanks @nieuwenhoven for the Windows test fixes. (#232)

3.4.1

• Fixed a code import bug that slipped in to 3.4. (#226)

3.4

• sqlite-utils insert --csv now accepts optional --delimiter and --quotechar options. See Alternative delimiters and quote characters. (#223)