Skip to content

Support for time-based integer IDs (e.g. snowflake) using encode/decode #729

@calebj

Description

@calebj

Creating a new issue from #528 (comment).

Currently the only supported ways to use IDs/integers in a control column are as a timestamp directly representing seconds, ms, us or ns since the UNIX epoch; or as a generic, non-timestamp ID range. #683 adds functionality to use ranges of UUID and text values via conversion functions. I want to add support for integers as well.

The primary example is "snowflake" IDs, used by X, Discord, Instagram, Mastodon, and others. I brought them up in #388 and the resulting changes let me use snowflake IDs as ordinary integers, with offsets for interval and retention. This works well enough, but leads to gaps when no data is being added for a while since premake only looks at the current maximum value.

For background, snowflake values fit in 63 bits (for signed int64) and encode a timestamp plus a combination of worker and sequence information. Since each implementation uses a different bit range and epoch for the timestamp, partman won't be able to cover all possibilities with a single setting for p_epoch like 'snowflake'.

I propose a new option for p_epoch of 'func', which indicates that partman should use the encoder/decoder functions for IDs the same way it does for UUIDs and text. This should be a straightforward change, since most of the work has already been done in #683. Also, the parameter documentation for encoder/decoder mentions snowflakes, but those are only allowed if they are stored as a string rather than a bigint.

Example Discord ID encoder and decoder:
-- Discord snowflakes use a 41 bit timestamp and 22 bit worker/process/sequence.
-- The timestamp is the number of milliseconds since the first second of 2015 UTC (1420070400s since epoch)
CREATE FUNCTION discord_snowflake_to_timestamp(id bigint)
    RETURNS TIMESTAMPTZ LANGUAGE SQL IMMUTABLE PARALLEL SAFE AS
    $$SELECT TO_TIMESTAMP((id >> 22)/1000.0 + 1420070400)$$;

CREATE FUNCTION timestamp_to_discord_snowflake(ts TIMESTAMPTZ)
    RETURNS BIGINT LANGUAGE SQL IMMUTABLE PARALLEL SAFE AS
    $$SELECT CAST((EXTRACT(epoch FROM ts) - 1420070400)*1000 AS BIGINT) << 22$$;

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions