Waiting for PostgreSQL 17 – Support identity columns in partitioned tables

On 16th of January 2024, Peter Eisentraut committed patch:

Support identity columns in partitioned tables
 
Previously, identity columns were disallowed on partitioned tables.
(The reason was mainly that no one had gotten around to working
through all the details to make it work.)  This makes it work now.
 
Some details on the behavior:
 
* A newly created partition inherits identity property
 
  The partitions of a partitioned table are integral part of the
  partitioned table.  A partition inherits identity columns from the
  partitioned table.  An identity column of a partition shares the
  identity space with the corresponding column of the partitioned
  table.  In other words, the same identity column across all
  partitions of a partitioned table share the same identity space.
  This is effected by sharing the same underlying sequence.
 
  When INSERTing directly into a partition, the sequence associated
  with the topmost partitioned table is used to calculate the value of
  the corresponding identity column.
 
  In regular inheritance, identity columns and their properties in a
  child table are independent of those in its parent tables.  A child
  table does not inherit identity columns or their properties
  automatically from the parent.  (This is unchanged.)
 
* Attached partition inherits identity column
 
  A table being attached as a partition inherits the identity property
  from the partitioned table.  This should be fine since we expect
  that the partition table's column has the same type as the
  partitioned table's corresponding column.  If the table being
  attached is a partitioned table, the identity properties are
  propagated down its partition hierarchy.
 
  An identity column in the partitioned table is also marked as NOT
  NULL.  The corresponding column in the partition needs to be marked
  as NOT NULL for the attach to succeed.
 
* Drop identity property when detaching partition
 
  A partition's identity column shares the identity space
  (i.e. underlying sequence) as the corresponding column of the
  partitioned table.  If a partition is detached it can longer share
  the identity space as before.  Hence the identity columns of the
  partition being detached loose their identity property.
 
  When identity of a column of a regular table is dropped it retains
  the NOT NULL constraint that came with the identity property.
  Similarly the columns of the partition being detached retain the NOT
  NULL constraints that came with identity property, even though the
  identity property itself is lost.
 
  The sequence associated with the identity property is linked to the
  partitioned table (and not the partition being detached).  That
  sequence is not dropped as part of detach operation.
 
* Partitions with their own identity columns are not allowed.
 
* The usual ALTER operations (add identity column, add identity
  property to existing column, alter properties of an indentity
  column, drop identity property) are supported for partitioned
  tables.  Changing a column only in a partitioned table or a
  partition is not allowed; the change needs to be applied to the
  whole partition hierarchy.
 
Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://www.postgresql.org/message-id/flat/CAExHW5uOykuTC+C6R1yDSp=o8Q83jr8xJdZxgPkxfZ1Ue5RRGg@mail.gmail.com

Commit message is pretty long and descriptive, so let's just see it in action.

=$ CREATE TABLE some_data (
    id             int8 generated always AS IDENTITY NOT NULL,
    created_at     timestamptz NOT NULL,
    payload        TEXT
) PARTITION BY RANGE ( created_at ) ;
=$ CREATE TABLE some_data_old PARTITION OF some_data FOR VALUES FROM (MINVALUE) TO ('2022-01-01');
=$ CREATE TABLE some_data_y2022 PARTITION OF some_data FOR VALUES FROM ('2022-01-01') TO ('2023-01-01');
=$ CREATE TABLE some_data_y2023 PARTITION OF some_data FOR VALUES FROM ('2023-01-01') TO ('2024-01-01');
=$ CREATE TABLE some_data_y2024 PARTITION OF some_data FOR VALUES FROM ('2024-01-01') TO ('2025-01-01');
=$ CREATE TABLE some_data_y2025 PARTITION OF some_data FOR VALUES FROM ('2025-01-01') TO ('2026-01-01');

Now, I can insert to all of these tables:

=$ INSERT INTO some_data (created_at, payload) SELECT now() - '5 years'::INTERVAL * random(), 'Row #' || i FROM generate_series(1,10) i;
INSERT 0 10
 
=$ SELECT tableoid::regclass, * FROM some_data ORDER BY created_at;
    tableoid     │ id │          created_at           │ payload
─────────────────┼────┼───────────────────────────────┼─────────
 some_data_old   │  82020-12-14 08:40:15.272099+00 │ ROW #8
 some_data_old   │  72021-01-27 12:28:57.147299+00 │ ROW #7
 some_data_old   │  42021-03-27 15:24:49.524899+00 │ ROW #4
 some_data_old   │  22021-10-14 01:33:39.531299+00 │ ROW #2
 some_data_y2022 │ 102022-10-04 06:04:15.905699+00 │ ROW #10
 some_data_y2023 │  62023-01-02 05:19:54.267299+00 │ ROW #6
 some_data_y2023 │  92023-03-14 09:37:02.196899+00 │ ROW #9
 some_data_y2023 │  12023-04-09 14:36:47.566499+00 │ ROW #1
 some_data_y2023 │  32023-10-28 22:47:40.980899+00 │ ROW #3
 some_data_y2023 │  52023-11-14 14:18:36.593699+00 │ ROW #5
(10 ROWS)

As you can see ids are being assigned in order of insertion, regardless of which partition the data ends up in.

This also works with direct inserts to the partitions:

=$ INSERT INTO some_data_y2024 (created_at, payload) VALUES (now(), 'manual #1') returning *;
 id │          created_at          │  payload
────┼──────────────────────────────┼───────────
 112024-02-07 15:54:46.95793+00 │ manual #1
(1 ROW)
 
=$ INSERT INTO some_data_y2023 (created_at, payload) VALUES (now() - '1 year'::INTERVAL, 'manual #2') returning *;
 id │          created_at           │  payload
────┼───────────────────────────────┼───────────
 122023-02-07 15:54:55.463043+00 │ manual #2
(1 ROW)

One issue it that we can't have primary key or unique on the id column:

=$ ALTER TABLE some_data ADD PRIMARY KEY (id);
ERROR:  UNIQUE CONSTRAINT ON partitioned TABLE must include ALL partitioning COLUMNS
DETAIL:  PRIMARY KEY CONSTRAINT ON TABLE "some_data" lacks COLUMN "created_at" which IS part OF the partition KEY.
 
=$ ALTER TABLE some_data ADD UNIQUE (id);
ERROR:  UNIQUE CONSTRAINT ON partitioned TABLE must include ALL partitioning COLUMNS
DETAIL:  UNIQUE CONSTRAINT ON TABLE "some_data" lacks COLUMN "created_at" which IS part OF the partition KEY.

Which isn't perfect, but we can always add unique on the partitions directly:

=$ ALTER TABLE some_data_old   ADD UNIQUE (id);
=$ ALTER TABLE some_data_y2022 ADD UNIQUE (id);
=$ ALTER TABLE some_data_y2023 ADD UNIQUE (id);
=$ ALTER TABLE some_data_y2024 ADD UNIQUE (id);
=$ ALTER TABLE some_data_y2025 ADD UNIQUE (id);

Of course, Pg can't know which partition has row with specific id, so it will have to scan them all:

=$ EXPLAIN SELECT * FROM some_data WHERE id = 7;
                                                    QUERY PLAN
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Append  (cost=0.00..12.35 ROWS=5 width=48)
   ->  Seq Scan ON some_data_old some_data_1  (cost=0.00..1.05 ROWS=1 width=48)
         FILTER: (id = 7)
   ->  Seq Scan ON some_data_y2022 some_data_2  (cost=0.00..1.01 ROWS=1 width=48)
         FILTER: (id = 7)
   ->  Seq Scan ON some_data_y2023 some_data_3  (cost=0.00..1.07 ROWS=1 width=48)
         FILTER: (id = 7)
   ->  Seq Scan ON some_data_y2024 some_data_4  (cost=0.00..1.01 ROWS=1 width=48)
         FILTER: (id = 7)
   ->  INDEX Scan USING some_data_y2025_id_key1 ON some_data_y2025 some_data_5  (cost=0.15..8.17 ROWS=1 width=48)
         INDEX Cond: (id = 7)
(11 ROWS)

But if will use index on id (once there will be enough data), so it should be fast.

On the other hand, if your partitioning schema is using identity column as base for partitioning, it will work well, and can have primary key on it:

=$ CREATE TABLE some_data (
    id             int8 generated always AS IDENTITY NOT NULL,
    payload        TEXT,
    PRIMARY KEY (id)
) PARTITION BY RANGE ( id ) ;
 
=$ CREATE TABLE some_data_old PARTITION OF some_data FOR VALUES FROM (MINVALUE) TO (100);
=$ CREATE TABLE some_data_1xx PARTITION OF some_data FOR VALUES FROM (100) TO (200);
=$ CREATE TABLE some_data_2xx PARTITION OF some_data FOR VALUES FROM (200) TO (300);
=$ CREATE TABLE some_data_3xx PARTITION OF some_data FOR VALUES FROM (300) TO (400);
=$ CREATE TABLE some_data_4xx PARTITION OF some_data FOR VALUES FROM (400) TO (500);

and then:

=$ \d some_data_2xx
                      TABLE "public.some_data_2xx"
 COLUMNTYPE  │ Collation │ NULLABLEDEFAULT
─────────┼────────┼───────────┼──────────┼──────────────────────────────
 id      │ BIGINT │           │ NOT NULL │ generated always AS IDENTITY
 payload │ text   │           │          │
Partition OF: some_data FOR VALUES FROM ('200') TO ('300')
Indexes:
    "some_data_2xx_pkey" PRIMARY KEY, btree (id)

As a side note, such syntax will break direct inserts to partitions:

$ INSERT INTO some_data_2xx (payload) VALUES ('a');
ERROR:  NEW ROW FOR relation "some_data_2xx" violates partition CONSTRAINT
DETAIL:  Failing ROW contains (1, a).

This is, of course, because identity is shared, and newly given ids will be “too old" for some_data_2xx, and should go to some_data_old.

I think it's great thing, and I really hope we will be able to get primary keys over partitions, even if they don't align with partition key.

Thanks to all involved 🙂

One thought on “Waiting for PostgreSQL 17 – Support identity columns in partitioned tables”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.