Waiting for PostgreSQL 12 – Add log_statement_sample_rate parameter

On 29th of November 2018, Alvaro Herrera committed patch:

Add log_statement_sample_rate parameter
 
This allows to set a lower log_min_duration_statement value without
incurring excessive log traffic (which reduces performance).  This can
be useful to analyze workloads with lots of short queries.
 
Author: Adrien Nayrat
 
Discussion: https://postgr.es/m/-ee1e-db9f-fa97-@anayrat.info

Continue reading Waiting for PostgreSQL 12 – Add log_statement_sample_rate parameter

Waiting for PostgreSQL 12 – Add CSV table output mode in psql.

On 26th of November 2018, Tom Lane committed patch:

Add CSV table output mode in psql.
 
"\pset format csv", or --csv, selects comma-separated values table format.
This is compliant with RFC 4180, except that we aren't too picky about
whether the record separator is LF or CRLF; also, the user may choose a
field separator other than comma.
 
This output format is directly compatible with the server's COPY CSV
format, and will also be useful as input to other programs.  It's
considerably safer for that purpose than the old recommendation to
use "unaligned" format, since the latter couldn't handle data
containing the field separator character.
 
Daniel Vérité, reviewed by Fabien Coelho and David Fetter, some
tweaking by me
 
Discussion: https://postgr.es/m/a8de371e-006f-4f92-ab72-2bbe3ee78f03@manitou-mail.org

Continue reading Waiting for PostgreSQL 12 – Add CSV table output mode in psql.

Waiting for PostgreSQL 12 – Integrate recovery.conf into postgresql.conf

On 25th of November 2018, Peter Eisentraut committed patch:

Integrate recovery.conf into postgresql.conf
 
recovery.conf settings are now set in postgresql.conf (or other GUC
sources).  Currently, all the affected settings are PGC_POSTMASTER;
this could be refined in the future case by case.
 
Recovery is now initiated by a file recovery.signal.  Standby mode is
initiated by a file standby.signal.  The standby_mode setting is
gone.  If a recovery.conf file is found, an error is issued.
 
The trigger_file setting has been renamed to promote_trigger_file as
part of the move.
 
The documentation chapter "Recovery Configuration" has been integrated
into "Server Configuration".
 
pg_basebackup -R now appends settings to postgresql.auto.conf and
creates a standby.signal file.
 
Author: Fujii Masao
Author: Simon Riggs
Author: Abhijit Menon-Sen
Author: Sergei Kornilov
Discussion: https://www.postgresql.org/message-id/flat/@web3g.yandex.ru/

Continue reading Waiting for PostgreSQL 12 – Integrate recovery.conf into postgresql.conf

Foreign Key to partitioned table – part 2

Previously I wrote about how to create foreign key pointing to partitioned table.

Final solution in there required four separate functions and four triggers for each key between two tables.

Let's see how fast it is, and if it's possible to make it simpler.

Continue reading Foreign Key to partitioned table – part 2

Waiting for PostgreSQL 12 – Add pg_partition_tree to display information about partitions

On 30th of October 2018, Michael Paquier committed patch:

Add pg_partition_tree to display information about partitions
 
 
This new function is useful to display a full tree of partitions with a
partitioned table given in output, and avoids the need of any complex
WITH RECURSIVE query when looking at partition trees which are
deep multiple levels.
 
It returns a set of records, one for each partition, containing the
partition's name, its immediate parent's name, a boolean value telling
if the relation is a leaf in the tree and an integer telling its level
in the partition tree with given table considered as root, beginning at
zero for the root, and incrementing by one each time the scan goes one
level down.
 
Author: Amit Langote
 
Discussion: https://postgr.es/m/-9a51-ad02-d53e-@lab.ntt.co.jp

Continue reading Waiting for PostgreSQL 12 – Add pg_partition_tree to display information about partitions

Waiting for PostgreSQL 12 – Add pg_promote function

On 25th of October 2018, Michael Paquier committed patch:

Add pg_promote function
 
 
This function is able to promote a standby with this new SQL-callable
function.  Execution access can be granted to non-superusers so that
failover tools can observe the principle of least privilege.
 
Catalog version is bumped.
 
Author: Laurenz Albe
 
Discussion: https://postgr.es/m/.camel@cybertec.at

Continue reading Waiting for PostgreSQL 12 – Add pg_promote function

pg_terminator released

I just released first version of pg_terminator.

It's a tool that is supposed to be run on PostgreSQL db server, monitor a database, and cancel or terminate offending queries/connections.

It can be used, for example to:

  • kill leftover psql sessions that are not doing anything
  • cancel too long queries on production servers
  • kill connections that have long idle-in-transaction state
  • make sure that some application will not use transactions longer than “x"
  • and more …

Basically – if you can write a where condition, that operates on pg_stat_activity view, that can list backends to cancel/terminate – you can make pg_terminate do it for you.

To work, it requires Ruby, and its Pg library.

Usage is free, and it's fully open-source (BSD license).

Waiting for PostgreSQL 12 – Allow multi-inserts during COPY into a partitioned table

On 1st of August 2018, Peter Eisentraut committed patch:

Allow multi-inserts during COPY into a partitioned table
 
 
CopyFrom allows multi-inserts to be used for non-partitioned tables, but
this was disabled for partitioned tables.  The reason for this appeared
to be that the tuple may not belong to the same partition as the
previous tuple did.  Not allowing multi-inserts here greatly slowed down
imports into partitioned tables.  These could take twice as long as a
copy to an equivalent non-partitioned table.  It seems wise to do
something about this, so this change allows the multi-inserts by
flushing the so-far inserted tuples to the partition when the next tuple
does not belong to the same partition, or when the buffer fills.  This
improves performance when the next tuple in the stream commonly belongs
to the same partition as the previous tuple.
 
In cases where the target partition changes on every tuple, using
multi-inserts slightly slows the performance.  To get around this we
track the average size of the batches that have been inserted and
adaptively enable or disable multi-inserts based on the size of the
batch.  Some testing was done and the regression only seems to exist
when the average size of the insert batch is close to 1, so let's just
enable multi-inserts when the average size is at least 1.3.  More
performance testing might reveal a better number for, this, but since
the slowdown was only 1-2% it does not seem critical enough to spend too
much time calculating it.  In any case it may depend on other factors
rather than just the size of the batch.
 
Allowing multi-inserts for partitions required a bit of work around the
per-tuple memory contexts as we must flush the tuples when the next
tuple does not belong the same partition.  In which case there is no
good time to reset the per-tuple context, as we've already built the new
tuple by this time.  In order to work around this we maintain two
per-tuple contexts and just switch between them every time the partition
changes and reset the old one.  This does mean that the first of each
batch of tuples is not allocated in the same memory context as the
others, but that does not matter since we only reset the context once
the previous batch has been inserted.
 
Author: David Rowley <david.rowley@2ndquadrant.com>

Continue reading Waiting for PostgreSQL 12 – Allow multi-inserts during COPY into a partitioned table