Waiting for 9.4 – Introduce logical decoding

On 3rd of March, Robert Haas committed patch:

Introduce logical decoding.
This feature, building on previous commits, allows the write-ahead log
stream to be decoded into a series of logical changes; that is,
inserts, updates, and deletes and the transactions which contain them.
It is capable of handling decoding even across changes to the schema
of the effected tables.  The output format is controlled by a
so-called "output plugin"; an example is included.  To make use of
this in a real replication system, the output plugin will need to be
modified to produce output in the format appropriate to that system,
and to perform filtering.
Currently, information can be extracted from the logical decoding
system only via SQL; future commits will add the ability to stream
changes via walsender.
Andres Freund, with review and other contributions from many other
people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan,
Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit
Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve

Continue reading Waiting for 9.4 – Introduce logical decoding

Waiting for 9.4 – Add pg_stat_archiver statistics view.

On 28th of January, Fujii Masao committed patch:

Add pg_stat_archiver statistics view.
This view shows the statistics about the WAL archiver process's activity.
Gabriele Bartolini, reviewed by Michael Paquier, refactored a bit by me.

Continue reading Waiting for 9.4 – Add pg_stat_archiver statistics view.

Waiting for 9.4 – Allow time delayed standbys and recovery

On 12th of December, Simon Riggs committed patch:

Allow time delayed standbys and recovery
Set min_recovery_apply_delay to force a delay in recovery apply for commit and
restore point WAL records. Other records are replayed immediately. Delay is
measured between WAL record time and local standby time.
Robert Haas, Fabrízio de Royes Mello and Simon Riggs
Detailed review by Mitsumasa Kondo

Continue reading Waiting for 9.4 – Allow time delayed standbys and recovery

Waiting for 9.4 – Add new wal_level, logical, sufficient for logical decoding.

On 11th of December, Robert Haas committed patch:

Add new wal_level, logical, sufficient for logical decoding.
When wal_level=logical, we'll log columns from the old tuple as
configured by the REPLICA IDENTITY facility added in commit
<a class="text" href="/gitweb/?p=postgresql.git;a=object;h=07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65">07cacba983ef79be4a84fcd0e0ca3b5fcb85dd65</a>.  This makes it possible
a properly-configured logical replication solution to correctly
follow table updates even if they change the chosen key columns,
or, with REPLICA IDENTITY FULL, even if the table has no key at
all.  Note that updates which do not modify the replica identity
column won't log anything extra, making the choice of a good key
(i.e. one that will rarely be changed) important to performance
when wal_level=logical is configured.
Each insert, update, or delete to a catalog table will also log
the CMIN and/or CMAX values of stamped by the current transaction.
This is necessary because logical decoding will require access to
historical snapshots of the catalog in order to decode some data
types, and the CMIN/CMAX values that we may need in order to judge
row visibility may have been overwritten by the time we need them.
Andres Freund, reviewed in various versions by myself, Heikki
Linnakangas, KONDO Mitsumasa, and many others.

Continue reading Waiting for 9.4 – Add new wal_level, logical, sufficient for logical decoding.

Waiting for 9.3 – Allow a streaming replication standby to follow a timeline switch.

On 13th of December, Heikki Linnakangas committed patch:

Allow a streaming replication standby to follow a timeline switch.
Before this patch, streaming replication would refuse to start replicating
if the timeline in the primary doesn't exactly match the standby. The
situation where it doesn't match is when you have a master, and two
standbys, and you promote one of the standbys to become new master.
Promoting bumps up the timeline ID, and after that bump, the other standby
would refuse to continue.
There's significantly more timeline related logic in streaming replication
now. First of all, when a standby connects to primary, it will ask the
primary for any timeline history files that are missing from the standby.
The missing files are sent using a new replication command TIMELINE_HISTORY,
and stored in standby's pg_xlog directory. Using the timeline history files,
the standby can follow the latest timeline present in the primary
(recovery_target_timeline='latest'), just as it can follow new timelines
appearing in an archive directory.
START_REPLICATION now takes a TIMELINE parameter, to specify exactly which
timeline to stream WAL from. This allows the standby to request the primary
to send over WAL that precedes the promotion. The replication protocol is
changed slightly (in a backwards-compatible way although there's little hope
of streaming replication working across major versions anyway), to allow
replication to stop when the end of timeline reached, putting the walsender
back into accepting a replication command.
Many thanks to Amit Kapila for testing and reviewing various versions of
this patch.

Continue reading Waiting for 9.3 – Allow a streaming replication standby to follow a timeline switch.

OmniPITR 0.7.0

Just released new version of OmniPITR.

This version has one important new feature: when you're calling omnipitr-backup-slave, it will make backups only of required xlog files, and not, as previously, of all in walarchive directory.

This is important, especially in case you have multiple slaves, or you keep shared long-term walarchive. Previously – backups would get all files from walarchive (-s option to omnipitr-backup-slave), but now, it picks just the ones that are needed.

On somehow related note – I will be working now, finally, to get omnipitr-monitor functionality working.

OmniPITR 0.6.0

Just released new version, 0.6.0 (it should be visible on pgxn soon) of OmniPITR set of tools.

New version has one new feature – parallelism.

This works in omnipitr-archive and omnipitr-backup-* programs, and allows for parallel delivery to remote destinations (multiple -dr switches).

Also – if you're using compresses wal archive and omnipitr-backup-slave reading from it – all the wal files have to be decompressed before making backup – and this decompression can be parallelized too.

All parallelization is controled using -PJ option (–parallel-jobs), so you can add “-PJ 10" to get up to 10 decompressions at the same time or up to 10 deliveries at the same time.

Waiting for 9.2 – pg_basebackup from slave

On 25th of January, Simon Riggs committed patch:

Allow pg_basebackup from standby node with safety checking.
Base backup follows recommended procedure, plus goes to great
lengths to ensure that partial page writes are avoided.
Jun Ishizuka and Fujii Masao, with minor modifications

Continue reading Waiting for 9.2 – pg_basebackup from slave

OmniPITR 0.3.0

Just released version 0.3.0 of our tool for handling WAL based replication in PostgreSQL – OmniPITR.

Version jump is related to addition of another tool – omnipitr-synch. This tool is used to copy PostgreSQL data dir (including all tablespaces of course) to remote location(s).

While this process is usually simple (call pg_start_backup(), transfer data, call pg_stop_backup()), thanks to the tool it can be wrapped as single call, with standardized logging, and tested logic. It also makes it trivial, and cheap, to setup more than one new slave at a time, without need to read data off master more than once.