April 2nd, 2013 by depesz | Tags: , , , | 3 comments »
Did it help? If yes - maybe you can help me? Donate BTC to 19zPa5diT2LZqGtTi8f8bfApLn8rw9zBHx

Yesterday, I did release new version of OmniPITR – 1.1.0.

You can get it from github or from PgXN.

What's new? Couple of things:

  • Fix: omnipitr-monitor had a bug with parsing timestamps from logs (in cases of “negative" timezones)
  • Fix: sanity-check.sh now returns correct status when exiting, so you can verify if everything is OK by simply checking $?, and not reading its output (helpful for automation scripts)
  • Fix: temp directories are made using PID in path, so you can now have (for example) two separate PG clusters on the same machine, each running omnipitr-restore, without them stepping on each others toes
  • New: added –version to all programs, so you can easily check which version you have installed
  • New: added –help option to all programs, so you can see the documentation without manually finding docs, and running perldoc on them

All of above are pretty cosmetic, but there are two bigger changes too. First of them – –remove-before option to omnipitr-restore. Let's imagine you have normal replication, but for whatever reason slave is lagging. Maybe Pg on slave machine was disabled, or maybe master is just so much faster. In any way – slave is lagging. And this means that walarchive is growing (I'm talking about xlog-file based replication, not streaming).

Walarchive would be growing, because omnipitr-archive delivers new xlogs, but omnipitr-restore will remove them only when it will catchup. Now, with this new option, omnipitr-restore always first checks if there is anything to delete, and if yes – removes single “batch" of files (according to –remove-at-a-time option).

This is not enabled by default, mostly not to change behaviour from previous versions, but also – because “remove-before" is slightly slower – before providing file, it has to run pg_controldata, and optionally remove some (3 by default) files.

The overhead is (in my opinion) minimal, but it is there, so it can't be enabled by default. But – if you're ever in position where walarchive grew too much, and slave was lagging – this is an option for you.

The other larger news is ability to provide config files. This has been added to all programs, but I think it's most important for omnipitr-restore (where changing options required restart of slave because options were in recovery.conf file read only on Pg start).

The file has very simple format:

  • one option per line
  • options are given just like in command line, with the exception that you don't need to quote values
  • both short and long version of switches are acceptable (-v/–verbose)
  • empty lines are ignored
  • lines starting with # are treated as comments
  • This means that instead of having (for example) in your postgresql.conf:

    archive_command = '/opt/OMNIpitr/bin/omnipitr-archive -PJ 5 -s /var/lib/pgsql/omnipitr/state -l /var/lib/pgsql/data/pg_log/omnipitr-^Y-^m-^d.log -v -dr rsync://pg-slave/xlogs/ -dl /mnt/nas/walarchive/ -dr rsync://backup/dba/pg/xlogs/ "%p"'

    you can have it now:

    archive_command = '/opt/OMNIpitr/bin/omnipitr-archive --config-file=/var/lib/pgsql/omnipitr/archive.conf "%p"'

    and in /var/lib/pgsql/omnipitr/archive.conf, you'd put:

    # Logging
    --log=/var/lib/pgsql/data/pg_log/omnipitr-^Y-^m-^d.log
    --verbose
     
    # State - needed because we have more than 1 destination:
    --state-dir=/var/lib/pgsql/omnipitr/state
     
    # Destinations
    --dst-local=/mnt/nas/walarchive/
    --dst-remote=rsync://pg-slave/xlogs/
    --dst-remote=rsync://backup/dba/pg/xlogs/
     
    # Speed up deliveries by running them concurrently
    --parallel-jobs=5

    Also – since config file is read on each run of omnipitr-* program, changes in the file work immediately, and do not require restart of PostgreSQL (nor reload).

    Hope you'll find it useful.

  1. 3 comments

  2. Apr 9, 2013

    Are there any plans to support non-exclusive base backups, e.g. pg_basebackup style ones?

    (At least AFAICT, there is no support for that at this point? Or did I miss it?)

  3. Apr 9, 2013

    Wasn’t in my plans, tbh. The reason is very simple – OmniPITR was started when there was no pg_basebackup, and it is doing its work very differently, as far as I can tell.

    Maybe I will make an option to run pg_basebackup through some omnipitr-backup-* thing, but I’m not entirely sure if it makes sense.

    The nearest todo points are:

    – support %r in omnipitr-restore
    – support remote (http/ftp) walarchive in omnipitr-restore and omnipitr-backup-slave (not sure if it would be possible for omnipitr-backup-master)

    If you can explain a bit to me why you’d want to reimplement pg_basebackup as part of omnipitr, or at least shed some light on the usecase, i might change priorities.

  4. Apr 9, 2013

    Yeah, I understand why it wasn’t there in the beginning.

    It might still be useful to integrate with though – either by running pg_basebackup or just using libpq.T he support for nonexclusive backups probably being the main reason. The “libpq only” isn’t really a big feature in an “OmniPTR envirnonemnt”, I think, but the nonexclusive backups might be.

    I don’t think it’s necessarily a high priority though :)

Leave a comment