Converting Logical Volume so that it’s striped

In case you're not familiar – there is a thing called LVM – it's a layer between physical disks, and filesystems, and allow certain interesting things, like extending, migrating, snapshotting and others.

At one of systems I've been dealing with, we stumbled upon specific requirement – change LV into striped. It took me a while to figure it out, so I'm writing it down, so I'll never have to research it again.

Continue reading Converting Logical Volume so that it's striped

OmniPITR 1.3.1

Right after releasing 1.3.0 I realized that I forgot about one thing.

If you're using ext3 (and possibly other, not sure) file system, removal of large file can cause problems due to heavy IO traffic.

We did hit this problem earlier at one of client sites, and devised a way to remove large files by truncating them, bit after bit, and getting them to small enough size to be removed in one go. I wrote about it earlier, of course.

Unfortunately – I forgot about this when releasing 1.3.0, but as soon as I tried to deploy at the client site, I noticed the missing functionality.

So, today I released 1.3.1, which adds two options to omnipitr-backup-cleanup:

  • –truncate
  • –sleep

If truncate is specified, and is more than 0, it will cause omnipitr-backup-slave to remove large files (larger than truncate value) in steps.

In pseudocode:

if param('truncate') {
  file_size = file_to_be_removed.size()
  while ( file_size > param('truncate') ) {
    file_size = file_size - param('truncate')
    file_to_be_removed.truncate_to( file_size )
    sleep( param('sleep') )
  }
}
file_to_be_removed.unlink()

So, for example, specifying –truncate=1000000, will remove the file truncating it first by 1MB blocks.

–sleep parameter is used to delay removal of next part of the file (it's used only in truncating loop, so has no meaning when truncate-loop is not used). It's value is in milliseconds, and defaults to 500 (0.5 second).

Hope you'll find it useful.

How much RAM is PostgreSQL using?

(disclaimer: all the data and examples in here are on Linux – the same data can be probably obtained on other systems too, it's just that I work on Linux and don't know other systems well).

This question pops occasionally in various places – PostgreSQL is using too much memory, why is that, and how can it be mitigated?

Before we can go to “optimizing", we should understand the problem. But do we? Both standard tools – ps and top – lie. How/why? Let's see.

Continue reading How much RAM is PostgreSQL using?

Named interfaces for OpenVPN and Cisco VPN

I use quite a lot of vpns. On any given moment I have between 3 and 10 active vpn connections from the machine I'm working on.

I generally tend to use OpenVPN, but I also do use vpnc (Cisco VPN client).

One thing that I noticed is not very commonly known, and ( in my case – helps a lot ), is that you don't have to have your tunnel interfaces named tun0, tun1, tun2 and so on.

You can do something like this:

in /etc/vpnc/tunnel.conf, add such line:

Interface name <strong>depesz</strong>

and enable the tunnel – you will see that instead of boring, and somewhat cryptic “tunX" interface you'll have interface named “depesz“.

Similar thing for openvpn – in it's conf file for tunnel add these 2 lines:

dev-type tun
dev <strong>smart</strong>

First line is new one (usually), and is required so that OpenVPN can know if you want TUN or TAP tunnels. “dev" is usually “tun", but it can be changed, and with above lines, will make the tunnel interface named “smart“.

Auto refreshing password file for pgbouncer

As you perhaps know I'm fan of pgbouncer – connection pooling solution for PostgreSQL.

It can do many really cool things, but has one slight issue.

Since it can reuse connections – it has to provide a way to check if user supplied password is correct without consulting database. And it lately (since 9.0 to be exact) became somewhat of a problem.

Continue reading Auto refreshing password file for pgbouncer

Faking “From ” header with procmail/formail

I have an unusual mail setup.

My company mails are handled by gmail, and on my account there, I setup forwarding to my own mail account on my own server. Where I read the mails, and respond.

So, when someone (let's say “president@whitehouse.gov") sends me an email to my company email (let's say “depesz@company.example.com"), it arrives to gmail, where it gets forwarded to my real account ( let's say “real@depesz.com" ).

So far so good. The problem is that gmail, when forwarding mail modifies return-path, and thus my local SMTP server changes “From " pseudo-header from normal “From president@whitehouse.gov some date" to “From depesz+some_bullshit_gmail_info@company.example.com some date".

This is bad, because it destroys procmail logs, which use value from “From " to log information about who sent the email, and instead of nice and readable:

From president@whitehouse.gov Wed Sep 22 00:10:36 2010
 Subject: test Wed Sep 22 00:10:30 CEST 2010
   Folder: /somewhere/not/important/new/23423     4870

I get utterly useless:

From depesz+some_bullshit_gmail_info@company.example.com Wed Sep 22 00:10:36 2010
 Subject: test Wed Sep 22 00:10:30 CEST 2010
   Folder: /somewhere/not/important/new/23423     4870

I tried to fix the problem using formail, but apparently, when you do something like this:

:0 fhw
| formail -I "From $REAL_EMAIL"

in procmail.rc – it all works fine, but the “From " line is generated at the end of headers, which is pretty dumb, as it should be leading header.

Tried various stuff to solve the problem, but finally found one that really works. Here it goes in case someone in future will need it:

ENVELOPE_FROM=`formail "-xFrom "`
 
:0
* ENVELOPE_FROM ?? ^depesz\+.*@company.example.com
{
    MAIL_FROM=`formail -xFrom: | perl -pe 's/.*?(\\S+@\\S+).*/\$1/;s/^<//;s/>\$//'`
    RESTORED_FROM=`echo "From $MAIL_FROM $( echo "$ENVELOPE_FROM" | sed 's/^[^[:space:]]* //' )"`
 
    :0 fhw
    | ( echo "$RESTORED_FROM"; formail -R "From " "X-Old_From_:" )
}

Important – this is at the beginning of procmailrc!

What it does?

First line: ENVELOPE_FROM=`formail "-xFrom "` gets current value of “From " header, and puts it to ENVELOPE_FROM variable.

Now, with:

:0
* ENVELOPE_FROM ?? ^depesz\+.*@company.example.com
{
    ...
}

I check if the ENVELOPE_FROM is the one that is broken (after all, someone else might mail me directly), and if yes – I run what's in side of the { block }.

Inside, I get value of From: (with colon) header, and extract from it email.

After running this line:

MAIL_FROM=`formail -xFrom: | perl -pe 's/.*?(\\S+@\\S+).*/\$1/;s/^<//;s/>\$//'`

Assuming mail had line like From: “Some important guy" <president@whitehouse.gov>, MAIL_FROM will contain “president@whitehouse.gov".

Next line:

RESTORED_FROM=`echo "From $MAIL_FROM $( echo "$ENVELOPE_FROM" | sed 's/^[^[:space:]]* //' )"`

builds new value of “From " header, using extracted email, and timestamp (after first space) from original “From “.

So, to wrap with example. Assuming we have email with headers like:

From depesz+some_bullshit_gmail_info@company.example.com Wed Sep 22 00:10:36 2010
Subject: test Wed Sep 22 00:10:30 CEST 2010
From: "Some important guy" <president@whitehouse.gov>

After the RESTORED_FROM line, we will have following values in variables:

ENVELOPE_FROM="depesz+some_bullshit_gmail_info@company.example.com Wed Sep 22 00:10:36 2010"
MAIL_FROM="president@whitehouse.gov"
RESTORED_FROM="From president@whitehouse.gov Wed Sep 22 00:10:36 2010"

Then goes last part:

:0 fhw
| ( echo "$RESTORED_FROM"; formail -R "From " "X-Old_From_:" )

Which, passed mail headers to command, and treats it as filter.

The command does:

  1. prints new “From " header
  2. passes mail headers through formail, which renames “From " header into “X-Old_From_:" header

The trick is that the print happens before formail even will get the headers, so the new, fixed “From " will be returned to procmail before rest of headers, as returned by formail.

Effect: everything works, and logged From is now much more sensible.

It would be even better if gmail would include original envelope from in the headers, but it doesn't, so I have to take the address from “From:" (which not always is good idea, but at the very least – it's much better than getting all mails with the same “From “.

Anyone knows better/easier approach?

Tips N’ Tricks – using GNU Screen as shell

I'm quite often doing stuff on remote machines, and quite frequently I start some long-running job, when I remember that I didn't ran it via screen – so it will break, if my network connection will die.

Is there any sane way to start screen automatically? YES.

Continue reading Tips N' Tricks – using GNU Screen as shell