The Fuglu truth of Mail Server update time...

Ever since MacOS X Server that I received installed on a new version of the Mac Mini called the Server version back around 2006 or seven which was marketed as being installed with a full licensed version of MacOS X Server (a failed/discontinued product of Apple and one of its biggest failures amongst others - basically consisting of a management system for the underlying tools it setup - badly - and in hopelessly irretrievable ways sometimes through its Mac GUI.  The frustration that a slight misconfiguration could cause was demoralising at times. It used bind and the smallest adjustments in MacOS Server's GUI tools could lead to horrible effects that were impossible to diagnose with poor logs. I don't know why I put up with it so long.

Snowballing bugs and effects that eventually lead to the self destruction of the server. Amavisd had a notorious memory leak that was known for a while and required restarting the system on a schedule to keep the machine from crashing. I went through those years learning much more about the underlying unix cli (I was familiar with CLI from the POSIX compliant AmigaOS from the 80's). Ultimately I ended up using the Mac with my own latest installs of the same tools it did, sometimes by compiling them from scratch.

That all ended up in trying a true unix in the form of Solaris 10 which I sucessfully installed onto a cheap HP Microserver and upgraded all the way to the last Solaris 11. Sun had some huge things right. The zones containerization it allowed gave me my first taste of this technology as far as a decade ago. It carried my mail system from its Mac OS X Server beginnings. If nothing I learnt about DNS, mail and how to setup a system at /etc file level eschewing the graphical disaster Apple had created. My first foray into this lead to losing all the accumulated email from around 2002. The second time I migrated the mail was from the MacOS server to the Solaris machine through an IMAP account interface (some sort of issue with file ownership prevented other ways) and since then it's moved again to a Hyper-V virtual machine.

I should mention that the Solaris machine with its ZFS and the extra ECC RAM which I thought was awesome,  required a full build of many mail tools from hours of reading and making the changes to compile and use on Solaris.  

It's from the Hyper V virtual machine running OpenSuSE (BtrFS has made many things possible with this) now that my mail server will move to a custom containerised stack of 4 containers with two to a team of mailserver and mailhelper. The mailservers have only the main postgresql server and postfix and dovecot (the same duo Apple had chosen to implement mail on MacOS Server and which I was very familiar with administering via command line (not some mail control panel web interface etc.). The mailhelper has virus and spam duties as well as dkim mail signing and other filters designed to greylist known spam sending domains and IPs and help with keeping certificates updated.

I'm planning to add on postfix only backup MX for each team also. At two separate locations - the second which has two separate WAN static IPs on different internet lines means my email should face fewer interruptions than ever before.

And I'm introducing new tools into the mix too. The one big issue I've had with all the mail servers I've had has been amavisd-new. It is the most fragile of the services and being built in Perl I never really thought it was the best use of CPU time. But since SpamAssassin was so conveniently managed through it I persisted through the inevitable systemd failed to start amavisd-new that would plague me from time to time.

So I have tried clamsmtpd an old in between service for ClamAV and it works without any issues so far. I am also trying Fuglu written in Python. I tried getting it to work on Alpine Linux but it wasn't fruitful at all. Alpine's departures from libc etc probably don't help - and even though it probably isn't insurmountable I don't have time to conquer it now nor I feel the need.

When I was looking at the next mailserver implementation - I considered going back to an all Linux solution and I think I am there now. Well soon anyway. It's been around 4 weeks of tinkering in my spare time to get to the point where all my old services are running as Docker managed containers including this very website which is an easily replicable throwaway container which I can create for anyone else who wanted one from me quickly (I haven't written a script for it but it would be trivial to do).

The mailserver was problematic on Docker as making a Dockerfile for it was difficult and mail server need monitoring for security problems - and though it seems mad I was for a time running  a parallel mailserver for offline use (mail folders were updated by rsync) on Tumbleweed edition of OpenSuSE and it worked fine. I don't like trusting solutions that are not mine entirely, short of writing the tools themselves. Docker's slight opacity in terms of access into the container to examine it for the 'secret sauce' was problematic in a mailserver and anything which I built myself was still going to require updating - with very little advantage over doing it in a container I had full control of...

root@lxc:/lxd/containers# du -sh *
170M	mailserver

So you can see that for 170MB I can have a full mailserver (postfix, dovecot and postgresql services on Alpine Linux base) with full control of the toolchain and no backdoors or mysterious bugs that I can't diagnose and control. I am in the last stages of testing this setup before deploying.

Fuglu has brought in many new possibilities into the toolchain and I am becoming more convinced than ever that amavisd-new has seen its last days on systems I manage. I think that I'll probably Dockerize some things in the setup in the future. The first candidate will be a database Docker container that will allow even simpler management of its storage needs in some ways. Each passing day gives me more confidence in lxc as I become more familiar with it. I'll keep my Hyper-V going for a while longer but I think it's days are coming to an end too for me. As soon as I can reliably get my Windows VMs into the Linux container stage it will be as good as gone too.

I think that anyone who manages their own mailserver knows that they have a job for life.