What, my packets are coming from what interface?

Once again I'm thwarted by the design of the Linux network stack. You see, by default, when a packet comes into one interface in Linux, you can't count on the reply coming back through the same interface. This is not always a problem, but often if that reply was through a firewall the firewall will reject the reply connection as it comes from a different interface/MAC address.

This happened to me again today, for about the third inconvenient time. This time it was too much hassle to move things around, so I found a solution:

http://www.novell.com/support/viewContent.do?externalId=7000318

I don't know if it will persist reboots, but it's enough to get me through a few days (we are moving to new IP space anyway). Here is what I did, in my case (192.x.98.192 was the newly added, badly behaving interface):

sudo vi rt_tables # add table vlan98 with an unused id (252)

sudo ip route add 192.x.98.0/24 dev eth1 src 192.124.98.192 table vlan98

sudo ip route add default via 192.x.98.1 dev eth1 src 192.124.98.192 table vlan98

sudo ip rule add from 192.x.98.192 table vlan98

I hope that saves someone some time.

LISA 2009 Takeaways

Last week I had the pleasure of attending the 23rd Annual Large Installation System Administration (LISA) conference in Baltimore. Well, it's been more than a week now as I've put off blogging too long. So, anyway, I figured I'd post about a few things I learned and how I intend to apply those learnings.

First off, cool people I met. I heard a lot of great talks, but of special significance was meeting Thomas Limoncelli who wrote one of my favorite books, Time Management for System Administrators. That book really changed the way I work, and quite frankly the way I live. I think it's the best book ever written on time management, but then again I'm a system administrator, so I may be biased. Tom also helped write The Practice of System and Network Administration, also a very fine book (which I haven't quite gotten to the bottom of, but working on it).

I also met a very active blogger/SA who does a lot of work surrounding small sites, and in fact now has a book deal to write on the subject, Matt Simmons. I think we will continue to see great things from him.

Finally, I met a couple of interesting fellows from the University of Illinois at Urbana-Champaign. We did not talk much about work, but we had a few good meals and drinks together.

I'm starting to feel like this post might not get up tonight, so I'm going to quickly get down to the meat: what I'm going to change after LISA '09:

1. Better Meeting Management
2. Implement Configuration Management (bcfg2)
3. Play around with ganeti/DRDB

Number one is quite obvious, who likes meetings? Well, I learned a few tips in a management class and some more in a time management session to hold them more effectively. I feel like I knew most of it, but was in denial or just being lazy. My first action is to hold a weekly "standing" meeting where everyone in the group talks for just 2-5 minutes on their current projects, and no more. That way everyone knows the gist of what's going on, without wasting everyone's time.

I was really surprised to hear that so many people, many at sites smaller than ours, use configuration management (bcfg2, puppet, cfengine, chef, etc.). I thought only huge sites were using this, or sites where you have tens to hundreds of identical servers (like web farms), but no, practically everyone is doing it, and so should we. I've seen the light.

Number three is more fun. When asked to implement a new virtual environment about a year ago, my response was VMware ESX / VMware Virtual Infrastructure over my perceived headache with Xen. However, through the virtualization workshop and ganeti BoF session, I feel like ganeti is a viable alternative without the costly SAN requirement. I don't think we will switch anytime soon, but I will experiment with ganeti and see how well it works. I was particularly impressed with its use of DRDB to replicate disk volumes over the network, instead of using a single image on a SAN. This buys you the ability to fail-over from one machine to another without the use of a SAN, and a lot cheaper than VMware Enterprise licenses (we don't have the Enterprise version, too expensive).

I did learn way more things, but I feel three is just about the right number of items to hold myself accountable to... it might actually get done that way.

There you have it, a practical brain dump summary from my LISA '09 experiences. Until next time...

net2ftp

Well, it's been quite a while since I've last wrote. I'll spare you all the normal excuses. My new strategy is to write more often, but smaller posts.

So with that out of the way, I'd like to talk a little bit about net2ftp. I've been looking around high and low the last few years for a really good web uploader/downloader application. Seems like there would be millions and little PHP or JavaScript applications out there to do this, but there never seems to be a very good one. So when I stumbled across net2ftp, I was pleasantly surprised.

While net2ftp isn't quite what I was looking for in uploading application (a two piece solution), it fits the bill nicely. Net2ftp is simply a PHP FTP client, but can be easily customized for your needs. I have found it to act very well as a simple website management tool for my customers. It allows them to easily upload contents to their old-style HTML websites, and hiding the idiosyncrasies of FTP from them.

I've paired up net2ftp with a simple vsFTP configuration that only listens to the local server interface and chroots all the users to their home directories. I was able to have the configuration up and running and uploading files within a half-hour.

There really isn't much else to say about net2ftp, but one killer feature I found was the ability to upload an entire directory from your computer to the remote machine. This killer feature is not found on any other PHP tool that I've tried before, but was sorely needed.

So if you ever have a need for a simple web publishing tool, check out net2ftp, I think you'll be very happy with it.

Until next time . . .

(PS: this post is brought to you in part by Dragon NaturallySpeaking, I am trying it out to give my wrists and hands a much-needed break.)

Just How Urgent are Backups?

After another long day at work today, and early start, I was really looking forward to relaxing at home and sleeping in tomorrow... but the tape library at work has issues so I'm staying up late with support trying to fix the issue. Now I can't claim to be totally blameless as to the issue, but it makes me wonder just how urgent (not important) backups are? I mean, should I really stay up late to fix a backup job or slew of jobs I could fix in the morning?

As usual it would seem there are merits to both sides of the argument. On the one hand you've already got backups from the previous seven to 365 days, but on the other hand Murphy's law proclaims if you miss one backup that will be the one with the CEO's accidentally deleted $30MM deal closing email on it.

Due to the latter I'm taking it safe... but hopefully next time I'll know a bit more about the system before I go mucking with it ;).