Turning the Users Loose

You never find all the bugs in anything until you turn real users loose on it.

Yesterday at work, while the library was closed, I made the changes to our network infrastructure that have been in the planning and testing stages for several months now. Instead of one firewall protecting our network from the outside world and the subnets from one another, we now have two firewalls: an outer firewall that protects the whole network from the outside world, and an inner firewall that protects mission-critical systems from the rest of the local network. At both points there is no ethernet path past the firewall, so the isolation is at the physical level, and the only stuff that goes through is what the firewall expressly forwards.

I tested everything I could think of to test, of course. My testing checklist, which I spent weeks compiling, was a full page, and many of the lines read along the lines of "Test foo, bar, and baz on all of the staff workstations". I tested ICMP echo. I tested ssh. I tested the web. I tested the ability to access our web catalog. I tested the ability to access our cgi server, and whether each of several internal databases thereon automatically authenticate the user by IP address (which they *are* supposed to do for staff workstations, and *not* for anything else). I tested ftp. I tested the ability to print, to each printer. I tested all of that on each and every computer in the building.

But I forgot to test encrypted websites (https). Naturally, this morning at 9:30 (we open at 9), about five users discovered this oversight at more or less the same time (within the space of a couple of minutes).

I had to actually go in to find out what was wrong, because my coworker who was describing the problem to me on the phone was too flustered (what with several people hounding her about it at once and everything, and not being very technically inclined anyhow) to explain it very well.

So all the way there I was thinking that the internet was completely not working, and I'm like, but I *tested* that right before I left. I tested *every* computer. They could all access the internet last night...

When I got there, of course, I walked over to a patron, who promptly explained that she could access Yahoo mail, but when she tried to "do anything" (which she demonstrates by clicking a "log in" button), ... Ah, yes. I did indeed forget to test any encrypted sites. (I even tested sites that have you log in, but the ones I tested were low-security sites where the only thing at stake would be a little public profile information, so they didn't bother with SSL...)

Sure enough, port 443, although I had intended for it to be forwarded, wasn't. Well, it was from some parts of the network, but not from the patron subnet. Oops. Fixing it, once I figured out what the problem was, was a simple matter.

You never find all the bugs until you deploy.