2019-11-26

What's wrong, Delta? Afraid of a little snow?

Last week, I got the news that my grandfather had died. The news was very bittersweet. I had talked to him a few weeks before, and I could tell he was just so tired of life. He had always been a very active person, but the last few years, his body just failed to keep up. Especially after my grandma died a little over a couple years ago, there just wasn't much for him to do or anyone to do it with. (Not that he was totally alone or that no one visited — he had 24/7 care, and family in the area that would check up on him often.) So, while I'm very sad to have him go, I'm happy he can be reunited with Grandma and other friends that have gone before him.

I'm also very fortunate that I was able to make it out there — and even more fortunate that I was able to make it home.

I have a cousin in the area who was very nice to allow me to come in and give me a place to stay. In talking with him as I made my travel plans, the most convenient flight out from Denver to San Francisco and back was with Delta. Despite having a layover in Salt Lake both ways, the times worked out decently well for the price.

The flight out was pretty uneventful, which in the context of travelling is a good thing. The flight back, however, is where things got interesting. My flight out was scheduled late in the day of the funeral, and we planned to go from the funeral to the airport for me to catch my flight.

Early in the morning, as we were getting ready to leave my cousin's house for the funeral, I got a text message from Delta indicating that there was a change to one of my flights. I checked using the Delta app (once again entering my confirmation number, since the app forgot it — by this time, I had it memorized, which would become very important). The flight from SFO to SLC was unchanged, but the connection from SLC to DEN had been delayed from that day to 6am the next morning, giving me an overnight layover in Salt Lake. Now, the convenient thing about the Delta app is, when you have a delay like this, it gives you a list of alternatives that you can re-book for no extra cost. Scrolling through the list, though, only showed me a list of flights that started the next day. I didn't want to impose on my cousin for another day (plus have to take another day off of work), so I kept scrolling until I finally saw an option that let me leave the same day. It started with the same flight from SFO to SLC, but the next flight was from SLC to Los Angeles, with about a 5-hour layover there until a 2:30am flight back to Denver. It wasn't great, but at least it got me home.

I clicked the button to book the flight. Then, I looked over my itinerary. SFO to SLC, SLC to LAX, and then LAX to DEN, departing at... 7pm the next day? My 5-hour layover in LAX had somehow turned into a 24-hour layover. And the option to reschedule had disappeared from the app.

With the app failing me, I decided to call the customer service number and talk to a human. They put me into a queue with a 20-minute wait, and the recording gave me the option to have them call me back instead of waiting on hold the whole time. I selected that, and we set off on the road for the funeral.

At this point, it's important to know that I have T-Mobile cell service. It's also important to know that, in the area where my cousin lives (and, as it happens, over most of the route between his place and where the funeral was scheduled), T-Mobile's coverage can charitably be described as "not great". Sure enough, about 25 minutes later, my phone alerted me to a voice mail message, which I played back and heard the automated attendant trying to connect me to an agent. They called back as promised, but I had no signal at the time.

So I called again, this time remaining on hold for the now-25-minute wait time, listening to really bad electronic hold music broken only by a recorded message telling me how I could make changes to my reservation using the Delta app. (Yeah, it was the app that got me into this mess in the first place, thanks.) After about 25 minutes, as promised, an agent picked up. I was able to give him my name and confirmation number, and I heard him comment on how strange my reservation looked now. "Let me see what I can do," I heard — right before we entered another T-Mobile dead zone and the call dropped.

This time, I borrowed my cousin's phone (connected to AT&T service, which had better coverage in that area). After another 25-minute wait (which included several more recorded messages telling me how great the Delta app was), I was able to talk to an agent, who noted that Denver was due for some bad weather, causing some flight complications. To his credit, he was able to find me a flight. It was on a New Zealand flight operated by United (shrug), with direct service from SFO to DEN. It was scheduled to leave San Francisco a little later (giving us more time to get there from the funeral), but arriving home at nearly the same time. Sounded like a win-win. After we got off the phone, I checked the Delta app, and it showed my new flight (though, since it was a United flight, without the ability to check in or make a seat selection or any of the great features provided by the "great" Delta app).

After the funeral, as we're heading to the airport, I decide once more to check the app. It still showed me booked on the New Zealand/United flight, with no indication that anything was amiss. For some reason, I decided to do a quick Bing search (hey, I get Rewards points for using Bing, don't judge) on the flight, and it displayed the status: Cancelled. My cousin asked if I wanted to just return to his house, where I could start making calls, but at that point, I figured we could just continue to the airport, and I could talk to the agents there in person to figure things out.

Once there, I went to the Delta desk. I kind of expected the answer that they wouldn't be able to help me after passing me over to United, and in that, I wasn't disappointed. Of course the Delta and United desks were on opposite ends of the terminal, so I had a bit of a walk over to United.

I got a little worried at the United desk, when they couldn't find my reservation in their system, on any flight. Delta's reservation number wasn't helpful, but the ticket number was — once I found it buried under several clicks in that "really great" Delta app. (There was an additional snag in that the computer wouldn't let the agent do anything with my ticket, but he got that straightened out talking to his tech support — something about the ticket using my full middle name where he tried searching by just my initial.) After a few minutes, though, he was able to put me on a United flight straight from SFO to DEN. He handed me the boarding pass, which showed a departure time of about 1:30pm, about three hours prior. But, not to worry; that was just this flight's original time; it had been delayed until 5:30, which gave me a fair amount of time to get through security and to the flight.

It turns out, this flight's plane had some mechanical issues, so they had replaced it with another plane from Chicago that unfortunately wasn't going to get into San Francisco until after 4. Well, unfortunate for the people who were originally booked on that flight, but fortunate for me, I suppose. I kind of felt bad as I sat in the waiting area, listening to people complain about being there for four hours, while I show up less than an hour prior.

The plane from Chicago ended up being a little delayed; and, after we boarded, they had an issue with the door seal that required a maintenance crew to give it a once-over and fill out some paperwork, further delaying take-off. Then, while I didn't have a window seat and couldn't confirm this, it sure felt like we taxied around to every runway before finally taking off at a little after 6:30pm.

But finally, 2 hours later, despite a snowstorm that was just kicking into high gear, we touched down in Denver, and I was as good as home — at just about the same time I was originally scheduled on my first Delta flight from Salt Lake. As if nothing happened.

And good thing, too; the storm dropped snow all through the night, with nearly 500 flights cancelled at Denver the next day. If I hadn't made it home that night, it could've easily taken me a couple days to get home, right in the middle of the Thanksgiving travel rush.

2019-10-04

Entity Framework 6, SQL, and nullable strings

I ran into an issue that appears to be caused by Microsoft attempting to protect me from myself. Although, truth be told, it wouldn't have been an issue if things were a little better designed.

Imagine, if you will, a SQL Server database with a table of transactions. One of the fields on this table is a CorrelationId. It's a text field that is populated by a different system to tie transactions together (for example, two sides of a transfer from one customer to another). This field always gets populated on new transactions; the uncorrelated ones will just be the only one with a given CorrelationId. However, this system is not new; it was converted to replace an older system that did not have a defined CorrelationId. So, although the five million or so transactions created by this system have a CorrelationId, there are 12 million "legacy" records that have a CorrelationId of NULL.

So, say, for a given transaction, you want to find all correlated transactions. In SQL Server, you might use a simple query like this:

SELECT *
FROM dbo.TransactionTable
WHERE CorrelationId =
(SELECT CorrelationId FROM dbo.TransactionTable WHERE Id = @TransactionId)

And this would work, for the most part (except for legacy records, since SQL will fail to match on the NULL value — but we can ignore this for now). If you took this query into SQL Management Studio and looked at the execution plan, you'd see a nice thin line from the index seek on the CorrelationId, showing that it found and processed a tiny number of matching records, resulting in a very quick response.

However, if you were trying to do this programmatically from a C# application using Entity Framework 6, you might write some code like:

var query = from txn in transactionTable.Entities
where txn.Id == transactionId
join txn2 in transactionTable.Entities on txn.CorrelationId equals txn2.CorrelationId
select txn2;

The problem is, in C# code, null values are equal to another; while in SQL, "null" is considered "unknown", and doesn't equal itself. (The theory is, you can't know if one "null", or unknown value, equals another "null"; so equality tests between "null" and "null" are false.) Instead of leaving it up to the programmer to explicitly code for this condition, Entity Framework "helpfully" writes the join clause that it gives to SQL server in this manner:

JOIN [dbo].[TransactionTable] AS [Extent2] ON (
([Extent1].[CorrelationId] = [Extent2].[CorrelationId])
OR
(([Extent1].[CorrelationId] IS NULL) AND ([Extent2].[CorrelationId] IS NULL))
)

The extra check for IS NULL on both sides has two unfortunate side effects in this case:

  1. If the transaction is one of the legacy records, it will return a positive match on all 12 million other legacy records with a null CorrelationId.
  2. If the transaction has a CorrelationId, because of the IS NULL, SQL Server will investigate the 12 million null values in the CorrelationId index, resulting in a big fat line from the index seek in the execution plan, and a return time of a couple seconds or more.

The really annoying part is that there doesn't appear to be a way to stop this. Even if you explicitly add a check for a not-equal-to-null on your target table, Entity Framework still wraps the equality test with checks for IS NULL. The result is almost comical. For instance, adding txn2.CorrelationId != null either in the join statement or as a where clause, results in this (with contradictory statements highlighted):

[Extent2].[CorrelationId] IS NOT NULL
AND (
([Extent1].[CorrelationId] = [Extent2].[CorrelationId])
OR
(([Extent1].[CorrelationId] IS NULL) AND ([Extent2].[CorrelationId] IS NULL))
)

Even trying to break up the work into two statements didn't help. This code:

var corrId = from txn in transactionTable.Entities where txn.Id == transactionId select txn.CorrelationId;
var txns = from txn in transactionTable.Entities where txn.CorrelationId == corrId select txn;

Resulted in this SQL:

WHERE ([Extent1].[CorrelationId] = @p__linq__0)
OR
(([Extent1].[CorrelationId] IS NULL) AND (@p__linq__0 IS NULL))

Granted, this is a really bad situation to be in to begin with. Indexes on text fields tend to perform poorly, and having such a huge number of null values in the index is likewise unhelpful. A better design would be to rip the text field off into another table, or somehow otherwise convert it into an integer that would be easier to index (something we've had to do in other tables on this very same project, where we've had more control of the data).

I'm willing to bet that Microsoft's translation goes completely unnoticed in over 99% of the cases where it occurs. And, if I had the time to make a design change (with all of the necessary changes to all points that hit this table, some of which I don't have direct control over), it could have been resolved without fighting Entity Framework. Even just populating all of the legacy transactions' CorrelationId with random, unique garbage would've solved the problem (though with a lot of wasted storage space that would've made the infrastructure team cry).

In the end, it was solved by creating stored procedures in the datbase to do correlated transaction lookups (where the behavior could be controlled and expected), and having C# code exectue those directly (bypassing EF6) to get transaction IDs. Standard Linq queries would then use those IDs, instead of trying to search the CorrelationId.

This whole exercise was prompted by a script that I had to run to get a bunch of data from a decent number of transactions. It took nearly eleven hours to complete, finishing close to 1am after I started it. If I had time to go through this debugging and implement the fix, it turns out I could've gotten it done in about a third of the time.

2018-05-12

The Church and the BSA

The news broke that the Church of Jesus Christ of Latter-day Saints has officially announced that it will no longer sponsor Boy Scouts of America troops. As a member of that church who currently serves in a calling as a Cub Scout den leader, and as a father of four boys — one of which just earned his Eagle Scout, and one who likely will within the next year — and being married to someone who currently volunteers as a Roundtable Commissioner for the BSA district, I have some thoughts.

First off, this has been a long time coming. The Church has members all across the world, and the Boy Scouts is an American institution. While Church leaders have often noted that the BSA is, essentially, the Young Men's organization within the Church, this has been completely unavailable outside of this country. We've heard rumors for a long time of the Church looking to institute something that would be available to members everywhere.

There are certain benefits to dumping the BSA from the Young Men's program. I've heard from numerous sources how the youth budget is often seen as unfair, with the Young Men getting far more allocation than the Young Women. However, the main reason for this is that the Church pays much of the registration fees, awards, and so forth required by the BSA — which is something that the Young Women do not have to deal with. The BSA also has many rules, regulations, and so forth that can be difficult for a ward full of unpaid volunteers to navigate. (One of the responsibilities my wife has is to help new scout troops and packs get through some of this — and the ones requiring help are not limited to LDS Church-sponsored troops.)

The move could also allow more freedom in Young Men's groups in their activities. Without having to follow strict rules and a proscribed list of activities and merit badges that boys are supposed to earn, it could be more easily tailored to the needs and desires of the people in the program (kids and leaders alike).

On the other hand, there are some things that would be a great loss. The structure and required activities are sometimes a benefit. When I asked my wife about her experience growing up in Activity Days (the program for girls the age of Cub Scout boys), she expressed her disappointment that it was little more than a cooking class. Her leaders were skilled in the kitchen and enjoyed doing that, so, without any real incentive to do anything else, that's what they did, week after week. Not being a very outdoorsy person myself, if I didn't have the Cub Scout requirements to drive me, I would likewise have very little incentive or direction to do some of the things that the boys in my care would enjoy (or should learn).

With the lack of BSA sponsorship, access to some of the campgrounds and resources that the Church currently enjoys will be lost. So although there will be more money to go around, some activities will cost more.

The Boy Scouts of America is a national institution, recognized by people regardless of their religious affiliation. Anecdotally, having an Eagle Scout rank is something that employers see as an asset in potential employees. Certainly, a Church program would be less likely to carry the same weight, even if its requirements were at all similar to those of the Eagle Scout.

There is very little commonly known about the new youth program-to-be. The official separation — and, I presume, the implementation of the new program — is scheduled for the beginning of 2020. Time will tell how many of the benefits and losses will be offset by the new benefits (and drawbacks) of the new program. My belief in an inspired leadership gives me great hope that many of these things will be addressed, and that the new program (which, incidentally, is to replace the current programs for boys and girls) will be successful.

One common refrain in the news is that the entire LDS membership of the Boy Scouts is going to go away. While that's most certainly an exaggeration, there is some truth to that. Without the encouragement inside of the Church to join Scouting, it's very likely that fewer boys will feel inclined to join. Also, something I've heard from many parents and have experienced myself, is that there seems to be very little time to fit in all of the extracurricular activities that all of your kids are involved in. With the Church bringing in its own program, boys that want to do Scouts will now have two activities to balance (Scouting and whatever the new program is) — and there's no guarantee that meeting times won't conflict. Also, although religion has been an integral part of the Boy Scouts (belief in God is a requirement, even if the manner of worship is completely open), I have heard various reports of non-LDS-sponsored packs and troops using Sunday as an activity day, which is something that most members of the Church try to avoid. Additionally, an LDS-sponsored pack and troop typically takes care of paying the dues for its members. Joining another troop will mean paying those membership costs out of the families' own pockets. (I have not had much personal experience in this, so I do not know how willing and how much other troops can assist with dues and fees for families that may need the help.) So even though the option to join Scouting is not out of the question, it will be more difficult for LDS youth to make that commitment. The BSA membership is probably going to take a significant hit. While it might not be 100% of LDS youth, it wouldn't surprise me to see it be very high.

The timing of the announcements has certainly been interesting. Several publications have noted that the announcement of the LDS/BSA split came within a week of the BSA announcing that they would change their program name from "Boy Scouts" to "Scouts" (as part of their move to include girls in the program), leading many to speculate that the name change was part of the cause. I won't deny that it certainly looks that way, but I suspect the reality is much more complicated.

The announcement of the split consisted of simultaneous press releases from both the Church and the BSA. While it may not be outside of the realm of possibility, I have a hard time believing that a coordinated PR move happens that quickly, which leads me to believe that both sides knew about this for a long time. Now, it's certainly possible also that the Church knew of the BSA's plans ahead of time as well, so there might yet be some merit to the idea that the Church made plans to leave because of the direction the BSA was moving. It's also possible that the BSA knew the Church was planning on striking out on their own with their own youth program, and the decision to include girls was made to try to mitigate the potential loss of LDS boys from their ranks. In other words, the cause-and-effect might be reversed.

Still, from the perspective of public perception (or at least the narrative that some press are driving), it does look like the Church is abandoning the Scouts because of the changes in the Scouts. While it may sound conspiratorial, I have to wonder if it wasn't timed to make the Church look bad, so everyone can point and laugh at the church, "Look at how those backwards Mormons run from simply having to include girls in their program!" On the other hand, if the timing were reversed, it may appear that the BSA would be the ones to appear reactionary, and the story would be, "Look at how the BSA is so desperate for members now, that they're changing their name to include girls!" The timing from the BSA could have been more defensive than offensive.

All told, I'm kind of looking forward to the change. While Scouting has had some great benefits for my boys and the other boys in the Church, I won't deny the administrative side has been a stress to deal with; and I'm hoping the new program will bring some positive change. I don't know if my own younger boys will continue in Scouting (my oldest is already an Eagle and moving into his adult life; my second oldest will be aging-out at nearly the same time the split becomes official), but it will be a choice we will prayerfully consider in the months to come.

2017-06-24

Forget this "Free Healthcare" business!

I had one of those shower moments where I started replaying conversations and debates I've ever had or witnessed. For some reason, my mind had settled on the idea of universal healthcare (not something I've argued much on either side, but definitely witnessed a lot). Proponents often describe this as "free healthcare", which leads to opponents arguing that "it's not free" since it's paid for by taxes. I've even seen one argument that you'd have a hard time convincing a doctor to use his skill and many years of medical school learning and training for no cost.

"Yeah," said the voice in my head, "just like police and firemen should expect to be paid for their service."

And that's when it occurred to me. The proposal shouldn't be "free healthcare"; it should be "make healthcare a public service". Because that's really the truth. No one's really suggesting that anything be "free". They're suggesting that the costs be covered by society as a whole (i.e., government, paid through taxes), rather than by the individual using the service at that point in time.

While I can understand the appeal of calling it "free", I think proponents do the discussion a great disservice by using that word. It implies, at best, a fundamental misunderstanding of economics, and, at worst, a lie covering it up (since both sides know that health care costs actual money, that it's not really "free" at all).

Do I think a mere change in word choice will clear up the whole discussion? Absolutely not. There are still plenty of points to argue — quality of care, the ability of government to manage, and the actual cost for the public, just to name a few — I do think it would at least let us get past the part where we argue about "free" being "free" or "not free".

2017-06-19

Pass a recordset to C# by way of XML

UPDATE: I have to discourage using this trick. For reasons I do not yet know, it doesn't seem to work with a large dataset. I do not know the exact point at which it fails, I just know that it does. I noticed that a significant number of values that should have been updated with text, actually got updated with nulls. As much as I would love to investigate this and try to see what is wrong and whether it's a failure in C#, SQL, or some combination, unfortunately, it's more important that my work actually get done; so I've had to abandon the XML route entirely.

Original post follows.


It's been a while since I've posted, well, anything. But I learned of a neat trick that I thought I'd post.

I'm currently working on a program that is converting data from two different sources into a single database. A lot of it is just done with carefully crafted SQL statements, but there are a few steps where I have to take data from one source and use some C# code to do some kind of processing before storing it in the target database. Since the data set is on the order of millions of rows, processing these records one at a time can be prohibitively time-consuming. And, since I have limited access to the SQL Server itself, using SQL CLR isn't a great option. (I probably could get the access if I needed to, but it will be an additional step to have to remember and configure when this goes to production, and the fewer moving parts I create for myself, the better.)

One of the tricks I've implemented is to use multi-threading to let the different steps run simultaneously — one thread extracts the records and puts them into a ConcurrentQueue<>, another thread processes that and puts the results into another queue, and a third thread updates the records in the database.

I've been trying to come up with ways to do the update in batches. There are ways to create a stored procedure that will take a table parameter, and ways to call that stored procedure by binding the parameter to an equivalent DataSet, but I didn't like the idea of creating a DataSet object just to pass the records in. It just seemed too "heavy" to me. (Though it might've been faster than calling a command object in a loop for records one-by-one.)

Another option was to create a VALUES table and build the command text dynamically. But, since I was working with strings, I didn't like the idea of building dynamic SQL and having to escape quotes or any other special characters that might cause SQL to choke. (Not to mention it's just bad practice, even if my code is unlikely to be used as a SQL injection vector.)

So, I came up with the idea of passing in values as an XML document. By building the XML with Linq-to-XML C# code, all necessary character escapes would be performed automatically. I could pass in as many values at once as I felt comfortable with, and let SQL do the work in a batch instead of one at a time.

To give some context to this code, I am taking email addresses that were encrypted in the source database, and converting them to their decrypted values in the target database. At this point, my queue consists of objects that have two properties: EncryptedEmail and DecryptedEmail. Earlier in my conversion work, I've simply copied the encrypted strings over into the Email field of the table, so all this method has to do is update the table and changing the Email field to its decrypted value.

var recordsToUpdate = GetRecordsToUpdateBatch(250); //Retrieves up to 250 records off of the queue at a time
if (recordsToUpdate.Any()) {

var xdoc = new XDocument(
new XElement("emails",
recordsToUpdate.Select(r => new XElement("email", new XAttribute("encrypted", r.EncryptedEmail), new XAttribute("decrypted", r.DecryptedEmail)))
)
);

using (var connection = new SqlConnection(GlobalSettings.DatabaseConnectionString)) {
await connection.OpenAsync();
using (var command = connection.CreateCommand()) {
command.CommandText = @"
WITH emails AS (
--Convert the XML document into a table that SQL can use normally SELECT Tbl.email.value('@encrypted','varchar(256)') AS Encrypted, Tbl.email.value('@decrypted','varchar(256)') AS Decrypted
FROM @emails.nodes('/emails/email') AS Tbl(email)
) UPDATE cust SET cust.Email = emails.Decrypted
FROM dbo.Customer cust
INNER JOIN emails ON cust.Email = emails.Encrypted;
"
;
command.CommandType = System.Data.CommandType.Text;
var param = command.CreateParameter();
param.ParameterName = "@emails";
param.SqlDbType = System.Data.SqlDbType.Xml;
param.Value = new SqlXml(xdoc.CreateReader());
command.Parameters.Add(param);

await command.ExecuteNonQueryAsync();
}
}

await Task.Run(() => Thread.Sleep(1));
}

2016-08-30

The server exploded.

As a warning, because I know the boys will tell you as soon as you walk in the door, the internet is down.

As I was debating whether or not I should leave work or try to get one more thing done first, I got a text from my wife saying the internet was down at the house. It happens from time to time. Usually, I just have to reboot something, and usually, that thing is the wireless router that most everything connects to. It's a Rosewill, which I bought mostly because my trusty Linksys WRT-54G was having a hard time keeping up with all the devices we kept adding to the mix. The Rosewill supports wireless N and has a much better signal range, but maybe once every week or so, things just go a little "wonky" and it has to be rebooted. Just this past weekend, in fact, it ended up "jamming" my home network completely, sending so much traffic (that wasn't actually going anywhere) that none of my machines could hear each other. I didn't immediately know it was the router at the time, but when I went to the basement to check the servers and all the networking gear, that's when I saw the rapid flashing on the switch connected to the Rosewill router.

I didn't expect it to be a big deal this time, either, but I got a couple more texts with some more details. Of course, my wife had tried rebooting the wireless router already. (Even the kids know that, sometimes, you just have to go over to it, pull the power plug, wait a few seconds, and plug it back in.) When that didn't help, she went down to the basement herself, and she heard some high-pitched beeping from what she described as a small box with blue lights on it. She turned it off, waited a bit, and tried turning it on again; and when it started screaming at her immediately, she just turned it back off.

From her description, I knew the device in question was the UPS. It seemed strange that the UPS would be beeping like that, unless the power was off and it was running out of batteries or something. It's a common story in tech support circles to get a call from someone who claims their computer doesn't work, and only after troubleshooting for a while does the clueless user say something like, "Well, I can't quite see, because the power is out and it's dark in here." I didn't believe my wife would fail to mention a power outage, though, so I figured it must be something else. The UPS going bad, perhaps? A tripped circuit breaker that cut the power to that outlet?

I got home and went downstairs to check things out. It was very quiet, which seemed like a bad sign. Two servers — the email server, and the main server that does just about everything else — are plugged into the UPS, but a third — the media server, which stores all our DVDs for easy access — is plugged straight into the outlet. If the UPS were bad, the media server should still have power. At this point, I'm thinking it's a tripped circuit breaker.

I go out to the power box to check the breakers, but none are tripped. I locate the one going to the servers and flip it off and on, just in case; then I head back inside. At this point, I'm getting a little concerned. We ran that power line ourselves; did we do something really wrong in the process? It's been fine for a few years, though. If it did go bad, what can I do to get power to that corner of the basement while we figure things out?

Back in the basement, the outlet still has no power. I noticed, though, that we installed a GFI outlet. Maybe that's what tripped. I pushed the red "reset" button, and as soon as it clicked, the media server hummed to life. Ok, that's what's keeping things down. Now, I'll bring things back up and see if it trips again, and then figure out what's causing the problem. I turned on the UPS, which gave only the slightest of beeps. The email server gave a soft beep as it got power, and then….

A little background on the main server. This thing runs pretty much everything. It is the only thing connected to the cable modem on one network card, and another network card connects to the switches that distribute internet traffic to the rest of the house. It was a machine I built several years ago, picking out the parts and assembling them myself. I didn't look for anything special in the case, but the one I happened to find on sale had some interesting LED lights on the fans and a clear side window, so you can see everything inside. I didn't even know about these features of the case when I bought it; I was just looking for something that would hold all the parts together for a decent price. Over the years, the server has been carefully configured to do everything I need it to. It has a web server, which is mostly used by my wife for her web design work. It has a minimal email server, which does some preliminary filtering before passing email on to my "real" email server inside the network. It does the firewall and routing, with some hand-crafted iptables scripts to make sure bits go where they're supposed to. It has a DNS server, which is configured to give easy access to important devices on the network by name, plus has the bonus of having a few hundred known advertising sites redirected to the address 0.0.0.0 as a convenient, network-wide ad block. (Fun fact: I tried to do the same with porn sites, but when I got a list of known sites and fed them into my DNS server, it promptly crashed. There were just way too many to filter out wholesale.) It also has a large file store with an FTP server used internally to back up, share, and keep files we want to hang on to.

Anyway, as the main server got its turn to power up, there was a series of three or four very loud POPs, accompanied by a bright flash that could be clearly seen through the case's clear side panel. Accompanying the popping noise, I shouted something that I don't quite remember. And then everything went quiet again as the GFI switch once again tripped and cut the power. A thin tendril of blue smoke leaked out of the power supply fan of the main server, and the smell of fried electrical parts hung in the air.

I went upstairs and told my wife the bad news. The server just exploded.

My wife helped me get the server unplugged (mostly because, even with the power cut, I was still a little terrified to touch the thing after what I had just seen), and I took it upstairs where I had more light and began taking it apart. There were a few cobwebs and a lot of dust inside, but no obvious sign of what blew up. Unfortunately, there's no real easy way to tell what may be good and what may be dangerous. Unwilling to risk frying any more components than necessary, I resigned myself to having to buy a new machine and rebuild.

I can only hope at this point that the hard drives are ok. The server contained four in total — two smaller ones that held most of the OS, and two larger ones that made up the file share, each pair in a RAID-1 array. But without access to the internet, downloading the appropriate installation media would be tricky. Not that I had a replacement server handy, anyway. First things first, find a replacement.

I took a quick trip to the nearest electronics-type store, that being Best Buy. I knew it was probably a long shot going in there, and, unfortunately, I was right. Plenty of laptops and costly consumer desktop systems, but nothing that would be good for a server. I wasn't willing to overspend on a system that wasn't suited for the task.

My next bet was Micro Center, which was a half hour away. Unfortunately, that, too, was a wasted trip. Pre-assembled systems were limited to the desktop and laptop variety. They do have a large array of components for building machines from parts, but, being perfectly honest with myself, I was not in a frame of mind to start piecing one together in a hurry. If I'm going to build something, I want to take the time to research, and really put together what I want for the best value. But I need a server, and quick. I figured Amazon is probably going to be my best bet.

In the parking lot of the Micro Center, I double-checked Amazon's site. (I had looked before I left the house, but I didn't commit to anything as I wanted to at least try to buy something from a local store that I could take home and start working on that night.) I found a couple possibilities, but my biggest issue was trying to find the internal specs on the machines. This mini-tower server looks like a good deal, but does it have the internal space and ports for four full-sized SATA hard drives? I don't know if it was because I was trying to use the mobile website, or if their site was really lacking that information, but I found it really hard to find. (I probably would have found more details on NewEgg, but I was wanting to take advantage of Amazon's better prices and faster shipping.) I found one that actually included a mention of "space for 6 drives" in the description, placed the order, and elected to pay extra for one-day shipping.

On my way home, I started to go over my options. I wouldn't be able to restore the web and file server until the new machine arrives, but what could I get up and running now? I had that old Linksys wireless router, which I had installed DD-WRT firmware on — meaning it is something that is very configurable and something I could really tweak. That, I figured, could take the duty of routing and firewalling for the internal network, and we would at least have internet access again. Email might be a bigger problem, though. Sure, the email server was alive, but the way I had it configured, I depended on the main server to filter email first. Maybe some of the security settings I had applied in the not-too-distant-past would allow me to grant it more direct access to the internet without becoming an open relay for spam mail. But that could be a secondary task.

I got home and set to work, hooking up the Linksys router in the place of the main server. I had some issues getting it configured, since my prior tinkering with the device (when it was just a toy to play with) had left it in a weird state. I ultimately had to reset it to its default state and rebuild it from there. DD-WRT has a very convenient web-based interface, though, and it took me much less time than I expected to get things to a working state. The thing that slowed me down the most was the fact that devices on the network still remembered their configuration from the main server, and didn't immediately update to point to the Linksys router when I brought it online.

With that accomplished, I figured I'd try setting up email. I did have a few small issues configuring the network, but again it came down to having to just reboot the server a couple times to force it to update its network configuration. I had some issues from there trying to get some external email server testing programs to talk to my email server, and that slowed me down a bit. It turned out that the email server didn't take too kindly to being forcibly rebooted, and the email services just plain hadn't started up. (It's amazing how much better things can work if the expected program is actually running.) I forwarded the secure email ports through the firewall easily enough, but I wasn't too sure about opening up the unsecured email port required to let outside email come in. It turned out much better than I expected. The security settings I had enabled recently were working perfectly. I ran a couple different open relay tests against my web server (which is something I always, always do when I tinker with the email server — last thing I want to do is to get shut down because my email server is sending out everyone else's spam mail), and it passed perfectly.

So, now I'm back up and running with internet access and email. The major items are taken care of, so everything else from here can get rebuilt on a much less rushed timeline. (Still want to do it quickly, but it doesn't have to be done yesterday.)

Time to count the blessings and see what I learned.

The biggest blessing is that nothing burned down. The GFI outlet tripped, but the UPS at that point should have still been providing power. Near as I can figure, it also detected something was wrong and cut power, then beeped as an alarm. When my wife turned it off and back on, it must have been able to still detect the problem and not try powering on the server. I'm not sure what changed when I got to it later, but when I tried turning things on and it started making loud boomy noises, the GFI tripped again and the UPS just shut itself off immediately. If it hadn't, there could have been much more damage done, and possibly an electrical fire as well. (I'm still keeping my fingers crossed that the hard drives aren't fried.)

We're up and running. Email and internet are the most important things we have to keep going, especially with one child doing homeschool and taking lessons over the internet. I pay for a backup email server that, when our server is down, will receive and hold our email in a queue until our server comes back online; so we haven't lost any email.

With the Linksys router doing the routing and firewall duties, I can rebuild the main server and just keep it behind the firewall itself, without having to configure it for routing as well. Whenever the server has an issue in the future, I won't have to bring down the whole network. Plus, keeping the file share off of the computer exposed to the internet is a better setup anyway.

I should probably look into offsite backups. While I can hope that the hard drives didn't get fried, if it turns out that they did, I could be up a very smelly creek in a barbed wire canoe without a paddle.

2016-07-21

When there's no will, there's no way

My relationship with my father can be described as "strained", at best. Growing up, I don't remember him being around much. He worked hard to make sure we never really wanted for anything; and though it meant we did have a very comfortable lifestyle, it did mean we didn't spend a lot of time together.

I've never been very good at communicating. I think there are many reasons for this, but I don't want to get into them for fear of being accused of trying to place blame. Even now, as a fully-grown adult, I get very anxious when I think about making a simple telephone call. I also tend to be forgetful and more than just a little lazy. My ideal vacation consists of one where I never leave the house. Not that I don't enjoy the occasional family trip once in a while, but I don't really ever fully relax until I can get back home.

My father has never accepted these aspects of me, though. While it could be considered noble to want me to be a better person, he's always taken my failings as a personal affront to him. If I don't call, it's because I don't respect him, and I'm a horrible, ungrateful little brat because of that. And he takes that as license to treat me as such. Which makes me more likely to act that way, and the cycle spirals downward.

My biggest issue, though, is that I am always the one to shoulder the blame, and the punishment. And, sometimes, that punishment borders on revenge.

Many years ago, we were having issues with email spam. An email address we had set up was getting hundreds of messages a day. We decided to change our email address to try to hide from some of these electronic assaults. I think that might have been the time when we set up our own email server, so that we could create new email accounts at-will and give a new email address to every website with which we were inclined to register. This is a huge benefit in that it means, when we start getting spam, we can identify exactly whose email list was hacked or sold, and we can terminate a single address without affecting our communication with any other person or business.

Anyway, we sent out notices about our email address change, and after a couple months, deactivated the account. I am not certain if we failed to send such a notice to my father, or if he failed to read it and update his address book (though I'm leaning heavily towards the latter, for reasons I will explain later), but eventually, he noticed that emails he was sending were going unanswered, and eventually started to bounce.

I'm not sure how long it was before he eventually got in contact with us, and we told him (or reminded him) of our email address change. I'm not sure how we failed to communicate this, because apparently what he heard was: "We changed our email address and didn't tell you so we wouldn't have to hear from you." And, being so affronted, he then stopped sending birthday or Christmas cards for a while.

This was annoying for two reasons. One, it was clearly a misunderstanding. Somehow our change of address didn't "take". Although we were reasonably sure we told everyone, we accepted responsibility, apologized, and made sure he had our correct email. A reasonable response would be to accept that it happened, it's been resolved, and move on. But his response was to be angry that it happened, accuse us of doing it on purpose, and to punish us by not sending Christmas cards anymore. Two, he went beyond us. Not only did he stop sending my wife and me anything, but he stopped sending birthday cards to his grandkids, too. He was punishing them for what he perceived as our "attack" on him.

This was a very long time ago, so I don't remember exactly when things might have turned around. It might have been when my son was admitted to the hospital a few years back, and I decided to call him and let him know. But in any case, we had been back to sending cards and gifts for birthdays and Christmas again.

But moving forward to a few years ago. He and his wife came to visit around Thanksgiving time, as they decided to take a vacation to the Rocky Mountains and would stop by for a couple days. They stayed for a day, and we chatted for a while in the living room, but he wasn't feeling well and ended up spending most of the time back at his hotel to get some rest.

The next month, he sent a Christmas card. Since he handed us a check when he came to visit, I didn't expect there to be anything inside. Which I'm not complaining about. In any case — as with all Christmas cards and checks we get from relatives that time of year — I didn't open it right away and put it on the tree, for opening on Christmas day. We had also sent him a package, with some hand-made quilted wall hangings that Karen had been busy making for relatives all that previous month.

Christmas day came, we opened our gifts, and I opened the card to find a rather sizeable check inside. We were pretty thankful for this. We knew he was planning on having a big family gathering at his house that next summer, and we were planning on going, but we were a little concerned about the logistics of making such a trip with four kids on a budget. (I have not done as well for my family as he did for his when it comes to finances. We've always had "enough", and in many cases even some "extra", but I don't know that I would ever consider us "affluent". Though maybe that's because my only real comparison is with how much my own father provided us.) I put the check aside, with the intent to deposit it later, and, as is typical for me, didn't think about it for a bit.

A couple weeks had gone by, and the checks were still on my "to-do" board, since I had forgotten to grab them on my way out to work every day since. (Again, typical me. If you ever send me a check, and you need it cashed right away, please call me and I'll make it happen. Because it's not unusual for a check to be sitting in my "out box" for a month before I remember it's there. I've had our church's clerk come to me and ask me to deposit a reimbursement check for things we've bought for the scout troop on more than one occasion.) So I receive this email:

We sent you a Christmas card with an extra check to maybe make your Christmas a little happier. We didn't hear from you for Christmas or New Year's which is not unusual as you hardly ever call. But I am concerned about the check being lost in the mail and I am going to put a stop payment on it. If you did receive the card and still intend to deposit the check, please let me know before Monday.

Ok, oops. I hadn't deposited the check yet. Need to rectify that. I'd better reply.

Yes, we sure did, just haven't remembered to take the checks to the bank yet. Thanks so much! We're going to be putting that away for our trip out there this summer. Looking forward to it.

How was your trip up to Harvard? I heard there was a big storm that hit New England around that time.

Oh, but he won't have any of that "cordiality" or "small talk".

Unbelievable. You would think I would not have to solicit a thank you, or a Merry Christmas.

And you'd think he would know by now that I'm just not that good at replying. But the sound of his emails imply that we made zero communication at all. Notice how there's no mention of the gift we sent him (that Karen had been stressing over making the prior few weeks), which did include a "Merry Christmas" card enclosed. I don't know whether to be concerned or angry — concerned that our package didn't get to him, or angry that he got it, didn't say anything about it, and tries to lay the guilt trip on me for not saying anything. I'll try to keep things cordial by assuming the first, and copy his wife just in case she got it but forgot to tell him.

Well, now I'm a little concerned. Did you not get our gift to you? Karen spent a lot of time and effort designing and making it. We sent it to the Alabama address around the second week of December. Did it arrive? Vail, have you seen the present we sent?

Enough already, I'm not going to play your game. Yes we got the Christmas present you sent and thank you. And, If I recall correctly, you said thank you when I gave you the Christmas card when we stopped to see you in November. I guess I thought that a check for [amount redacted] would be something a little extra special and something you might pick up the phone and call and express your appreciation, or call on Christmas, or on New Years. I know I don't call, I stopped several years ago because I felt communication with you was a one-way street.

Maybe our feelings toward each other are mutual. Maybe you think I am rude, inconsiderate, and ungrateful for anything you do. That is the way I feel about you and it just isn't worth it to me to try anymore. Please take the check and put it to good use for your family. You don't need to think I sent it to you so you could come visit us.

Yes, see, I forgot the rules of the "game". The rules that say I'm totally at fault for everything, and how dare I accuse him of not taking some of the responsibility himself. He claims to want to talk to me, but he'll be damned to be bothered to actually pick up the phone himself and call, because he doesn't feel that I don't call him enough. Which, yes, I don't, because that's just me. Yet he takes it as some personal slight to him, and because I don't live my life according to his rules, he can't be bothered to try.

I will give him credit for the masterful way he phrased this, though. He managed to call me rude, inconsiderate, and ungrateful, but worded it in a way that put the blame squarely on me for thinking that's what he thinks of me.

But, at least he did address my concern — our hand-made Christmas gift did, in fact, arrive; so it was just his own hypocrisy at work here. "How dare you not thank me for my gift! I don't have to thank you for yours, because you wouldn't appreciate it anyway!" But now I'm starting to wonder about this upcoming trip. Do I really want my kids to be exposed by this kind of treatment? Maybe he doesn't want me there, anyway. Of course, he's unlikely to admit to any such thing, instead making it so that my failure to attend is my decision and fault anyway. Perhaps he would answer a direct question.

No game was being played. Your previous message seemed to imply that you received absolutely nothing from us with regards to Christmas, and we were concerned that what we did send didn't make it. But I’m glad to hear that it did arrive.

You have always made your feelings for me abundantly clear. You can rest assured that none of this was any revelation.

We were legitimately looking forward to using the money you sent for coming out to visit, but the end of your message makes it sound like you don't want us to do that. We are willing to come, but it is your home, so I will leave the decision entirely up to you. Do you, or do you not, want us to come and visit this summer?


Don't bother.

Your actions have been abundantly clear since you've been a teanager and I don't know that there will ever be anything I can do to change your feelings toward me, either. Just like my father told me a few years ago,he did for me and I did for you what I thought was best for you.

Have a good life. I'll stay out of it.

I'll be there if you ever appreciate or respect me as a father, all be it not a perfect one. But I am not going to continuing doing things that are not acknowledged or appreciated.

This will be my last communication.

"Don't bother." As close to a direct answer as I could expect.

The cry to appreciate or respect him as he is, is, once again, hypocritical and insulting, when he obviously has no respect for the kind of person I am. You know, "all be it not a perfect one." And I really hope I don't harbor resentment for my own kids the way they act as teenagers the way he does for me. I'm sure he didn't act out at all when he was a kid, either.

I replied to make it clear that I refuse to accept total responsibility.

As you wish. If you ever decide to change your mind and treat me with the level of respect you seem to expect from me, you know how to reach me.

I did deposit the check, only to get a call from my bank a few days later to inform me that he did stop payment on it. I guess he really didn't want me to "take the check and put it to good use for your family", though I'm more inclined to think it was his reply to my last email (which he couldn't actually write since he already said it was his "last communication"). He had to get the last word in somehow.

A couple years later, we took a trip to visit family. When we got to my father's sister's house, we got a lot of admonishments as to how horribly we were treating my father, and how I should call him. Apparently, he has spent a lot of time telling his family what a horrible son he has and how I'm exiling him from our and his grandkids' life. Up until this point, I have not discussed our relationship with anyone in the family — I did not feel it was anyone else's business and didn't feel the need to burden anyone else with our "issues". Besides, I don't communicate much, anyway. My father, on the other hand, obviously did not feel the same. We had to spend a lot of time explaining our side of things while we were there.

I can only assume that some of our conversations with the family got back to him (one of our biggest complaints being how he seemed to punish the grandchildren for our disagreements or misunderstandings, when he severed all communication), because he actually sent us a card for the next occasion. We sent him one back, and, later, he mailed us a thank-you note that he had tried to email to us. Remember when I suspected that he just didn't update his address book when we changed our email? Yeah, the thank-you email had bounced back from an email service we hadn't used in well over a decade now. Apparently, if he wants to chide or berate me, he has no problem using the correct email address; but when it comes to sending something nice, suddenly he can't use the right address?

We continued to send cards back and forth. (Though not gifts; if he's not going to acknowledge the time and effort we spend into making and sending gifts, it's not worth sending. See, I did learn something from him after all.) We also made sure to send a thank-you card for every gift he sent us and the kids, since that seemed to be very important to him. But it was very obvious he had no interest in talking to us, just the grandkids, when birthday cards would come in for each of the kids and our days were conspicuously skipped. We had almost missed Father's Day, so we sent a quick e-card and, respecting his apparent wishes, "signed" it from the grandkids.

Apparently, the absence of my and Karen's names on this email was the last straw, as this email came in addressed to Karen:

Please inform my son that I have an appointment with my estate attorney in a few days with the purpose being to take him out of my will. I don't know what I did that was so bad that has resulted in his lack of any respect for me. I do believe I was not half as hard on him as my father was on me and I do believe had I not been hard on him he might not have been what he is today.

I intend to send birthday cards and gifts through the end of this year. After that, it just isn't worth it to me to make any more effort.

TL;DR: How dare you snub me after I snubbed you! I quit!

So, I suppose, that's it. Because he makes efforts to punish me for not being the kind of person he demands me to be, and because I don't kowtow and give him the love and adoration and respect he demands while he treats me like dirt, I'm not worth his "effort".

Some of my reaction may seem harsh with the limited discussion here, but this is just a small sampling of my interactions with him. And most of this has occurred after I have gotten tired of the beating down, bullying, and lack of respect as a person that I've endured for many years. He was a very toxic influence on my emotional well-being. I've tried to maintain the lines of communication so as not to cut him off from his grandchildren. But I have felt no guilt for declining to go above the minimal effort required when nothing is ever good enough.