2014-07-17

MVC, JSON, and DateTime (oh my)

It's been a while since I've had a programming issue that warrants a blog post, but here's an interesting one.
First, let me set up the situation. I have an MVC project (MVC version 5.0) that communicates to RESTful services using WebAPI (version 5.1). In one particular view, I write out an object in JSON with the intent that the client can make changes to it and POST it back to an MVC action to get an HTML table back. Here's the model:

public class PaymentScheduleRequestModel {
[JsonProperty(PropertyName = "effDt", NullValueHandling = NullValueHandling.Ignore)]
public DateTime EffectiveDate { get; set; }

[JsonProperty(PropertyName = "termId", NullValueHandling = NullValueHandling.Ignore)]
public Nullable<int> ContractTermId { get; set; }

[JsonProperty(PropertyName = "billDay", NullValueHandling = NullValueHandling.Ignore)]
public int BillingDayOfMonth { get; set; }

[JsonProperty(PropertyName = "freq", NullValueHandling = NullValueHandling.Ignore)]
public PaymentScheduleFrequencyId PaymentScheduleFrequencyId { get; set; }

[JsonProperty(PropertyName = "prem", NullValueHandling = NullValueHandling.Ignore)]
public decimal TotalPremium { get; set; }
}

(Note that the JsonProperty decorators exist because the same model is used to communicate to the WebAPI service -- we are using shorter property names to lighten the payload.)

And the MVC action that is set up to process it looks like this:

[HttpPost] public async Task<ActionResult> NewPaymentSchedule(PaymentScheduleRequestModel request) { … }

To put it in JavaScript, the view has this block of code:

window.ScriptModel = @Html.Raw(Json.Encode(new {
PaymentScheduleUrl = Url.Action("NewPaymentSchedule", "Contract"),
PaymentScheduleRequestModel = new PaymentScheduleRequestModel {
ContractTermId = Model.ContractTermId,
EffectiveDate = Model.StartDate ?? DateTime.Now,
TotalPremium = 200m
}
}));

Unfortunately, Json.Encode doesn't work well with dates. This is the output that the browser sees (with line breaks added for legibility):

window.ScriptModel={
"PaymentScheduleUrl":"/Contract/NewPaymentSchedule",
"PaymentScheduleRequestModel":{
"EffectiveDate":"\/Date(1078729200000)\/",
"ContractTermId":null,
"BillingDayOfMonth":0,
"PaymentScheduleFrequencyId":0,
"TotalPremium":200
}
};

Notice two things about System.Web.Helpers.Json.Encode's output:

  1. The property names are the .Net property names. The same code could not be used to post what should be the same model to MVC or to WebAPI.
  2. The EffectiveDate field has been converted to a Date function, enclosed in a string. Even if the Date function would reveal the correct value, the fact that it's in a string means JavaScript will not see it as a Date.

If I don't do anything with this object and just post it back to the MVC app, the resulting object does not have a valid date — all other values carry over, but the EffectiveDate property is 01/01/0001. So not only does it look odd, not only is it inconvenient in that JavaScript can't use it as-is (without picking apart the string), but it doesn't even work for round-tripping data to and from the client.

In doing some research on this topic, I came across Scott Hanselman's blog post describing the problem, and stating that the release of WebAPI won't have the issue since it will use JSON.Net (a.k.a. Newtonsoft.Json). Since I'm using the same model in an WebAPI call further downstream, I can verify that it does work as intended. It uses the JsonProperty decorators to rename the properties, and it serializes and deserializes like magic.

To solve this problem in MVC, you have to alter how it deals with JSON on the way out and on the way in.

On the way out is easy in my case, since I am manually spitting out JSON into the HTML. I just exchanged Json.Encode with Newtonsoft's serializer:

window.ScriptModel = @Html.Raw(Newtonsoft.Json.JsonConvert.SerializeObject(new {
PaymentScheduleUrl = Url.Action("NewPaymentSchedule", "Contract"),
PaymentScheduleRequestModel = new PaymentScheduleRequestModel {
ContractTermId = Model.ContractTermId,
EffectiveDate = Model.StartDate ?? DateTime.Now,
TotalPremium = 200m
}
}));

And the browser sees:

window.ScriptModel={
"PaymentScheduleUrl":"/Contract/NewPaymentSchedule",
"PaymentScheduleRequestModel":{
"effDt":"2004-03-08T00:00:00",
"billDay":0,
"freq":0,
"prem":200
}
};

This is better. Of course, the controller action doesn't understand this. It's still looking for the .Net property names, and, since they don't exist on the incoming object, all values come back empty. (In this simple example, not only is EffectiveDate 01/01/0001, but TotalPremium is 0.0.)

The trick here is to override MVC's default model binder, so that it, too, uses the Newtonsoft.Json library. It is also consistent to have MVC use a value provider factory that also uses JSON.Net.

Fortunately, people smarter than I figured out these two steps. I found a value provider factory on this blog: http://www.dalsoft.co.uk/blog/index.php/2012/01/10/asp-net-mvc-3-improved-jsonvalueproviderfactory-using-json-net/
and the important piece, the model binder, that will translate the JSON property names to their real .Net names, is detailed here: http://stackoverflow.com/questions/4164114/posting-json-data-to-asp-net-mvc

My implementation looks like this:

And this code gets called from Application_Start (I actually added it to my WebApiConfig class (which, for some reason, exists in my MVC app even though it's obviously not the same as WebAPI), since other configuration-type things were being done here):

ValueProviderFactories.Factories.Remove(ValueProviderFactories.Factories.OfType<JsonValueProviderFactory>().FirstOrDefault());
ValueProviderFactories.Factories.Add(new JsonDotNetValueProviderFactory());

ModelBinders.Binders.DefaultBinder = new JsonDotNetDefaultModelBinder();

All this because, although Microsoft updated one of their web interfaces (WebAPI) to use the JSON.Net library that works, another one of their interfaces (MVC) uses their own, broken, JSON serialization library.

2014-04-30

Consequences of free speech

A lot has been made recently of Los Angeles Clippers' owner Donald Sterling, being caught on tape saying some pretty racist things, bad enough for the NBA to ban him for life, fine him $2½ million, and force him to sell his team. From what I've heard, this result seems like a net positive for society as a whole, but how this whole thing has come about makes me very worried for the direction this country is headed.

According to the reports I've been reading, Sterling is not a very nice guy. Some say he has a long history of being racist. By the sound of things, the NBA will be better off without him. And yet, the NAACP gave him an award back in 2009 because (at least in part) he gave away a lot of tickets. The more cynical reports suggest he was given this award, with his racism overlooked, because he gave money to the right people. He was, in fact, due to receive another award this year from the NAACP, before recent events made the organization reconsider.

But what were the "recent events"? Did he use his power as an owner in real estate to deny housing to blacks or hispanics? Did he make an employment decision based on race? No, that was years ago (before, during, and after the NAACP was giving him his first lifetime achievement award). It happened because his girlfriend recorded a private conversation, coaxed him to say what he did, and sold the tape to the online tabloid TMZ.

I've read conflicting comments as to whether or not Sterling knew he was being recorded, but those that care to mention the girlfriend seem to agree that she really worked on Sterling to get him to make his damaging statements, leading him to say what he did and really dragging it out of him. Of course that brings a lot of rumor and speculation about how much, and from whom, she was getting paid to do this, and whether or not it was illegal (most likely if Sterling didn't know the tape was rolling).

Kareem Abdul-Jabbar wrote an excellent opinion of the issue for Time, which echoes a lot of the comments I've heard that led me to my conclusion above. It also comes to pretty much the same end — Sterling's eviction from the NBA is a good thing and long overdue, but how it happened is very, very wrong.

The big question, of course, is, did Sterling deserve his NBA exile, fine, and loss of ownership of his team for what he said? Maybe his speech just served to bring about awareness of what he was actually doing, but because he only got punished when he expressed his views in the privacy of his own home, it certainly looks like that's his sin. He said something that was unpopular, and now he must pay.

It is true that "freedom of speech" does not make you free of consequences. If you say something that someone doesn't like, that person is just as entitled to use their own freedom of speech to speak out against you, exercise their freedom of association to refuse to do business with you, and even freely encourage others to avoid you as well. But why does it seem like it's only the speech — or even the perceived interpretation of speech — that deserves to be punished, when actions can go unnoticed or be forgiven?

We've been sliding down this slope for a while. Paula Deen was condemned for using a racial slur in the past (including in a description of someone who had a gun to her head), despite having apparently changed her ways since. Phil Robertson lost his place on the show Duck Dynasty for daring to honestly answer an interview question asking what he considered a sin (only to be reinstated after enough public support). Brendan Eich found he was unable to do his new job as Netscape CEO due to all the protests that, six years prior, he gave a paltry (for him) sum for California Proposition 8 — a view that, even though is becoming less so now, was the popular view at the time (Prop 8 passed) and, at the time, was even supported by presidential candidate Barack Obama. Cliven Bundy was supported by many in the (mostly right-wing) media, before he expressed his political opinion that came off as racist (though that might be less actual racism and more poor word choice and selective editing by the media, depending on your point of view).

So now, it's not just what you say in public. It's not even what you said in public in the past, whether or not you've recanted since. But what you say in private, behind closed doors, can and will be used against you in the court of public opinion; and if your opinion goes against the prevailing wisdom, whichever way it happens to be blowing at the time, then God help you.

2014-03-13

My New Year's Resolutions

I try to avoid making New Year's Resolutions. They tend to be some big life-changing or life-enhancing promises that start with really good intentions, and then fall flat within a month or two. Personally, I just can't see myself deciding to completely change my habits or lifestyle overnight. But this year, I decided there were a couple things I could do better. They aren't major issues, but at least it's something I can do and accomplish and feel like I've done something to improve.

The first is a bit of housekeeping. Our HVAC system has stainless steel air filters (rather than cheap cardboard-framed filters), and they are supposed to be cleaned out periodically. Historically, I've been very bad at cleaning those filters, going way back to when we first had the system installed. I remember cleaning them one day and then being very surprised that I actually saw curtains moving in the breeze from the air vents. Our house also tends to be very dusty, which I don't doubt is at least partially the fault of the HVAC's inability to move air through the completely clogged filters. So, I resolved this year to incorporate into my Sunday routine, cleaning out those air filters.

For the month of January, I did manage to clean the filters out every week. At least in those first couple weeks, it really needed it, as the filters were pretty well coated when I pulled them out. But by the time I got to February, the filters were still pretty clear when I went to clean them. So, I've backed off a bit on my initial resolution, instead cleaning the filters out every other week. Even then, they're not catching nearly as much dust as they were, so I feel that I'm reaching my ultimate goal of keeping them clear so the heater & A/C will work efficiently. I'm not certain it's cut down on the dust that gathers in the rest of the house, as that's pretty hard to quantify (especially when we just simply don't dust around here), but I like to think so. I was hoping to see my energy bill go down a bit as a result of the better air flow, but that, too, failed to come to pass. Still, I can at least pretend that I'm doing the right thing.

The second has to do with my church. I believe in the Law of Tithing, where I give 10% of everything I earn back to the Lord. The way I've kept track of what I should pay, on every increase, I enter a transaction in my financial software with a future date with the 10% amount. It's always there at the bottom of my register, so I always know how much I owe; and I can always make sure I don't spend too much such that there isn't enough for the tithing amount. If I forget to take a check to church one week, I just keep adding to that transaction in my register, until I get around to writing the check and giving it a real date.

I thought it was a pretty good system, but a couple things bothered me. One, I noticed in months where money was tight, I would find myself "borrowing" from that total to pay the bills. Oh, I wouldn't change the amount, but I would ignore the fact that the tithing transaction was actually making my balance negative, because it would be fine when the next paycheck showed up. But I found that it seemed to happen more frequently and with greater amounts the longer I left that transaction in there unpaid (not to mention the tithing amount was growing over time as well).

The other thing occurred to me as we discussed tithing in church one Sunday. While it's relatively easy for us to pay tithing now by just writing a check, in the past it involved real items — for example, agricultural farmers would bring food from their harvest, and livestock farmers would bring animals. They always referred to it as bringing the "firstfruits" of the harvest or the flock, because it was a sign of devotion and respect to God that we give Him the best portion of what He gave us. When I applied this to my system of reserving money in my register, I realized that what I was doing was closer to making sure everything else was paid first, and then giving God what was left over. In practice, I had it backwards.

My resolution for tithing is to pay as soon as possible. That means, on Saturday night, I check to see if I need to pay anything for tithing (since I don't really think about which weeks are the pay weeks and which aren't), and write out the check, so it's ready to turn in the very next day. It just feels like the right thing to do, and I feel better about doing it this way.

So yeah, that's it. Those are my New Year's resolutions. Nothing earth-shattering or anything, but I feel like they are realistic, attainable goals that I can actually keep. And now, here they are in writing, so I can hold myself accountable.

2013-11-21

A no good very bad end of the day

So I went to a karate class this evening. Usually, I just wear my uniform pants (fairly thin), a t-shirt, and flip-flops. Tonight, my wife was helping out with a Cub Scout activity, and since she had to leave before I would be home, she would have to take the kids with her, and I would come by and pick them up after my class. I figured I would at least bring a set of sweats with me, so I wouldn't show up to get the kids just in my sweaty karate gear.

On my way to the church after my class, I went to turn on the street. I wasn't going very fast, but the road was very icy, and my car just wouldn't stop. I slid right into the curb, there was a loud smack, and then my car refused to drive. I got out and checked the damage, and found the tire that hit the curb had been driven right into the wheel well, pretty much jammed in there.

I didn't think to take my phone with me to karate, so there I was in sweats and flip-flops, just under 20°F outside, with no way to call for help.

Fortunately, I was only about three blocks away from the church (blessing number 1, I suppose), so I set my hazard lights on, left the car, and started walking. I got to the church, found my wife, grabbed her phone, and left her to her business while I walked back to the car.

The company for whom I work happens to provide roadside assistance plans, and one of the perks of my employment is enrollment in that plan (blessing number 2), so I called for help. I'm not sure if I had a bad signal or not, but the agent on the phone seemed to have a very hard time understanding the road names I recited and spelled for him several times. Then he told me that a tow truck would be there in about an hour and a half.

So there I sat in the car at a very icy intersection, waiting for help. How it is that no other car hit me coming around that corner, I'll never know (blessing number 3). I was at least able to start the car periodically to run the heater so I didn't freeze. I also happened to find a pair of socks that my son had left in the car, and that I had asked him to take out more than once. They weren't clean, but it was better than just wearing flip-flops. Once I had the socks on, I realized I had a spare pair of boots in the car. They're not a perfect fit, but I keep them there for emergencies. (Looking back, I wish I had realized that earlier and put them on, even without socks, before walking three blocks back and forth.)

While I was waiting, the tow truck dispatcher called saying he couldn't find the streets on the map. No wonder, since the agent had gotten them very wrong. I was able to give him directions at least.

Help finally arrived. Somehow, the details of my problem got mixed up, as he was expecting to just winch me out of something, but fortunately he didn't have an issue giving me a tow. I just had him tow me back to my house (not even a quarter mile away; I could have just walked it, if it didn't mean leaving my car in the middle of an icy road where it would be likely to get hit). I had a shop in mind, but I figured it'd be easier to get all the details together and set up the appointment in the morning.

So at least I'm home. Though with money already tight, I'm not looking forward to trying to figure out how to pay for this repair.

A shame. I mean, it wasn't that bad of a day up until tonight.

2012-12-13

Animaniacs Volume 4 is coming!

If you know the origin of my nom de plume, you'll also understand why this news is very exciting to me.

The 3rd volume of the 1990s cartoon show Animaniacs was released back in 2007, but it did not complete the series. Fans (including myself) have been waiting for the fourth volume to be released with the remainder of the series, but Warner Bros. has been sitting on them for the past five years. My kids have developed an appreciation for the show (and Pinky and The Brain, whose full 3-volume collection has already been released, save for the forgettable Pinky, Elmira, and The Brain fiasco). There are some real gems in the missing 4th volume, like the excellent parodies The Sound of Warners, Cutie and the Beast, and Jokahontas, that I've only been able to relive via poor-quality uploads on YouTube. Now I'll finally get to see them and share them with my kids.

This news apparently came out two months ago. I don't know what prompted me to search for evidence of Volume 4, but I'm glad I did. And with a February release date, I know what's going on my birthday wish list!

2012-12-05

Switches Gone Wild

Had an interesting network error the other day. Very suddenly, while I was paying bills online and my kids were watching Doctor Who on Netflix, the network was completely non-responsive. My first thought was that I had lost my connection to the internet again. But as I tried to connect to my server to check, I was unable to connect to that machine, too. Even when my internet connection drops, I don't lose connection to machines on the internal network.

I went downstairs to check the server. It appeared to be running normally, and I was able to log in to the machine using the keyboard and ancient CRT monitor I keep plugged in for emergencies. I tried resetting the network stack. Curiously, it threw up an error message trying to bring up the network card for the internal network, something about an inability to allocate memory. But the external interface came up without error. I tried a second time, and again, the internal interface still couldn't connect to any other machine.

Thinking there shouldn't be anything wrong with the network adapter itself, I spent quite a bit of time checking my iptables rules. I assumed there must be something in there preventing packets from the internal network from getting processed. Perhaps I had set the wrong rule when I was trying to get a couple new devices to play nice with the email server. But why would it work fine for a while and only now, many days later, decide to drop internal traffic? And why couldn't I fix it? Even when I dropped all filtering rules, I couldn't get any traffic through my internal network to that network card.

Not long ago, my computer failed to get an IP address from my ISP, and for whatever reason, it wouldn't get one until I gave up and rebooted the server. Figuring this was another symptom of whatever the problem was before, I just rebooted the server. It came up again, with only errors I've seen before (fairly obscure warning messages that haven't before caused any noticeable issues). It acquired the ISP's network quickly enough, but still, the internal interface didn't appear to respond.

I finally decided to take a close look at the network switch. One light was flickering rapidly, which was on the port connected to the entertainment center. This seemed odd, since the only devices on at the time were the TV (which does have an Ethernet port, but its features have been pretty useless so far) and the Blu-ray player (which was streaming Netflix before the network crashed, had been powered off by the kids, and was currently displaying a screen complaining about a lack of network connectivity since I turned it on to check).

Just to check if it was a bad port, I pulled the cord for the entertainment center and plugged it into an empty port. The light on that port came on solid, then started to flicker rapidly as before. I then unplugged the switch's power cable, waited several seconds, and plugged it back in. Each of the lights cycled in sequence as the switch went through the startup sequence, then all lights on ports connected to live devices turned on solid. Some started to blink, and the one connected to the entertainment center started to flicker rapidly again.

I went up to the entertainment center to check on the switch there. The lights on ports connected to the TV and Blu-ray player were on solid. Only the light on the port connected to the wall (and back down to the entertainment center) was blinking, and it was flashing rapidly.

This was seriously odd. No lights connected to any computers or devices were blinking with any intensity — only the lights on the switches that connected to each other. Were the switches generating their own traffic, talking back and forth to each other? These are fairly inexpensive, unmanaged switches, with no network address of their own to speak of; what could they possibly be saying to each other, and how?

I unplugged the power cord on the switch behind the entertainment center, waited a few seconds, and plugged it back in. All its lights came on briefly as it powered up, and then finally, mercifully, the lights on all connected ports came on and stayed solid, including the light leading to the switch on the server. I checked the Blu-ray player and my laptop, and both were able to connect to the server and, by extension, the rest of the internet.

I still have no clue what was causing all that traffic. I did notice that, when the Blu-ray player started streaming Netflix again, the lights on both ports on the switch that connect to the wall and the Blu-ray player were flickering quickly, looking much like the flickering on the one port when things weren't working (although there's no way to tell by sight if the flickering was exactly the same). Near as I can figure, there was so much of this mysterious traffic that it jammed the main switch so thoroughly that no traffic could get through any other port. (Since they are both 1Gb switches, it certainly could do it. The only other gigabit network device on my home network is the network adapter on the server facing the internal network, but the light on the switch on that port wasn't showing anything but the most rudimentary activity.)

2012-11-28

MVC: Any object you want, but not a string

I often refer to MVC — the Model-View-Controller pattern for building websites — as "Magic, Voodoo, & Conjuring", because a lot of things just happen by what seems like magic. Moreover, when something goes wrong, it's very difficult to find exactly where it went wrong, because the process is happening without writing any code.

I had one such example of something going wrong today. It was only after I got another pair of eyes to confirm I wasn't crazy that I found a StackOverflow question that addressed my question.

We had a JavaScript method that took a method name, a key, and an object, and posted that to an AJAX handler on the server, more or less like so:

$.ajax({
    url: "/Home/" + methodName + "?key=" + keyId,
    contentType: 'application/json',
    data: $.toJSON(dataObject),
    success: function (result) { doSomething(); },
    type: "POST"
});

This JavaScript code block was part of a plug-in that we wrote to apply to many different forms on the site, that shared common functionality. Depending on the data being processed, the dataObject could be anything from a string to a full-blown object. The controller methods all had similar signatures:

public ActionResult ProcessStringData(string key, string newData);

public ActionResult ProcessDataEntity(string key, DataEntity newData);

When the AJAX call was made, the querystring parameter key was correctly mapped to the method parameter key, and the JSON object in the request payload was magically deserialized and mapped to the newData parameter.

The problem was (as you may have guessed from the StackOverflow question), while this worked great for the object, it was completely ineffective for the string — despite the fact that the data was in the request input stream, the value would always be "null".

This is why it was such a head-scratcher. If the data was a DataEntity object and we were calling the ProcessDataEntity method, the AJAX payload would simply be:

{ prop1: "value", prop2: "value2" }

Note that nowhere in that text is the name newData, and yet the MVC framework somehow managed to interpret it as a DataEntity object and pass it as the newData parameter.

Calling the ProcessStringData method with a string as data, however, resulted in this payload:

"data value"

So why wasn't it magically treating this as a string and assigning it to the correct parameter?

Maybe the JSON deserialization magic was throwing an unseen error. Using Fiddler, I tried submitting a payload of { "data value" }. This resulted in a very visible error response from the server about an invalid JSON object. Since "data value" did not return the error, the deserializer must've been relatively fine with it.

Curiously, if I changed the payload to:

{ newData: "data value" }

it did, in fact, work. However, I didn't want to have to wrap a simple string with a named parameter object to get it to work — it would be inconsistent with the object-processing methods that did not require this massaging. (Consistency makes for much more maintainable code. Try bringing up a new developer to speed and explaining that data is in a certain format, except in a list of "special cases" that you're lucky if you remember to mention, and you discover this truth quickly.)

Looking through StackOverflow, it seems that it is possible to remove the content type to process strings. Again, this would have required determining if I was calling a string-based or object-based method and setting the content type (because I actually did want the application/json type for objects). It just didn't "feel" right.

I ended up wrapping the payload in an array. For whatever reason, it is unable to take a string payload and map it to a string parameter, but it is perfectly fine mapping a payload of the form [ "value" ] to a parameter of type List<string>.

It's just one of those things that requires some coding around, but the hard part is trying to understand why. And because it's all handled by magic, voodoo, and conjuring (a.k.a. some method buried deep inside the framework), it's not in your working code, and therefore very difficult to track down.