Internet Explorer 6 problem with character sets

I recently encountered a problem with the membership sign-up form on the Artistic Network site. The form had been, overall, in use on other site's I've built for a few years. There never seemed to be much of a problem. But then suddenly I started getting complaints that a few members could not sign into our site.

I started getting emails that said, "I tried to sign onto your site, but I kept getting a notice that my first name was missing. I entered my first name again and again but the error kept happening."

The users received the notice because the server that processes the sign-up checks to see if all required information is present--and indeed first name is required. The script on the server was working fine and as expected--but for some reason the first name field in the form was never even reaching the server. I knew this because I came to learn this was only happening on Internet Explorer 6 running on Windows XP with Service Pack 2. Firefox users did not have this problem.

I had several suspicions at first. My first two inclinations were that an errant JavaScript was somehow deleting the first name input field or that special hidden characters were somehow choking the form submission. As it turned out, the second issue is what was causing the problem.

To find the error, fellow developer Andrew Penry and I started playing with profiles on the site that would not update using Internet Explorer. We indeed verified that certain member profiles would not update using Internet Explorer--but those same profiles updated fine in Firefox. Deleting chunks of profile text at a time led us to discover that text displaying special symbols like smart quotes, em dashes, umlauts, etc. were choking the form submission. Removing them from the textarea fields saved our script.

My first workaround was to create a new hidden field and altering the JavaScript that verified forms to copy the field from the first name to the new field. That way, on the server, we could use the backup first name if the proper field was missing. This solved our problem, but it still left the mystery as to why this was happening. Andrew suggested we insert a dummy hidden field at the beginning of the form--before the first name field. Indeed, this action solved our original problem--the first name field now always showed up. However, forms submitted with special characters were now missing the new dummy input.

Our find did not answer why this was happening, but exposed a strange bug in Internet Explorer on Windows XP running Service Pack 2. Andrew considered the fact that our site was using the iso-8859-1 character set, while Service Pack 2 defaults the character set to UTF-8. While this default should have been overruled by the html header declaration of iso-8859-1, we found that Internet Explorer seemed to be ignoring our declaration. Changing the declaration to UTF-8 solved our problem.

Now that we changed it, we have to go into each existing article and profile and change the old special characters in iso-8859-1 to the new UTF-8 encoding. At least this happened before there were thousands of members and articles.

I am personally a fan of Internet Explorer, though my co-worker Andrew is more skeptical. Nonetheless, it is a little frightening when a sudden upgrade in technology suddenly breaks years of work. Oh well--such is the travails of web developers.