"Unicode data in a Unicode-only collation..." Error and Writing a Migration to Fix It

The problem

I've been maintaining a Rails project at work during my down-time, and recently ran into the following error after launching my web app to CloudFoundry:

Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier.

The project started in 2007 and has experienced on-and-off development for a long time, so when I saw this error, I wasn't entirely surprised: Practically every tool we use in that project is deprecated now that it's nearly 2013. So, where did this error come from? Why did it decide to show up 5 years into development? And why does it not show up in the development build? Well, this project has seen a lot of developers, and each one for only a couple weeks at a time. So I imagine this problem has always been there, or popped up during a recent CloudFoundry update, and it's just been ignored until now.

Note: I have recently found that this fix does not work as well as I thought it did. Looking for a better solution for the time being, will update later.

The first question: What caused this error?

Dissecting the error given above, it looks like a database issue with Unicode processing caused by an ntext variable. For various vague reasons, cloud-based applications occasionally don't get along well with nvarchar and ntext variables in SQL. Why? Well, these kinds of variables are used to store Unicode characters to potentially store multi-lingual data which can become quite large very quickly. About twice as large, in fact. Why do sites like CloudFoundry and Heroku get mad at these variables sometimes but seemingly not all the time? I have no idea.

Anyway, I looked through the columns in my database and, sure enough, there was a single nvarchar(MAX) column staring me in the face.

How do we fix this?

Write a simple migration to change the column. Looking into it briefly, the maximum number of bytes a varchar can hold in SQL is 8000. Now I know what you're saying. 8000 bytes? That's not nearly enough for my data! That's likely not true. This wall of text is 8000 bytes. Unless your writing a web application to have people submit high school essays or college theses, 8000 bytes is a lot. So let's write a migration! Fortunately, this is trivial. Assume our column name is "about_me" and our table name is "profile". Following the Rails Migration Guide, we start the terminal in our root directory:

> rails generate migration ChangeAboutMeColumnToVarChar

Now, we write our up and down methods. Remember, our column is about_me and our table is profile. In my case, the about_me column used to be a nvarchar(MAX) variable, which translates to a text variable in Rails. Following the Rails Migration Guide again, we have:

# db/migrate/ChangeAboutMeColumnToVarChar

class ChangeAboutMeColumnToVarChar < ActiveRecord::Migration
  def up
    change_column :profile, :about_me, :string, :limit => 8000
  end
 
  def down
    change_column :profile, :about_me, :text
  end
end

Now we run our migrations:

rake db:migrate RAILS_ENV="development"

And the problem should be fixed. Of course, after running the migrations on the local database, change the RAILS_ENV option to any other database that may need updated when you're ready.

Capital Punishment - Countries and Capitals

Capital Punishment - Countries and Capitals

Using the knowledge I'd gained in the tutorials I mentioned last post, I created the above web application and got it hosted on Heroku.

By my estimates, this project took about 16 hours of actual work plus about 4 hours of tutorial work. 20 hours to create, polish, and publish a web application doesn't seem bad to me.

The tutorial that I originally used was great, but it was for a very simple application and didn't scope quite as well as I had hoped. When I made Capital Punishment, everything worked well for me when I ran the app from my local machine. Then I pushed it to Heroku and everything changed.

The initial page would load, which made me fairly happy. It was clicking things and performing actions that began to disappoint me. Whenever I would navigate to a different page, Heroku would give me an Internal Server Error. So I checked my logs to see what was going on:

NameError - uninitialized class variable @@question_statement in BasicObject: heroku

To be honest, I should have seen this coming.  I had created a large app with very little Object-Oriented design put into it. What was happening was the context-switching caused by performing POST actions was losing track of my class variables. But "It Worked On My Machine." Why? I'm not entirely sure, but I think it has something to do with the way Heroku loads the pages.

So how did I fix this? Massive refactoring (my favorite kind of refactoring). I created a base class with all the shared logic, and then created two Singleton classes for the two game types: CountriesAndCapitals and StatesAndCapitals. I then referenced these singleton instances from the HTML and from my POST methods, and the problem was fixed.

Refactoring my code to be more OO provided an incredible amount of optimization of my original code. Now, the data for each game type is only being loaded once and the US data won't be loaded at all unless you start playing the US game. All of the repetitive code I wrote to load the two entities was modified to be in the base class. Everything that was painful to change because it was buried inside poorly-named methods became easy to refactor.

In this case, I was suffering from a combination of laziness and over-enthusiasm. I really wanted to get that code done before the weekend was over, and so I just threw a bunch of non-OO text at Netbeans until it worked.

I think I've learned a lesson from this: Write good code. It doesn't matter if you're in a hurry, or following some tutorial using sub-par styles; you should always try to write the best code first.

Great Tutorial On Ruby Web Applications

The tutorial linked below is quite nice. It shows the basics of using Nokogiri for data scraping, Sinatra to help create the web application, and Heroku for actually hosting the web application. A three-for-one deal.

http://hunterpowers.com/data-scraping-and-more-with-ruby-nokogiri-sinatra-and-heroku/

I'm so proud of my little copy of this blogger's tutorial.

Lessons Learned Presentation

So I'll be giving a short presentation on 9/6 with some of my fellow company representatives in front of some college students. Our topic is 'Lessons Learned,' and we are discussing things we wish we could tell our college selves now that we've been in industry. The following is my presentation transcribed from a slideshow. The plain text is a pseudo-script; the bold text is what actually appears on the slides. I've decided to go with a very simplistic slide style because that's what I'm into these days.

Padding Your Resume

Tips For Not Overloading Your Resume With Mindless Drivel

Since it's IR week, I'm sure every engineer on campus has been tweaking their resume around the clock, adding experience gained over the summer, deleting all the stuff you put in there about your high school soccer team, and trying to figure out if 8-point font is too small for a recruiter to read. I was like that around this time last year, and so I'd like to share some of my tips for not overloading your resume with mindless drivel.

Topic 1: Clubs

Tell me: How many activities do you participate in just so you can tack it on the end of your resume? We all do this. We join the Ambassadors, the Student Council, the Railroad Club, a few honors societies, the League of Knitters, and the Rock Band Club. We attend all their callouts and we eat more than our fair share of pizza. And then we get this wild idea. We self-nominate ourselves to be officers during that first meeting just to see that faint chance of actually becoming elected into a position of power so we can utter that one sweet word to company recruiters: "Leadership."

STOP IT

I've got two words for you. STOP IT! You're one person. These organizations may appreciate the boost in attendance you bring to the table, but after a few meetings you'll be bored and leave the organization, and there will be no proof that you were ever even there. You should be joining organizations that represent activities you feel passionate about.

You can do better!

I know you. You can do better. What are you passionate about? Spreading the word about Linux, cleaning up the Wabash, cutting down all those trees on campus that stink in the fall? Join the club, or create one. Limit yourself to only a few clubs. Then, instead of just sitting back, do something meaningful. Write a script to replace your club's secretary. Write software to manage your organization's budget. Make this the year where your club finally has female students attend the callout. Recruiters will be much happier talking to you for 10 minutes about a club you're passionate about than they would be listening to you list all the clubs you're a member of for 20 seconds.

Topic 2: Electives

Electives are important. Unlike classes specifically required by your degree, you have chosen to take these courses. That means you get a choice, though that choice may still be limited. So, what electives should you pick? Bowling? History of Rock & Roll? Wine Tasting? Ooh, I know, what about that class where you just watch movies?

Meaningful electives

Wrong, wrong, wrong. Ok, maybe Wine Tasting if you can register early enough. Do yourself a favor, and stop looking for the easiest electives. You can go bowling, listen to rock music, drink wine, and watch movies with your buddies on the weekends. Find classes that are interesting, maybe even challenging. Find a class that makes you a better person or a better professional. Look through classes that could combine with your major to make something cool: Math, Physics, and Biology can all be combined with electrical and computer engineering to form useful and innovative applications, and additional experience in these topics will make you stand out to companies in these kinds of industries.

Topic 3: Objective

Objectives. This is a tough one. Every person trying to help you with your resume so far in your life has told you that an objective is the most important part of a resume and that yours must be clear-cut for a company to notice you. But what is a clear-cut objective? Well, what do you want? You want a job. Hopefully not in fast-food.  Let's see what we've got here...

Objective: To obtain a job not in fast-food.

Ok, I guess that's pretty good. Maybe they'll let you work the drive-thru. Most objectives sound very generic and a recruiter skips right over it. You have two options at this point. The first is my favorite choice.

Objective:

Don't even bother. Most recruiters know that you're looking for a job, and it'll allow more space on your resume for meaningful information. The second choice is more difficult.

Objective: To be awesome and make more money than my friends.

Put some feeling into it. Put some spin on it. Say that you want a job with a certain kind of atmosphere, or list some of your skills in your objective. For example: I want to obtain a job which utilizes bleeding-edge technology and allows me to use my skills as an Agile expert to deliver innovative results.

Conclusion:

Your resume is sacred

Your resume is sacred. It represents you and everything you've accomplished up to this point in your professional career. In order to have a spectacular resume, you must be the resume. Join clubs you are passionate for and help out in meaningful ways. Take classes that are challenging and meaningful. Have an objective that shows off your individualism or do so by having no objective at all.

Lessons I Wished I'd Learned Earlier As An Undergrad

Having worked at a real company now for several months, I've been given the opportunity to go back to my Alma mater and participate in a lecture with some up-and-coming students. And get free food, of course, since this is a university event. I received fairly short notice (5 days) to come up with something engaging to talk about for anywhere between 5 and 20 minutes.

The topic: Lessons Learned After Graduating That Would Have Made University a whole helluva lot easier. Here's a short list of things I'm considering:

• Involvement
• Find groups that encourage development of your programming skills including user groups, tech lectures, open-source projects, and startup weekends.
• Use IDEs
• Why did this one take me so long to figure out? At my university, they taught us to use MATLAB freshman year (which hardly qualifies as an IDE without any form of auto-complete), and they showed us how to install Netbeans and Eclipse my senior year (but did not encourage us to use them. They also taught us how to use a VHDL IDE my senior year, but that's a different subject. Using IDEs greatly increases productivity, syntactical consistency, and enhances organization. A problem is that it can dumb a programmer down by not forcing them to look things up, and a dev can become lazy using auto-complete all the time.
• Reach outwards
• Don't do what you always do. Do things that you've never done before. My degree had a strong focus on C: so I did almost all projects where I was given a choice in either C or (eventually) C++. Given a distant deadline, a project could be started in any number of interesting languages, including Ruby or Perl or CoffeeScript or Scala.
• Don't do an activity just for your resume
• This one is the worst. People join a half-dozen clubs just to pad their resumes. This should not be done. Join one or two clubs that you're passionate about instead, and put a lot of time and effort into making that organization better. Companies will be far more interested in your automation of the secretary position in your Linux Club than your perfect attendance at your Rock Band Club.
• Use Office Hours
• Stuck on something? Go to office hours. It doesn't matter if it seems mundane, or the professor already explained it in class, or you skipped the lecture on this material, you will save yourself a world of time by going to office hours.
• Don't avoid lesser-known companies
• When I went to career fairs, I would be sure to visit all the companies on the list that I recognized. I would walk up to these companies, hand them my resume, walk away, and likely be offered an interview via email several days later. However, one career fair I took it upon myself to stop by several companies of which I had no idea what they were. So I stopped and I asked them if they were looking for someone like me. Some said yes, some said no. The ones who said yes took my resume, talked to me about all the information on it, and signed me up for an interview on the spot. The larger companies seemed distant and made me feel like a hamster in a cage. The smaller companies were warm to me and made me feel like a real person.
• Be "That Person"
• We won't have to worry about the project if we have "that guy" in our group. "That guy" works hard and gets the job done. Just do it. Work late. Understand things. Become an expert in whatever project you're working on. Your groupmates will respect you and any presentations you give will show off your vast amounts of knowledge.
• Use the internet
• Stuck on something that you know hundreds of other people have already solved? Use the internet. The internet knows the solution to that integral. The internet knows how to create a stack using a linked list. The internet knows how to get that program installed on your dev machine. You'd be surprised at what seemingly arbitrary things you can find answers to on the internet.
• Learn keyboard shortcuts
• Makes you more productive. I could list some, but they're on the internet.
• Write down everything
• You say you solved this problem three weeks ago but now you can't remember how to do it? The professor has your homework with the solution on it but never got it graded? You found some awesome Ruby syntax to do 10 lines of work in a single keyword? You should write these helpful things down in a notebook somewhere. Have several notebooks for several different topics. You'll thank yourself later.

Shakespeare Programming

The following lines of code group a list of customers by city and orders the groups with more than two values by key.

var custQuery =
    from cust in customers
    group cust by cust.City into custGroup
    where custGroup.Count() > 2
    orderby custGroup.Key
    select custGroup;

I dub this style of coding... "Shakespeare Coding." Why do we write code like this? This code is taken directly from the MSDN website, which implies that this is how Microsoft expects programmers to write code.

My problem with Shakespeare Coding is that it is too close to the syntax of a language near and dear to my heart. A language with weak typing, ambiguous references, poor commenting, and a rotund standard library filled with functions that rarely anyone will ever use. That language is English.

Shakespeare Programming is coding in a style which is akin to writing prose. I attest that this style of coding is not compatible with a programming mindset. Programming syntax should be readable, obvious, and terse. Prose has many nuances and intricacies that can make it ambiguous and difficult to interpret

Let's modify the code above to be a little bit more gentle on my eyes, shall we?

var custQuery = customers
                    .GroupBy(cust => cust.City)
                    .Where(group => group.Count() > 2)
                    .OrderBy(group => group.Key);

That wasn't so hard. And now what we have are three clauses, each of which has a very obvious and meaningful purpose in the main statement.

Desktop Deleted in Ubuntu 12.04

While trying to save files to ~/Desktop yesterday, I had the misfortune of saving a file as ~/Desktop. I wasn't actually aware at the time, and went about my usual business for several days. Then I restarted my computer and my desktop was loaded up with all the documents and directories located in my home directory. In addition to the items I expected to see in my home directory was a PDF named Desktop. Deleting this file and restarting one's computer is not enough to fix this problem. The fix: 1. Open up ~/.config/user-dirs.dirs 2. Modify the line XDG_DESKTOP_DIR to say XDG_DESKTOP_DIR="$HOME/Desktop" 3. Save and close that file. 4. Open up a terminal 5. `killall nautilus` Voila! Far easier than I anticipated. Note that if you're one of those people who stores non-temporary files in ~/Desktop, you will not recover those files using this method.