"Unicode data in a Unicode-only collation..." Error and Writing a Migration to Fix It

The problem

I've been maintaining a Rails project at work during my down-time, and recently ran into the following error after launching my web app to CloudFoundry:

Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier.

The project started in 2007 and has experienced on-and-off development for a long time, so when I saw this error, I wasn't entirely surprised: Practically every tool we use in that project is deprecated now that it's nearly 2013. So, where did this error come from? Why did it decide to show up 5 years into development? And why does it not show up in the development build? Well, this project has seen a lot of developers, and each one for only a couple weeks at a time. So I imagine this problem has always been there, or popped up during a recent CloudFoundry update, and it's just been ignored until now.

Note: I have recently found that this fix does not work as well as I thought it did. Looking for a better solution for the time being, will update later.

The first question: What caused this error?

Dissecting the error given above, it looks like a database issue with Unicode processing caused by an ntext variable. For various vague reasons, cloud-based applications occasionally don't get along well with nvarchar and ntext variables in SQL. Why? Well, these kinds of variables are used to store Unicode characters to potentially store multi-lingual data which can become quite large very quickly. About twice as large, in fact. Why do sites like CloudFoundry and Heroku get mad at these variables sometimes but seemingly not all the time? I have no idea.

Anyway, I looked through the columns in my database and, sure enough, there was a single nvarchar(MAX) column staring me in the face.

How do we fix this?

Write a simple migration to change the column. Looking into it briefly, the maximum number of bytes a varchar can hold in SQL is 8000. Now I know what you're saying. 8000 bytes? That's not nearly enough for my data! That's likely not true. This wall of text is 8000 bytes. Unless your writing a web application to have people submit high school essays or college theses, 8000 bytes is a lot. So let's write a migration! Fortunately, this is trivial. Assume our column name is "about_me" and our table name is "profile". Following the Rails Migration Guide, we start the terminal in our root directory:

> rails generate migration ChangeAboutMeColumnToVarChar

Now, we write our up and down methods. Remember, our column is about_me and our table is profile. In my case, the about_me column used to be a nvarchar(MAX) variable, which translates to a text variable in Rails. Following the Rails Migration Guide again, we have:

# db/migrate/ChangeAboutMeColumnToVarChar

class ChangeAboutMeColumnToVarChar < ActiveRecord::Migration
  def up
    change_column :profile, :about_me, :string, :limit => 8000
  end
 
  def down
    change_column :profile, :about_me, :text
  end
end

Now we run our migrations:

rake db:migrate RAILS_ENV="development"

And the problem should be fixed. Of course, after running the migrations on the local database, change the RAILS_ENV option to any other database that may need updated when you're ready.

Capital Punishment - Countries and Capitals

Capital Punishment - Countries and Capitals

Using the knowledge I'd gained in the tutorials I mentioned last post, I created the above web application and got it hosted on Heroku.

By my estimates, this project took about 16 hours of actual work plus about 4 hours of tutorial work. 20 hours to create, polish, and publish a web application doesn't seem bad to me.

The tutorial that I originally used was great, but it was for a very simple application and didn't scope quite as well as I had hoped. When I made Capital Punishment, everything worked well for me when I ran the app from my local machine. Then I pushed it to Heroku and everything changed.

The initial page would load, which made me fairly happy. It was clicking things and performing actions that began to disappoint me. Whenever I would navigate to a different page, Heroku would give me an Internal Server Error. So I checked my logs to see what was going on:

NameError - uninitialized class variable @@question_statement in BasicObject: heroku

To be honest, I should have seen this coming.  I had created a large app with very little Object-Oriented design put into it. What was happening was the context-switching caused by performing POST actions was losing track of my class variables. But "It Worked On My Machine." Why? I'm not entirely sure, but I think it has something to do with the way Heroku loads the pages.

So how did I fix this? Massive refactoring (my favorite kind of refactoring). I created a base class with all the shared logic, and then created two Singleton classes for the two game types: CountriesAndCapitals and StatesAndCapitals. I then referenced these singleton instances from the HTML and from my POST methods, and the problem was fixed.

Refactoring my code to be more OO provided an incredible amount of optimization of my original code. Now, the data for each game type is only being loaded once and the US data won't be loaded at all unless you start playing the US game. All of the repetitive code I wrote to load the two entities was modified to be in the base class. Everything that was painful to change because it was buried inside poorly-named methods became easy to refactor.

In this case, I was suffering from a combination of laziness and over-enthusiasm. I really wanted to get that code done before the weekend was over, and so I just threw a bunch of non-OO text at Netbeans until it worked.

I think I've learned a lesson from this: Write good code. It doesn't matter if you're in a hurry, or following some tutorial using sub-par styles; you should always try to write the best code first.

Great Tutorial On Ruby Web Applications

The tutorial linked below is quite nice. It shows the basics of using Nokogiri for data scraping, Sinatra to help create the web application, and Heroku for actually hosting the web application. A three-for-one deal.

http://hunterpowers.com/data-scraping-and-more-with-ruby-nokogiri-sinatra-and-heroku/

I'm so proud of my little copy of this blogger's tutorial.

Lessons Learned Presentation

So I'll be giving a short presentation on 9/6 with some of my fellow company representatives in front of some college students. Our topic is 'Lessons Learned,' and we are discussing things we wish we could tell our college selves now that we've been in industry. The following is my presentation transcribed from a slideshow. The plain text is a pseudo-script; the bold text is what actually appears on the slides. I've decided to go with a very simplistic slide style because that's what I'm into these days.

Padding Your Resume

Tips For Not Overloading Your Resume With Mindless Drivel

Since it's IR week, I'm sure every engineer on campus has been tweaking their resume around the clock, adding experience gained over the summer, deleting all the stuff you put in there about your high school soccer team, and trying to figure out if 8-point font is too small for a recruiter to read. I was like that around this time last year, and so I'd like to share some of my tips for not overloading your resume with mindless drivel.

Topic 1: Clubs

Tell me: How many activities do you participate in just so you can tack it on the end of your resume? We all do this. We join the Ambassadors, the Student Council, the Railroad Club, a few honors societies, the League of Knitters, and the Rock Band Club. We attend all their callouts and we eat more than our fair share of pizza. And then we get this wild idea. We self-nominate ourselves to be officers during that first meeting just to see that faint chance of actually becoming elected into a position of power so we can utter that one sweet word to company recruiters: "Leadership."

STOP IT

I've got two words for you. STOP IT! You're one person. These organizations may appreciate the boost in attendance you bring to the table, but after a few meetings you'll be bored and leave the organization, and there will be no proof that you were ever even there. You should be joining organizations that represent activities you feel passionate about.

You can do better!

I know you. You can do better. What are you passionate about? Spreading the word about Linux, cleaning up the Wabash, cutting down all those trees on campus that stink in the fall? Join the club, or create one. Limit yourself to only a few clubs. Then, instead of just sitting back, do something meaningful. Write a script to replace your club's secretary. Write software to manage your organization's budget. Make this the year where your club finally has female students attend the callout. Recruiters will be much happier talking to you for 10 minutes about a club you're passionate about than they would be listening to you list all the clubs you're a member of for 20 seconds.

Topic 2: Electives

Electives are important. Unlike classes specifically required by your degree, you have chosen to take these courses. That means you get a choice, though that choice may still be limited. So, what electives should you pick? Bowling? History of Rock & Roll? Wine Tasting? Ooh, I know, what about that class where you just watch movies?

Meaningful electives

Wrong, wrong, wrong. Ok, maybe Wine Tasting if you can register early enough. Do yourself a favor, and stop looking for the easiest electives. You can go bowling, listen to rock music, drink wine, and watch movies with your buddies on the weekends. Find classes that are interesting, maybe even challenging. Find a class that makes you a better person or a better professional. Look through classes that could combine with your major to make something cool: Math, Physics, and Biology can all be combined with electrical and computer engineering to form useful and innovative applications, and additional experience in these topics will make you stand out to companies in these kinds of industries.

Topic 3: Objective

Objectives. This is a tough one. Every person trying to help you with your resume so far in your life has told you that an objective is the most important part of a resume and that yours must be clear-cut for a company to notice you. But what is a clear-cut objective? Well, what do you want? You want a job. Hopefully not in fast-food.  Let's see what we've got here...

Objective: To obtain a job not in fast-food.

Ok, I guess that's pretty good. Maybe they'll let you work the drive-thru. Most objectives sound very generic and a recruiter skips right over it. You have two options at this point. The first is my favorite choice.

Objective:

Don't even bother. Most recruiters know that you're looking for a job, and it'll allow more space on your resume for meaningful information. The second choice is more difficult.

Objective: To be awesome and make more money than my friends.

Put some feeling into it. Put some spin on it. Say that you want a job with a certain kind of atmosphere, or list some of your skills in your objective. For example: I want to obtain a job which utilizes bleeding-edge technology and allows me to use my skills as an Agile expert to deliver innovative results.

Conclusion:

Your resume is sacred

Your resume is sacred. It represents you and everything you've accomplished up to this point in your professional career. In order to have a spectacular resume, you must be the resume. Join clubs you are passionate for and help out in meaningful ways. Take classes that are challenging and meaningful. Have an objective that shows off your individualism or do so by having no objective at all.

Lessons I Wished I'd Learned Earlier As An Undergrad

Having worked at a real company now for several months, I've been given the opportunity to go back to my Alma mater and participate in a lecture with some up-and-coming students. And get free food, of course, since this is a university event. I received fairly short notice (5 days) to come up with something engaging to talk about for anywhere between 5 and 20 minutes.

The topic: Lessons Learned After Graduating That Would Have Made University a whole helluva lot easier. Here's a short list of things I'm considering:

• Involvement
• Find groups that encourage development of your programming skills including user groups, tech lectures, open-source projects, and startup weekends.
• Use IDEs
• Why did this one take me so long to figure out? At my university, they taught us to use MATLAB freshman year (which hardly qualifies as an IDE without any form of auto-complete), and they showed us how to install Netbeans and Eclipse my senior year (but did not encourage us to use them. They also taught us how to use a VHDL IDE my senior year, but that's a different subject. Using IDEs greatly increases productivity, syntactical consistency, and enhances organization. A problem is that it can dumb a programmer down by not forcing them to look things up, and a dev can become lazy using auto-complete all the time.
• Reach outwards
• Don't do what you always do. Do things that you've never done before. My degree had a strong focus on C: so I did almost all projects where I was given a choice in either C or (eventually) C++. Given a distant deadline, a project could be started in any number of interesting languages, including Ruby or Perl or CoffeeScript or Scala.
• Don't do an activity just for your resume
• This one is the worst. People join a half-dozen clubs just to pad their resumes. This should not be done. Join one or two clubs that you're passionate about instead, and put a lot of time and effort into making that organization better. Companies will be far more interested in your automation of the secretary position in your Linux Club than your perfect attendance at your Rock Band Club.
• Use Office Hours
• Stuck on something? Go to office hours. It doesn't matter if it seems mundane, or the professor already explained it in class, or you skipped the lecture on this material, you will save yourself a world of time by going to office hours.
• Don't avoid lesser-known companies
• When I went to career fairs, I would be sure to visit all the companies on the list that I recognized. I would walk up to these companies, hand them my resume, walk away, and likely be offered an interview via email several days later. However, one career fair I took it upon myself to stop by several companies of which I had no idea what they were. So I stopped and I asked them if they were looking for someone like me. Some said yes, some said no. The ones who said yes took my resume, talked to me about all the information on it, and signed me up for an interview on the spot. The larger companies seemed distant and made me feel like a hamster in a cage. The smaller companies were warm to me and made me feel like a real person.
• Be "That Person"
• We won't have to worry about the project if we have "that guy" in our group. "That guy" works hard and gets the job done. Just do it. Work late. Understand things. Become an expert in whatever project you're working on. Your groupmates will respect you and any presentations you give will show off your vast amounts of knowledge.
• Use the internet
• Stuck on something that you know hundreds of other people have already solved? Use the internet. The internet knows the solution to that integral. The internet knows how to create a stack using a linked list. The internet knows how to get that program installed on your dev machine. You'd be surprised at what seemingly arbitrary things you can find answers to on the internet.
• Learn keyboard shortcuts
• Makes you more productive. I could list some, but they're on the internet.
• Write down everything
• You say you solved this problem three weeks ago but now you can't remember how to do it? The professor has your homework with the solution on it but never got it graded? You found some awesome Ruby syntax to do 10 lines of work in a single keyword? You should write these helpful things down in a notebook somewhere. Have several notebooks for several different topics. You'll thank yourself later.

Shakespeare Programming

The following lines of code group a list of customers by city and orders the groups with more than two values by key.

var custQuery =
    from cust in customers
    group cust by cust.City into custGroup
    where custGroup.Count() > 2
    orderby custGroup.Key
    select custGroup;

I dub this style of coding... "Shakespeare Coding." Why do we write code like this? This code is taken directly from the MSDN website, which implies that this is how Microsoft expects programmers to write code.

My problem with Shakespeare Coding is that it is too close to the syntax of a language near and dear to my heart. A language with weak typing, ambiguous references, poor commenting, and a rotund standard library filled with functions that rarely anyone will ever use. That language is English.

Shakespeare Programming is coding in a style which is akin to writing prose. I attest that this style of coding is not compatible with a programming mindset. Programming syntax should be readable, obvious, and terse. Prose has many nuances and intricacies that can make it ambiguous and difficult to interpret

Let's modify the code above to be a little bit more gentle on my eyes, shall we?

var custQuery = customers
                    .GroupBy(cust => cust.City)
                    .Where(group => group.Count() > 2)
                    .OrderBy(group => group.Key);

That wasn't so hard. And now what we have are three clauses, each of which has a very obvious and meaningful purpose in the main statement.

Desktop Deleted in Ubuntu 12.04

While trying to save files to ~/Desktop yesterday, I had the misfortune of saving a file as ~/Desktop. I wasn't actually aware at the time, and went about my usual business for several days. Then I restarted my computer and my desktop was loaded up with all the documents and directories located in my home directory. In addition to the items I expected to see in my home directory was a PDF named Desktop. Deleting this file and restarting one's computer is not enough to fix this problem. The fix: 1. Open up ~/.config/user-dirs.dirs 2. Modify the line XDG_DESKTOP_DIR to say XDG_DESKTOP_DIR="$HOME/Desktop" 3. Save and close that file. 4. Open up a terminal 5. `killall nautilus` Voila! Far easier than I anticipated. Note that if you're one of those people who stores non-temporary files in ~/Desktop, you will not recover those files using this method.

Why I've Grown To Love C#

I admit, I'm somewhat of a Linux programming snob. Why? Mostly ease, somewhat because it was how I was taught in school, and partially because I love the idea of FOSS.

But times change quickly, and I was thrown onto a C# project with a bunch of folks who were quite used to a Windows environment. Despite the obvious hassle of having to use Windows and deal with Visual Studio, .NET has some pretty cool things to offer.

Take the example where I have a class of students; my goal is to find their averages and get the student with the highest grade so I can single them out to the rest of the class. Let's first do this in vanilla C++:

Vector<Student> Students; // contains a short int array of test scores called Tests
...
short highestGrade = -1;
Student teachersPet;

for (int i=0; i<Students.Size(); i++)
{
    unsigned int sum = 0;
    for (int j=0; j<Students[i].Tests.Size(); j++)
    {
        sum += Students[i].Tests[j];
    }
    sum /= Students[i].Tests.Size();
    if (sum > highestGrade)
    {
        highestGrade = sum;
        teachersPet = Students[i];
    }
}

That's a mouthful. Not including intialization code, there are 7 lines of "useful" code, 5 auxiliary variables, 1 division, and a bit of array indexing. Let's perform a literal translation to C#:

List<Student> Students;
...

Student teachersPet;
short highestGrade = -1;
foreach (student in Students)
{
    int avg = 0;
    foreach (score in student.Tests)
    {
        avg += score;
    }
    avg /= student.Tests.Count();
    if (avg > highestGrade)
    {
        highestGrade = avg;
        teachersPet = student;
    }
}
</code></pre>

Pretty much the same code. We are utilizing the foreach operator, which lets us get away without any array indexing. Still 7 lines of "useful" code, still 3 auxiliary variables. Assuming the Tests property in the Student class is a array of ints, we can find the average in a much more succinct fashion:

foreach (student in Students)
{
    if (student.Tests.Average() > highestGrade)
    {
        teachersPet = student;
        highestGrade = student.Tests.Average();
    }
}

Four lines of functional code, two auxiliary variables. Why stop there? Using a LINQ-based approach and adding an Average property to the Student class, we can get rid of all auxiliary variables and find the max using the average in one line of code:

var teachersPet = Students.ForEach(x => x.Average = x.Tests.Average()).MaxBy(x => x.Average);

The same functionality in a single contiguous line of code! Not to mention these Average, ForEach, and Max functions can be better optimized by our compiler than what we had before. But is that as far down as we can squish it? What if we dropped the entire ForEach statement and just compared the averages:

var teachersPet = Students.MaxBy(x => x.Tests.Average());

And that's why I've grown attached to C#. LINQ is probably my favorite thing to ever come out of Microsoft

What's with Who Moved My Cheese?

Who Moved My Cheese?


Johnson, Spencer






Who Moved My Cheese? was a "required" book to read for new employees at my current company. Although the story is very short, the message is very straightforward: Don't get stuck in a rut. That's it. It took about 40 minutes to read through it.

I must admit I wasn't a fan of this book. It seems to have been written for a career path where getting into a rut was easy and relatively safe, but the world of software development is certainly not like that. Now that it's 2012, fewer and fewer people are still programming in COBOL, FORTRAN, BASIC, or even C for that matter. We are frequently given opportunities to learn new technologies that are more powerful and make our lives easier.

Having graduated from university recently, the message was somewhat lost on me. As students take different college courses every semester, they are frequently thrown for a loop and forced to "find cheese" for each new class in its own way.


Don't get me wrong. I don't think Cheese? is completely useless. I think a great demographic for this book would be the recently laid-off employee who may need some motivation to get back out in the work force. However, I don't think this book is a must-read for everyone. In particular, there was a line included in my version of the book which said that anyone who disregards the story is a know-it-all. I take offense to that statement; it has yet to be proven whether or not I actually "know it all."

If your coworkers have all read this book, read it because it only takes about an hour to read and no one wants to feel left out. If you need some motivation to get out of a rut, you don't need to read a book to tell you to get off your rump and do something about it; just do it.

Link: Researcher reveals how “Computer Geeks” replaced “Computer Girls”

Below is a link to an interesting article on what are called the first programmers (link courtesy r/programming). Unfortunately, women were booted out of the programming sector once the world realized how complex the job was. Now, 50 years later, we no longer tell women they can't be intellectuals and yet software development is still a male-dominated career path. Maybe the "Computer Girls" will make a comeback. Maybe Cosmo will have articles about software aimed at visionary young women. Just imagine the right answers to the sex quizzes leading you away from the brawny athlete and instead to the programming dreamboat!

Researcher reveals how “Computer Geeks” replaced “Computer Girls”

Having Read The Passionate Programmer

	The Passionate Programmer (2nd edition): Creating a Remarkable Career in Software Development
	Chad Fowler

The Passionate Programmer is essentially a book about self-motivation. It is neatly wrapped up into several sections about marketability and rekindling the passion for software development, and these sections are broken up into delightfully bite-sized chapters which make the book a breeze to go through.

This book preaches that we should be happy with our jobs but not complacent. The author, Chad Fowler, believes that, at all times, we should be looking for ways to better ourselves as professionals and as individuals. This is certainly a tall task. I've been having trouble finding time to do very many extracurricular activities lately, what with moving to a new city, starting my career, buying a car, and planning a wedding, but what this book really makes me want to do is search the internet for some new software technology I don't understand and learn everything I can about it. It's all about branching out from the norm and doing things one may be uncomfortable doing. I'm fortunate to have started a job at a company where I have the opportunity to frequently switch projects and do things that are completely different fairly regularly.

I also see it as a good thing that I was thrown into technologies I'd never had to deal with before during my first day on the job (Windows programming, .NET, WPF, TestStand, CruiseControl...).  All jobs should be like this. Companies hiring people for their potential to do anything instead of their potential to do a specific task.

Fowler also mentions being social as an important step in a passionate career. Managers, programmers from other companies, managers of managers, developers at conferences, managers of managers of managers, and even folks in online forums or blogs may be the link to an opportunity. I normally just live under the thought chain of "If I write it, it's good, and I'm efficient, I'll get noticed." But Fowler insists this is not true! Talking to these kinds of people and letting them know what you've been up to is beneficial in that it gives managers a sense of how the project's going, it may give other developers ideas or inspiration to solve problems, and it can help you feel more accomplished. It's a good mindset to be in, and it simply involves climbing out of the cubicle every once in a while and finding the right people to talk to.

This book provides good motivation to continue pushing oneself and it provides inspiration for how to better manage one's non-working life.

Programming for FTIR - A Starting Point

I recently finished a large project which involved creating a sizable touch surface. The results surprised a lot of people, and what we accomplished was no small feat.

It's no Microsoft Surface, but it cost us less than $800 to build and we ended up with a lot of spare parts. All the applications shown were written by me or my teammates in this project.

I utilized a lot of amazing open-source software to get to this end, including Ubuntu, reacTIVision, and TUIO. When I first started researching the technology I'd be using, I had no idea what I was getting myself into. It all started with this article by Maximum PC.

All applications were written in C++ using the SDL and TUIO libraries. For SDL, I started by following the tutorials found here and expanded to fit my needs. The reacTIVision software was used to track touches from a modified Playstation Eye and send them to TUIO.

To get all these tools working on your machine may not be easy. For my build, I started with a fresh install of 32-bit Ubuntu 10.04.2. I used this version for its stability and the Linux kernel was still within version 2.6.x which has a patch available to allow the Playstation Eye to run at maximum speed. The steps to patch the Linux kernel for optimal use of the Eye are detailed on many websites, but the one I followed (several times for several fresh installs) was found in a blog post of Multitouch Dev.

Many people use touch-tracking software called CCV on their hobbyist FTIR projects. CCV (also known as tBeta) is an open-source touch-tracking program. The UI was slick, but this application did not fit my computational needs. I was running on an Intel Atom board, and CCV used up 100% of one of my hyperthreaded cores no matter what I tried to do, forcing me to search for alternatives.

This was when I found reacTIVision. reacTIVision was a more mature IR touch-tracking software that achieved very similar goals to CCV. The best part about reacTIVision was that it used much less of my CPU: somewhere around 20-30% as opposed to 100%. With this software, I was able to run my modified Playstation Eye at 60 FPS with a 640x480 resolution. Fortunately for Ubuntu users, a .deb package is available for easy installation of reacTIVision.

The SDL packages can be found in the Synaptic Package Manager. Additionally, I used SDL_image to load .png images and SDL_mixer to load sounds into my applications. One should also have the build-essential packages installed on their machine for programming.

Finally comes interpreting of the tracked data from the reacTIVision software. For this, I used libTUIO. TUIO is a wonderful library with client implementations in many modern languages, including C++, Java, C#, and Flash. You can download these libraries from the TUIO website. For my purposes, I used the C++ libraries. Once downloaded, I performed a make on these files and got some build errors. These build errors tend to occur for some of the sample applications included with the library. Luckily, the library itself was already built, and so I moved the .so, .a, and header files to appropriate locations in my filesystem.

From there, one can start making their own touch programs and use the TUIO simulator to perform testing. To actually get touches to work, you should follow the MaximumPC article's section on setting up the LED array. We packed the LEDs in about 3/4" apart on all sides of our 34"x26" screen and were able to get relatively accurate touches. Be warned that the bigger the screen, the less accurate the touches tend to be. Due to our projector size, we also had to use an angled mirror to get the image size we desired, which caused some slight distortion in our display.

In the future, I'd like to go over actually writing a C++ touch application using TUIO.