iconv misunderstands UTF-16 strings with no BOM

I had a problem last week with converting UTF-16 encoded strings to UTF-8 using PHP’s iconv library on a Linux server. my code worked fine on my machine but the same code resulted in a rubbish unreadable characters on our production server.

Let me take you to the beginning of the problem. I had a Hexadecimal representation of a UTF-16 string like that

0635 0628 0627 062D 0020 0627 0644 062E 064A 0631

this is the equivalent of “Good morning” in Arabic “صباح الخير” . I had some lines of code that will convert this into a normal stream of UTF-16 bytes so I can be able to use iconv to convert the string to UTF-8. maybe you noticed that there is no BOM at the beginning of Hexadecimal representation of the string. so let me quote what is written on Unicode’s BOM FAQ page about this.

Q: Why do some of the UTFs have a BE or LE in their label, such as UTF-16LE?

A: UTF-16 and UTF-32 use code units that are two and four bytes long respectively. For these UTFs, there are three sub-flavors: BE, LE and unmarked. The BE form uses big-endian byte serialization (most significant byte first), the LE form uses little-endian byte serialization (least significant byte first) and the unmarked form uses big-endian byte serialization by default, but may include a byte order mark at the beginning to indicate the actual byte serialization used.

so when there is no BOM, the string should be treated as big-endian. libiconv has a different opinion about this and will try to guess if it should use big-endian or little-endian depending on the operating system. so you will get different results on different machines.

The simple solution to this problem (after a long time trying to identify it), is to just tell iconv that I’m converting from UTF-16BE (big-endian) so it won’t try to guess the endianess of the bytes. so in php it will be like that

$result = iconv('UTF-16BE', 'UTF-8', $str);

or better, I can check the BOM before converting the Hexadecimal codes to a stream of bytes and taking the decision of converting from UTF-16BE or UTF-16LE depending on if it begins with FEFF or FFFE.

Developers in open work spaces

Are you working in open work space where all your co-workers are sitting in the same place with no walls between each desk? If your answer is yes, then mostly you know that the majority of people don’t understand how their loud talks can affect the productivity of their colleagues. some jobs require a big amount of concentration, let me just be specific and talk about software developers.

Most companies don’t give each software developer his own office and a door that he can close whenever he wants to concentrate and get isolated of the world. to provide an office to every developer you need a big place and big places are expensive and most software companies are managed by people who don’t get why would these developers need offices; they say that developers are just normal employees, not super stars.

We know that it’s not about being super stars as much as it’s about a fact that developers work on solving problems that contain a lot of little details, and the normal human brain won’t be able to hold those details together for longer than 5 minutes and connecting the little details is another harder task. I don’t imagine that a developer working while surrounded with many talking people will write code with good quality. he won’t be able to focus and won’t notice many logical errors and after some days this will give him more bugs to solve and more time to lose. this will force the company to pay more money and lose more time.

I don’t think that you will get your own office soon; unless you get the chance to work in a company that treats developers as super stars or maybe you will wait longer till you become a manager.

But now, what can you do? I think you can try to talk with your managers about how this noise affects you and your colleagues. and try to come up with some easy to implement ideas like making every 2 or more guys who want to discuss something go out of the work area till they finish their talk or even just stay in the work area and keep it down as much as they can. it won’t be a hard thing if your co-workers respect the value of working in a quiet place. but some times it won’t be that easy, like when you find yourself sharing the same work space with people who talk a lot as their job forces them to. like customer care and marketing guys. and I don’t know a solution for this case.

Another thing that you have to put a solution for is the interruptions done by co-workers. haven’t you ever been working on a very hard to solve problem that needed all your brain power and you were just seconds away from finding the solution and suddenly you find someone tapping on your shoulder to ask you about the name of a function that he doesn’t remember ?! I know that you hate this guy and at that moment maybe you wanted to kick him in the face, but you controlled your anger and just answered his question and left him to leave in peace and tried to go back to what you were doing and you couldn’t remember the last point you arrived to in your thinking, and probably it took you around 15 minutes to get back to your last concentration level.

You have to talk to your colleagues about this problem and tell them that they can send you emails that you will answer later when you are free, or maybe you can use a piece of paper with “No Interruption, please” written on it. you can put this paper on your chair/desk whenever you need to focus on something important, so your co-workers will know that you aren’t ready to talk at the moment.

there are many pros and cons of working in open work spaces, but we can get over the cons by putting effective rules that can control the noise and interruptions, and you won’t succeed in that unless your co-workers are understanding the problem and believe in the solutions.

maybe you want to read more. Joel Spolsky talked about this topic before when he was talking about the design of his company’s new office and it was the 8th rule in the famous Joel Test “Do programmers have quiet working conditions?” . also,  you can make your managers and colleagues read the field guide to developers.

so at last, I have to say that I don’t hate open work spaces but it will be a good work environment if all co-workers understand the new rules that have to be applied to make it a healthy work environment .

Irssi as your IRC client with notification using shnotify.pl

After spending the last years trying many IRC clients, I found that all GUI clients suck. so I decided to use a terminal based client and all recommendations pointed me to use Irssi. It took me around 2 hours to get used to how it works, just some commands and installing some of its scripts got me to do what I want. maybe you will love using it too if you are spending your day in front of terminals or if you just believe that GUI clients aren’t your best choice.

Installing it on Ubuntu wasn’t a problem, just like what you always do

sudo aptitude install irssi

now run irssi in your terminal to start it. now you can connect to any IRC server by typing this

/connect irc.freenode.net

Irssi won’t remember any settings and auto start it till you tell it to do so, and I wanted to make Irssi auto-connect to freenode.net on startup

/SERVER ADD -auto -network freenode irc.freenode.net

also, I didn’t want to type my password every time I connect to this server

/NETWORK ADD -autosendcmd "/msg nickserv identify mypassword;wait 2000" freenode

this will run the IDENTIFY command with my password after connecting to the freenode server and will wait for 2 seconds before doing any thing else.

now I can join any channel the normal IRC way by typing

/join #channelName

and I can jump between my opened channels using CTRL+n and CTRL+p , or by pressing ALT+ window number . if you are using gnome-terminal with many opened tabs, maybe you will have problems with moving between channels using ALT+number as this shortcut is used by gnome-terminal to jump between tabs.

Also, I found some nice scripts for Irssi like
nickcolor.pl , this script will give each user in your opened channels a separate color.

trackbar.pl , this one will put a separator (one line) after the last message appeared in every channel before you leave it.

installing these script is easy, you can put these script in ~/.irssi/scripts/ , then in irssi load them using

/run scriptname.pl

if you like a script and want to auto-load it on start, just create another directory “~/.irssi/scripts/autorun/” and create links in this directory pointing to your scripts that should be auto-loaded.

I wanted Irssi to notify me when anybody mention my name by showing me a notification bubble. I found here a script “mumbles.pl” written be Matthew Weier O’Phinney our Zend Framework warrior. but for my bad luck I didn’t know how to get the mumbles notification server specially as its website is down. so I tried to play with that script to make it show notifications using zenity. you will find the code here, you can create a new file “shnotify.pl” in your ~/.irssi/scripts directory and put that code in it. this will show you a notification bubble whenever your nickname or any other word you tell Irssi to highlight using the /hilight yourword command is mentioned.

use strict;
use Irssi;
use Irssi::Irc;
use vars qw($VERSION %IRSSI);

$VERSION = '0.1.0';
%IRSSI = (
 authors     => "Ahmed Shreef",
 contact     => 'ahmed@shreef.com',
 name        => 'notification bubble for Irssi',
 description => 'This script will show a notificatin bubble near the systray when your nickname is mentioned',
 license     => 'New BSD',
 changed     => "2010-6-9"
);

sub shnotify {
 my ($dest, $text, $stripped) = @_;

 if (($dest->{level} & (MSGLEVEL_HILIGHT|MSGLEVEL_MSGS)) && ($dest->{level} & MSGLEVEL_NOHILIGHT) == 0)
 {
 if ($dest->{level} & MSGLEVEL_PUBLIC)
 {
 my $message = $dest->{target} .": " . $stripped;
 `echo "message:$message" | zenity --notification --listen --window-icon=null`;
 }
 }
}

Irssi::signal_add({
 'print text'    => \&shnotify
});

tell me if you like it or if you need some help.

Introduction to flying in AIR

In the last days, I had the chance to play with the SDK of AIR beta (Adobe Integrated Runtime). on Adobe’s website they introduce it as :

Adobe® AIR™, formerly code-named Apollo, is a cross-operating system runtime that allows developers to use their existing web development skills to build and deploy rich Internet applications to the desktop.

AIR is still in beta and till now it only supports the following operating systems

  • Windows XP SP2
  • Windows Vista Home Ultimate Edition
  • Mac OS 10.4.8 and 10.4.9 (Intel and PowerPC)

* they will start working on supporting linux after the 1.0 release.

AIR mainly targets web developers. it allows you to develop desktop applications using your web development skills. you can develop AIR applications using HTML, JavaScript and Flash. you are not forced to use flash. you can only use HTML and JavaScript if you want.

not like the normal web development, in your AIR apps you won’t need to check for the type of the user’s browser . AIR uses the open source WebKit browser engine that’s used by known web browsers like KHTML on KDE and Safari.

as I didn’t work with Flash before, I chose the option of developing using HTML and JavaScript only. the nice thing is that I still have full control as I can access the AIR, Flash player and ActionScript APIs from my JavaScript code. this will allow me to do things I’m not allowed to do in a normal web application.

AIR comes with

  • embedded SQLite database
  • native windowing API
  • file IO API
  • application update API
  • full control over your application’s chrome
  • online/offline API
  • drag and drop support
    – Between operating system and Adobe AIR applications
    – Between Adobe AIR applications
    – Between native applications and Adobe AIR applications
  • native icons
  • standard installer

you can see that not like the normal web applications, the AIR applications have access to your file system and this will make you think about security. so like any normal desktop app, you have to use trusted applications only.

What not to expect from AIR?
AIR won’t replace the normal desktop applications. Kevin Hoyt explains this here. AIR aims to get the web to the desktop (not vice versa).