Backward compatible

Why PHP is better than JavaScript

Sat, 12 May 2012 21:56:00 +0200

I started developing a small project using node.js with express and socket.io. Node is a nice server and socket.io is great. However, I’m having issues with javascript. Currently, two things really get on my nerves:

1. the plus operator

Most of the hard-to-debug bugs in my javascript code come from the + operator. It decides to concatenate strings instead of add numbers. Considering that all stuff that goes over the wire (i.e. socket.io) is treated as strings, it’s really painful and ugly to have parseInt(…, 10) everywhere. PHP solves this issue with simple dot operator. Simple, no-brainer and always does what you expect. You don’t have to think where does the data come from.

2. foreach

I miss PHP’s foreach so bad. Consider:

for (ix in really.long[expression].toGet.theStuff) {
    if (really.long[expression].toGet.theStuff[ix].value < 10 && really.long[expression].toGet.theStuff[ix] > 5) {
        ...do something

versus PHP’s:

foreach ($really.long[expression].toGet.theStuff as $ix=>$value) {
    if ($value < 10 && $value > 5) {
        ....do something

Of course, one could assign the array element to some local variable, and so I have local variables all around wasting code lines and making code error prone (if you need to change the collection you are iterating, you have to change in two places).

Google multi-account get screwd up again

Thu, 10 May 2012 10:30:49 +0200

Looks like guys at Google really have trouble with multi-accounts and sessions. Everything was working fine for months now, but they messed it up again. What does the problem look like:

I have 2 google accounts, one @gmail.com and other @mydomain.com
I cannot login into @gmail one directly. I have to log into @mydomain and then use the “switch account” feature
I cannot bookmark both gmails. Although bookmarks are different, both open @mydomain account
Most other google services I use (ex. Analytics) are tied to my @gmail account. I cannot access those at all, unless I log out of everything, clear all the cookies and then log just into @gmail.

My user experience with google is getting worse every day:

multi-account login problems
google docs become painfully slow when spreadsheets grow 300+ rows (only about 10 columns though)
search is polluted with g+ spam, translation offerings, etc.
on one of my computers it shows my location in the wrong country and I cannot find a way to tell it differently.

If someone build free replacements, I would surely give those a shot.

GMail ignores Reply-to header [SOLVED]

Fri, 04 May 2012 11:30:21 +0200

I have a conact form on my website. People can leave their e-mail so that I contact them back. I set it up so that e-mail is sent From my e-mail address (general rule: never put user’s e-mail in From field), and Reply-to set to user’s e-mail address.

However, when I click “Reply” in GMail, the reply gets sent to back to me. Looks like some glitch in GMail’s design, and they did not bother to fix it for a long time.

The solution is rather simple, just change the From field to some other address you own (different from GMail account address). For example if your e-mail is office@example.com, you can use support@example.com in the header. After this little change, Reply-to started working properly.

I hope this helps someone.

How to extract mp3 from YouTube video using Linux?

Fri, 27 Apr 2012 23:33:34 +0200

It’s rather easy. I used 3 components:

DownloadHelper extension for Firefox
MPlayer
lame

When DownloadHelper is installed and you open a YouTube video, it gives you the options to download the .flv or .mp4 file to your computer.

After the file is downloaded, we can use MPlayer to play it and also to dump the audio. It’s nice because it plays both .flv and .mp4, so you just need one program. To dump the audio to .WAV format use:

mplayer -ao pcm:waveheader FILENAME.flv

This will create file called audiodump.wav. Now, use mp3lame to encode it to mp3 format. You can also use oggenc to convert it to OGG if you perfer open formats.

lame audiodump.wav song.mp3

That’s all. I put these commands in a simple shell script (video2mp3.sh):

mplayer -ao pcm:waveheader $1
lame audiodump.wav $1.mp3

Run it from command line like this:

. video2mp3.sh FILENAME.flv

Lean Startup vs Rework Some time ago I finished reading the Lean...

Wed, 11 Apr 2012 11:29:27 +0200

Lean Startup vs Rework

Some time ago I finished reading the Lean Startup book by Eric Ries. Although I have been using some techniques from it before I have learned a lot. Yesterday, I found a mention of Rework on some website. Rework is a book by founders of 37 Signals, which I also read before. I started to compare the content of Lean Startup and Rework and I got some interesting conclusions.

Rework and 37 signals business model is really only a sub-set of Lean Startup philosophy. Basically, Rework stops somewhere during lean startup process and says “we’re content with this”. You build MVP, test it with customers, tweak a little bit and whoa, if you get good product market fit, what’s next? Depends who you ask: 37signals guys would tell you: “Well done, now enjoy your success”. They refuse to grow business, add features and go for a larger market. I’m not saying this is bad, sometimes you need to know what is your field of competence and stay there.

Which one would you follow? It depends on your personality. If you’re going to become a serial entrepreneur, rework is not enough. If you have an urge to move forward, discover new horizons, you might need to use Lean Startup in each new project again. After all, one can hardly call 37signals a startup anymore. They behave like established business, not a startup. As DHH said on Twist, “if you’re not doing your best idea now, you’re doing it wrong”. But, how do you know what is your best idea going to be if you do not explore? Maybe something looks like my best idea now, and I should be working on it. But, by the time that project becomes mature and stable, I might get a dozen of better ideas. And once I can turn the reigns of the current project to some good manager, I can go back to “startup” mode and explore new boundaries.

Twitter Timestamp out of bounds [solved]

Fri, 23 Mar 2012 12:40:52 +0100

Some time ago, automatic sending of status updates from one of my applications stopped working. I haven’t changed anything on the server, so this was strange. Looking into error message from Twitter:

[request] => /1/statuses/update_with_media.json 
[error] => Timestamp out of bounds

well, that’s strange. I recall time was moved to DST (daylight savings) Summer time in US recently, and apparently so did Twitter’s servers. The rest of the world - bah, they don’t seem to be interested, they can barely run the servers for US users apparently.

So, until DST change comes to the rest of the world, we need a hack. At first I thought that Twitter does not like timestamps to be in the future, so I thought about decreasing the timestamp of tweets. That did not work. So, I tried to increase the timestamp and everything is working now. The change is quite simple, just increase OAuth oauth_timestamp by a couple of hours and you’re done. In PHP OAuth client it looks like this (I added 5 hours):

  private static function generate_timestamp() {
    return time()+5*3600;
  }

What's the conversion on e-mail newsletter with a call to action?

Thu, 23 Feb 2012 21:19:00 +0100

I have measured that in the past week. One of my websites has more than 500.000 user accounts. I picked the users who were not on the site in the last week, because they have already seen the news on the site, and got some 384.000+ distinct e-mail addresses. I needed to contact them regarding an important issue about the website. The e-mail was composed like this:

Hello,

you are reading this mail because you are a member of [mysite link].

Issue explained and link with [call to action]

Regards,

Your webmaster
[mysite link]

Links where not plain text, but special URLs I used to track the clicks.

I sent the e-mail slowely over a 7 day period. I wanted to track weekends and working days as well.

And here are the stats:

384408 different e-mail addresses
13966 bounced back (3.6% bounce rate)
9345 clicks (yielding 2.5% conversion rate)

Hot spots in the e-mail message:

23% clicks on link in the first sentence
65% clicks on call-to-action
12% clicks on website link in signature

Here are the weekday stats:

Monday 13%
Tuesday 16%
Wednesday 18%
Thursday 18%
Friday 15%
Saturday 9%
Sunday 11%

Using JavaScript to split text string into word tokens, taking account of punctuation and whitespace and UTF-8 charset

Wed, 22 Feb 2012 22:37:00 +0100

I got an interesting problem today. I was supposed to check some HTML form before submitting to see if the text entered by the user in textarea has some specific words in it. Googling around I found a lot of stuff like “how to split text separated by commas” and such, but I simply wanted to extract words from a paragraph like this one.

My instinct was to use String.split() function, but it splits on a single character and I would have to write a recursive or iterative function to split on all non-word characters. Not being able to predict all the crap users can enter, this did not look like the right choice.

Luckily, I discovered String.match() which uses regex and is able to split text into an array of words, using something like this:

var arr = inputString.match(/\w+/g);

Cool, eh? Now, this all went fine for ASCII English text. But I need to work with UTF-8, or more specifically, Serbian language. Serbian Latin script used by my users has only 5 characters that are not from ASCII set, so I wrote a small replace function to replace those 5 with their closest matches. The final code looks like this:


var s = srb2lat(inputString.toUpperCase());
var a = s.match(/\w+/g);
for (var i = 0; a && i < a.length; i++)
{
    if (a[i] == 'SPECIAL')
        alert('Special word found!');
}

function srb2lat(str)
{
    var len = str.length;
    var res = '';
    var rules = { 'Đ':'DJ', 'Ž':'Z', 'Ć':'C', 'Č':'C', 'Š':'S'};
    for (var i = 0; i < len; i++)
    {
        var ch = str.substring(i, i+1);
        if (rules[ch])
            res += rules[ch];
        else
            res += ch;
    }
    return res;
}
";

If you use some other language, just replace the rules array with different transliteration rules.

How to use IDB files of Quicken Home Inventory on 64 bit Windows?

Mon, 09 Jan 2012 23:17:17 +0100

Directly load your IDB file from Quicken Home Inventory on any 64 bit Windows system. It works on 32 bit as well, of course. Today, a new version of Attic Manager is released, version 3.00. This version is able to load data directly from IDB files, there is no need to install any additional software. You don’t even have to have Quicken installed. This also means that you can run this option on 64 bit Windows 7 for example, or even on Linux.

Attic Manager can also load the inventory data from QHI and MDF files. QHI files are also loaded without any additional software.

For MDF files you need to have Microsoft SQL Server Express Edition installed. This is a freeware from Microsoft that comes with QHIM, so if you already have Quicken installed on the same computer, you don’t need to install anything.

In any case, Attic Manager is now unique software on the market, being able to load all Quicken Home Inventory formats and allowing you to keep track of your items on any PC.

There are even hints of Mac version coming soon.

Why aren't you using InnoDB instead of MyISAM?

Wed, 04 Jan 2012 19:07:40 +0100

For this particular application, I simply cannot afford it. MyISAM is able to cache index and data separately, using it I can keep the whole index in RAM and website works great. I tried to convert to InnoDB, the result was 4x larger database and 20 times worse performance, mostly due to the fast that index and data gets the same priority for caching so it was killed by disk I/O. If this website was earning enough money to buy at least 4 x more RAM it might not be a bad idea. However, it this case I’d rather use Firebird - it has a similar memory footprint as InnoDB, but has much more features (*real* stored procedures and triggers that work without problems, ability to use table aliases in delete statements, database events functionality, better resistance to system crashes, etc.)

Why is MySQL still a toy database

Tue, 03 Jan 2012 12:53:00 +0100

I have been using MySQL for a very intensive read-write web application (averaging 102 queries per second) for more than two years. I had ups and downs with it, like crazy MyISAM behavior that readers can block writers AND OTHER READERS. Basically, a table level lock is issued for read. I have 100+ million records in a table, so it takes a while to find anything that is not indexed. In the meantime, users are pondering (102qps, remember) and load goes up so much because of web server processes queuing like crazy. Ok, I learned not to do that anymore. I now use binary logging, restore to a different server and query there. Maybe a switch to InnoDB would be a good idea, but in this case I’d rather use a serious MVCC database like Firebird. Why, you might ask… well, here’s one of many reasons, the one that prompted my to write this:

In Firebird, I can happily do this:

delete from atable a1 
where exists ( 
select 1 from atable a2 where a1.data = a2.data and a1.id <> a2.id );

It just does it, and fast, because index on primary key field ID is used. In MySQL, to quote the manual:

“Currently, you cannot delete from a table and select from the same table in a subquery.”

Come on, this is one of the most basic database operation. So, what am I now to do? Waste my time dumping the list of IDs to delete to some temporary location, and then iterating that list to delete. :(

Scrolling back in screen

Fri, 30 Dec 2011 13:26:43 +0100

A few years ago I discovered screen, a nice Linux tool that enables you to detached from terminal with commands running and all in the background. You can even connect later from a different computer and continue where you left off. I initially used it for rtorrent, but now I also use it to administer remote computers, for example when I start to do something that might take more than a day, I can log back in tomorrow. Also loggin in from home/work to complete some task, etc. Another use is administering remote computers on dial-up (yes, there are some) or slow and unstable 3G connections. Even if connection breaks down, I can log in later and pick up where it stopped.

One of the annoying “problems” with screen is that shift+page up/down does not scroll the buffer. This is due to the fact that screen has its own buffers. To work with them you need to enter the “copy mode” using Ctrl+a followed by [. Since I use non-English keyboard that’s Ctrl+a, AltGr+f. Hard to remember when you don’t use it often.

I use Konsole, and I found a way to make it work by adding the following lines to .screenrc (in my home directory):

termcapinfo xterm|xterms|xs|rxvt ti@:te@

Why is Firebird better DBMS than Oracle?

Fri, 16 Dec 2011 12:07:33 +0100

Beside being free (both as beer and also open source), you don’t need 24x7 DBA and there are generally less headaches. Here’s a nice example explained by Norman Dumbar in a mailing-list post. Norman administers over 600 Oracle databases and about 40 Firebird ones:

Oracle uses log files for REDO and has ROLLBACK_SEGMENTS or UNDO Segments (depending on Oracle version) for UNDO. It never uses log files for UNDO - and UNDO is what provides Read Consistency/MVCC in an Oracle database.

Changes are written to the LOG_BUFFER (n memory) and periodically - on commit, every 3 seconds max, or when the buffer is 33% full - flushed to the REDO logs. These REDO logs might be archived to disc when they fill up. That Depends on the database archive log mode though.

These logs are used when a database is restored and rolled forward (using the RECOVER DATABASE command, for example).

In order to roll back changes and to ensure read consistency, UNDO is used. These do live on disc - as tablespace files - but remain in memory in the buffer cache alongside data blocks etc.

When a SELECT is started, the data returned are the data from the data blocks. Each row in a block has an indicator that tells when it was last updated. If a pending update is taking place (currently uncommitted) or if a commit has taken place since this SELECT started then the data read from that data block has changed - and is not consistent with the start time of this SELECT transaction.

When this is detected, Oracle “rolls back” the changes to the start time of the SELECT taking place by looking for the UNDO block(s) associated with the transaction that made the changes. If that results in the correct (consistent) data, that’s what you get.

If it turns out that there were other transactions that also changed the data, they too will be detected and undone.

In this way you only ever see data that was consistent at the start of your own transaction.

As long as the DBA correctly sizes the UNDO tablespace and correctly sets the UNDO_RETENTION parameter to a decent enough value, data changes are able to be rolled back happily all the time.

If the DBA failed miserably in his/her duties, the ORA-01555 Snapshot too old” errors are the result. And are most irritating. Long running SELECTS - batch reports for example - tend to show up this error mostly.

Of course, you would never see such problems with Firebird, because the old record versions are stored in database and not the log files. You don’t have to care if system crashes - after reboot it simply works.

You might think that engineers who build Firebird are smarter than Oracle’s but sometimes I think Oracle is deliberately made so complicated to require DBA and also offer them job security. And also makes sure nobody can complain it’s too easy to use.

Using Quicken Home Inventory Manager on 64bit systems [FINALLY]

Fri, 02 Dec 2011 20:59:17 +0100

No, Quicken does not support 64bit Windows 7 yet. And there are no plans to do so. A few months back, GuacoSoft has released a new version of Attic Manager that is able to load data from Quicken directly. You can then export it into csv, excel, whatever OR simply use Attic Manager to manage the inventory.

Initial version of Attic Manager with this support (2.03) was only able to load data from .MDF files. However, a new version (2.50) is out now that supports .QHI files as well. It can load all data from .MDF. For files with .QHI extension, it loads all the data except image thumbnails. However, if you still keep your original images on the disk in same location where they were when you loaded them into QHIM, the Attic Manager will pick them up while importing and create thumbnails automatically. Not only that, but it will store a copy of each image into it’s database, so that you never lose it in the future.

So far, this is the only way to extract data from Quicken, and it’s really the only Home Inventory program on the market that enables you to transfer all your data before migrating to a new program.

YouTube bug report bug LOL

Tue, 08 Nov 2011 21:29:22 +0100

Looking for a way to report a problem with YouTube software I found a link “Report a bug” a the bottom of the page. However, when I clicked it, I got redirected to:

http://www.google.com/tools/feedback/intl/en/error.html

Which says:

An error has occurred

We are sorry but we were not able to capture your feedback.

Stuck SSH session

Mon, 07 Nov 2011 12:58:03 +0100

When logging from my laptop to remote SSH servers I had a strange problem. Whenever a big chunk of text needs to be returned, my SSH session would stuck and completely stop working. It would not disconnect, but just stay there doing nothing. I would have to log in again. By “big chunk” I mean something like 20+ lines. Output of “ps ax” for example.

This mad me so mad, because if was working on server for a few minutes making sure that I “head” and “tail” every command to reduce output and then I would forget that some command might output more. For example, using “vi” or “mcedit” was completely impossible.

My Internet connection goes through PPPoE. Websites work fine, HTTP works really well, but SSH… no go. The server on the other side is behind a firewall, so tunneling and port forwarding are here.

I searched around, and found that TCP/IP packet size might be the problem, so I tried different MTU values for my PPPoE connection, but without much luck. I was able to get a little bit more before it would stuck again.

And then I landed on this Debian bug report from 2005:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=296811

Apparently still valid. It looks like it only relates to some D-Link routers, although I have no clue what’s at the other end where the server is connected. The solution is to reduce MTU server-side. Luckily, I can still run a one-liner command, and so I did:

/sbin/ifconfig eth0 mtu 1000

Everything runs fine now. I just wonder if this would decrease server through-output on the local LAN where it runs.

Reducing dentry (slab) usage on machines with a lot of...

Mon, 31 Oct 2011 17:48:00 +0100

Reducing dentry (slab) usage on machines with a lot of RAM

Recently I switched my main website from 2-core AMD 4GB RAM machine to 8-core 16GB RAM Intel i7 one. I also switched from CentOS 5 to CentOS 6. I set up everything the same, but suddenly the system was using much more RAM than before. And I’m not talking about filesystem cache here. I thought that increasing RAM would only increase filesystem cache, but something else was occupying RAM like crazy. Looking at output of “free”, “top” and “ps” I simply could not determine what eats RAM because running processes were fine.

So, I googled a little bit, and found that problem was in dentry cache used by Linux kernel. You can see the kernel memory usage with “slabtop” command, and my dentry was crazy, something like 5GB and growing. Googling even more, I found horror stories about servers going down, OOM killing vital processes like Apache or MySQL, etc. So I wanted to stop this.

Quick fix is to clear the cache manually. Some people even “solved” this problem by adding the command to cron job.

echo 2 > /proc/sys/vm/drop_caches

On the MRTG screenshot you can see the dentry cache size in megabytes marked as a blue line. 4000 means 4GB of cache. I have 16GB, remember. When you run the drop_caches command above, you get the effect marked by the red arrow.

I did not like the approach of adding this to crontab, so I investigated further, asked at mailing lists, learned that Linus himself says that “unused memory is dead memory” and that’s why kernel is hungry. Still, I decided to reduce the hunger and added this to /etc/sysctl.conf

vm.vfs_cache_pressure=10000

That did slow it down, but it was still growing. You can run sysctl -p to apply changes to the running kernel without restarting. Next I added these as well:

vm.overcommit_ratio=2
vm.dirty_background_ratio=5
vm.dirty_ratio=20

However, it was still growing, and I decided to leave it be and see what happens. Is my server going to crash, become unavailable, or something. 24 hours later, dentry was again going up like crazy and suddenly it dropped. By itself. See the blue arrow in the screenshot. It seems like kernel figure out that RAM is going to be exhausted, filesystem cache would be reduced, etc. After this point, everything went back to normal.

I tried this experiment again, about a week later, with same results. High-rise, drop and things going back to normal. So, if you’re worried your dentry cache is growing like crazy, don’t. Just tweak those settings in sysctl and wait for at least 48 hours before drawing any conclusions.

Disabling alerts stops JavaScript execution in #Firefox

Sun, 30 Oct 2011 18:34:00 +0100

Today I learned about interesting issue with newer versions of Firefox (I use FF7). It has a nice web developer-friendly feature to disable alerts. This is really useful when you place alert() by mistake in some loop and you can’t get out because as soon as you click OK, you get another one.

New Firefox has a checkbox to disable future alerts. And this is great. So, what’s the problem? Once you disable alerts, and javascript code is executed that would display it, it does not keep running, but rather throws an exception. This does not look like correct behavior to me.

Imagine a web application that alerts user about something and then keeps running to finish the job. If user disabled alerts because he was in a hurry and clicked fast on different message boxes, the script would not keep going but stop. And there is no way to revert that short of reloading the page (yikes!).

I found a workaround, I created a function called tryalert that wraps the alert in try..catch block. It looks like this:

function tryalert(message) 
{ 
    try { alert(message); } catch(e) {} 
}

This is a fine workaround. Now instead of alert() I call tryalert() and although the alert is not displayed anymore, the code keeps going as if user has been alerted.

The problem is introducing tryalert to ALL applications I’ve written so far. It’s impossible. I hope Firefox team changes this.

How to export data from QHI?

Sat, 29 Oct 2011 17:58:20 +0200

How to export data from QHI?:

Unfortunately, Quicken Home Inventory does not work on Windows 7, and you might have a hard time switching to another program because QHI does not have an option to export the data.

However, there’s a way to work around this. A program called Attic Manager, can import the data directly from Quicken database, even if you don’t have Quicken installed. It even works on 64bit Windows. You just need to have your QHI.MDF database backup file.

Once data is in Attic Manager you can export it to CSV format which can be imported into Excel, OpenOffice and almost all the other Home Inventory software. Or, maybe once you try it, you would stick to using Attic Manager.

How to use Quicken Home Inventory on Windows 7 [SOLVED]

Wed, 26 Oct 2011 16:20:10 +0200

If you are looking for a way to use all the data you have already entered on Windows 7 box, you came to the right place. Although the short answer is: “you really can’t do that with QHI”, there is an easy solution to this problem…

There is a nice inexpensive replacement called Attic Manager, which is able to load data from QHI even on Windows 7 computer without Quicken instalation.

It can load locations, categories, items and images (photos) of items.

Most importantly, it runs on all modern operating systems including Windows 7 and various Linux distributions.

If you don’t have access to your old copy of QHI or Quicken Classic, it does not really matter, because Attic Manager can load the data directly from QHI database.

P.S. If you decide to buy it, use the coupon code CNVRT4 to get 40% discount off the price.