Backward compatible

New way of spamming: taking over free e-mail accounts

Mon, 13 Feb 2012 11:23:43 +0100

Today I found out something really interesting. I send an e-mail newsletter to some 380.000+ subscribers. Suddenly, I got a flood of auto-reply messages, coming from various Yahoo mail users. The auto-reply message contains different spammy content with links, all written in a very personalized manner. Looks like spammers were able to hack into user accounts, most probably using easy to guess passwords or password recovery questions, and then set up a nice auto-reply.

I applaud Yahoo for not allowing the change of the subject line. They all start with Auto-reply so it’s very easy to see through the scam. Nevertheless, if you have same auto-reply for thousands of users, it should really ring some alarm in Yahoo mail team, shouldn’t it?

How to use IDB files of Quicken Home Inventory on 64 bit Windows?

Mon, 09 Jan 2012 23:17:17 +0100

Directly load your IDB file from Quicken Home Inventory on any 64 bit Windows system. It works on 32 bit as well, of course. Today, a new version of Attic Manager is released, version 3.00. This version is able to load data directly from IDB files, there is no need to install any additional software. You don’t even have to have Quicken installed. This also means that you can run this option on 64 bit Windows 7 for example, or even on Linux.

Attic Manager can also load the inventory data from QHI and MDF files. QHI files are also loaded without any additional software.

For MDF files you need to have Microsoft SQL Server Express Edition installed. This is a freeware from Microsoft that comes with QHIM, so if you already have Quicken installed on the same computer, you don’t need to install anything.

In any case, Attic Manager is now unique software on the market, being able to load all Quicken Home Inventory formats and allowing you to keep track of your items on any PC.

There are even hints of Mac version coming soon.

Why aren't you using InnoDB instead of MyISAM?

Wed, 04 Jan 2012 19:07:40 +0100

For this particular application, I simply cannot afford it. MyISAM is able to cache index and data separately, using it I can keep the whole index in RAM and website works great. I tried to convert to InnoDB, the result was 4x larger database and 20 times worse performance, mostly due to the fast that index and data gets the same priority for caching so it was killed by disk I/O. If this website was earning enough money to buy at least 4 x more RAM it might not be a bad idea. However, it this case I’d rather use Firebird - it has a similar memory footprint as InnoDB, but has much more features (*real* stored procedures and triggers that work without problems, ability to use table aliases in delete statements, database events functionality, better resistance to system crashes, etc.)

Why is MySQL still a toy database

Tue, 03 Jan 2012 12:53:59 +0100

I have been using MySQL for a very intensive read-write web application (averaging 102 queries per second) for more than two years. I had ups and downs with it, like crazy MyISAM behavior that readers can block writers AND OTHER READERS. Basically, a table level lock is issued for read. I have 100+ records in a table, so it takes a while to find anything that is not indexed. In the meantime, users are pondering (102qps, remember) and load goes up so much because of web server processes queuing like crazy. Ok, I learned not to do that anymore. I now use binary logging, restore to a different server and query there. Maybe a switch to InnoDB would be a good idea, but in this case I’d rather use a serious MVCC database like Firebird. Why, you might ask… well, here’s one of many reasons, the one that prompted my to write this:

In Firebird, I can happily do this:

delete from atable a1 
where exists ( 
select 1 from atable a2 where a1.data = a2.data and a1.id <> a2.id );

It just does it, and fast, because index on primary key field ID is used. In MySQL, to quote the manual:

“Currently, you cannot delete from a table and select from the same table in a subquery.”

Come on, this is one of the most basic database operation. So, what am I now to do? Waste my time dumping the list of IDs to delete to some temporary location, and then iterating that list to delete. :(

Scrolling back in screen

Fri, 30 Dec 2011 13:26:43 +0100

A few years ago I discovered screen, a nice Linux tool that enables you to detached from terminal with commands running and all in the background. You can even connect later from a different computer and continue where you left off. I initially used it for rtorrent, but now I also use it to administer remote computers, for example when I start to do something that might take more than a day, I can log back in tomorrow. Also loggin in from home/work to complete some task, etc. Another use is administering remote computers on dial-up (yes, there are some) or slow and unstable 3G connections. Even if connection breaks down, I can log in later and pick up where it stopped.

One of the annoying “problems” with screen is that shift+page up/down does not scroll the buffer. This is due to the fact that screen has its own buffers. To work with them you need to enter the “copy mode” using Ctrl+a followed by [. Since I use non-English keyboard that’s Ctrl+a, AltGr+f. Hard to remember when you don’t use it often.

I use Konsole, and I found a way to make it work by adding the following lines to .screenrc (in my home directory):

termcapinfo xterm|xterms|xs|rxvt ti@:te@

Why is Firebird better DBMS than Oracle?

Fri, 16 Dec 2011 12:07:33 +0100

Beside being free (both as beer and also open source), you don’t need 24x7 DBA and there are generally less headaches. Here’s a nice example explained by Norman Dumbar in a mailing-list post. Norman administers over 600 Oracle databases and about 40 Firebird ones:

Oracle uses log files for REDO and has ROLLBACK_SEGMENTS or UNDO Segments (depending on Oracle version) for UNDO. It never uses log files for UNDO - and UNDO is what provides Read Consistency/MVCC in an Oracle database.

Changes are written to the LOG_BUFFER (n memory) and periodically - on commit, every 3 seconds max, or when the buffer is 33% full - flushed to the REDO logs. These REDO logs might be archived to disc when they fill up. That Depends on the database archive log mode though.

These logs are used when a database is restored and rolled forward (using the RECOVER DATABASE command, for example).

In order to roll back changes and to ensure read consistency, UNDO is used. These do live on disc - as tablespace files - but remain in memory in the buffer cache alongside data blocks etc.

When a SELECT is started, the data returned are the data from the data blocks. Each row in a block has an indicator that tells when it was last updated. If a pending update is taking place (currently uncommitted) or if a commit has taken place since this SELECT started then the data read from that data block has changed - and is not consistent with the start time of this SELECT transaction.

When this is detected, Oracle “rolls back” the changes to the start time of the SELECT taking place by looking for the UNDO block(s) associated with the transaction that made the changes. If that results in the correct (consistent) data, that’s what you get.

If it turns out that there were other transactions that also changed the data, they too will be detected and undone.

In this way you only ever see data that was consistent at the start of your own transaction.

As long as the DBA correctly sizes the UNDO tablespace and correctly sets the UNDO_RETENTION parameter to a decent enough value, data changes are able to be rolled back happily all the time.

If the DBA failed miserably in his/her duties, the ORA-01555 Snapshot too old” errors are the result. And are most irritating. Long running SELECTS - batch reports for example - tend to show up this error mostly.

Of course, you would never see such problems with Firebird, because the old record versions are stored in database and not the log files. You don’t have to care if system crashes - after reboot it simply works.

You might think that engineers who build Firebird are smarter than Oracle’s but sometimes I think Oracle is deliberately made so complicated to require DBA and also offer them job security. And also makes sure nobody can complain it’s too easy to use.

Using Quicken Home Inventory Manager on 64bit systems [FINALLY]

Fri, 02 Dec 2011 20:59:17 +0100

No, Quicken does not support 64bit Windows 7 yet. And there are no plans to do so. A few months back, GuacoSoft has released a new version of Attic Manager that is able to load data from Quicken directly. You can then export it into csv, excel, whatever OR simply use Attic Manager to manage the inventory.

Initial version of Attic Manager with this support (2.03) was only able to load data from .MDF files. However, a new version (2.50) is out now that supports .QHI files as well. It can load all data from .MDF. For files with .QHI extension, it loads all the data except image thumbnails. However, if you still keep your original images on the disk in same location where they were when you loaded them into QHIM, the Attic Manager will pick them up while importing and create thumbnails automatically. Not only that, but it will store a copy of each image into it’s database, so that you never lose it in the future.

So far, this is the only way to extract data from Quicken, and it’s really the only Home Inventory program on the market that enables you to transfer all your data before migrating to a new program.

YouTube bug report bug LOL

Tue, 08 Nov 2011 21:29:22 +0100

Looking for a way to report a problem with YouTube software I found a link “Report a bug” a the bottom of the page. However, when I clicked it, I got redirected to:

http://www.google.com/tools/feedback/intl/en/error.html

Which says:

An error has occurred

We are sorry but we were not able to capture your feedback.

Stuck SSH session

Mon, 07 Nov 2011 12:58:03 +0100

When logging from my laptop to remote SSH servers I had a strange problem. Whenever a big chunk of text needs to be returned, my SSH session would stuck and completely stop working. It would not disconnect, but just stay there doing nothing. I would have to log in again. By “big chunk” I mean something like 20+ lines. Output of “ps ax” for example.

This mad me so mad, because if was working on server for a few minutes making sure that I “head” and “tail” every command to reduce output and then I would forget that some command might output more. For example, using “vi” or “mcedit” was completely impossible.

My Internet connection goes through PPPoE. Websites work fine, HTTP works really well, but SSH… no go. The server on the other side is behind a firewall, so tunneling and port forwarding are here.

I searched around, and found that TCP/IP packet size might be the problem, so I tried different MTU values for my PPPoE connection, but without much luck. I was able to get a little bit more before it would stuck again.

And then I landed on this Debian bug report from 2005:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=296811

Apparently still valid. It looks like it only relates to some D-Link routers, although I have no clue what’s at the other end where the server is connected. The solution is to reduce MTU server-side. Luckily, I can still run a one-liner command, and so I did:

/sbin/ifconfig eth0 mtu 1000

Everything runs fine now. I just wonder if this would decrease server through-output on the local LAN where it runs.

Reducing dentry (slab) usage on machines with a lot of...

Mon, 31 Oct 2011 17:48:00 +0100

Reducing dentry (slab) usage on machines with a lot of RAM

Recently I switched my main website from 2-core AMD 4GB RAM machine to 8-core 16GB RAM Intel i7 one. I also switched from CentOS 5 to CentOS 6. I set up everything the same, but suddenly the system was using much more RAM than before. And I’m not talking about filesystem cache here. I thought that increasing RAM would only increase filesystem cache, but something else was occupying RAM like crazy. Looking at output of “free”, “top” and “ps” I simply could not determine what eats RAM because running processes were fine.

So, I googled a little bit, and found that problem was in dentry cache used by Linux kernel. You can see the kernel memory usage with “slabtop” command, and my dentry was crazy, something like 5GB and growing. Googling even more, I found horror stories about servers going down, OOM killing vital processes like Apache or MySQL, etc. So I wanted to stop this.

Quick fix is to clear the cache manually. Some people even “solved” this problem by adding the command to cron job.

echo 2 > /proc/sys/vm/drop_caches

On the MRTG screenshot you can see the dentry cache size in megabytes marked as a blue line. 4000 means 4GB of cache. I have 16GB, remember. When you run the drop_caches command above, you get the effect marked by the red arrow.

I did not like the approach of adding this to crontab, so I investigated further, asked at mailing lists, learned that Linus himself says that “unused memory is dead memory” and that’s why kernel is hungry. Still, I decided to reduce the hunger and added this to /etc/sysctl.conf

vm.vfs_cache_pressure=10000

That did slow it down, but it was still growing. You can run sysctl -p to apply changes to the running kernel without restarting. Next I added these as well:

vm.overcommit_ratio=2
vm.dirty_background_ratio=5
vm.dirty_ratio=20

However, it was still growing, and I decided to leave it be and see what happens. Is my server going to crash, become unavailable, or something. 24 hours later, dentry was again going up like crazy and suddenly it dropped. By itself. See the blue arrow in the screenshot. It seems like kernel figure out that RAM is going to be exhausted, filesystem cache would be reduced, etc. After this point, everything went back to normal.

I tried this experiment again, about a week later, with same results. High-rise, drop and things going back to normal. So, if you’re worried your dentry cache is growing like crazy, don’t. Just tweak those settings in sysctl and wait for at least 48 hours before drawing any conclusions.

Disabling alerts stops JavaScript execution in #Firefox

Sun, 30 Oct 2011 18:34:00 +0100

Today I learned about interesting issue with newer versions of Firefox (I use FF7). It has a nice web developer-friendly feature to disable alerts. This is really useful when you place alert() by mistake in some loop and you can’t get out because as soon as you click OK, you get another one.

New Firefox has a checkbox to disable future alerts. And this is great. So, what’s the problem? Once you disable alerts, and javascript code is executed that would display it, it does not keep running, but rather throws an exception. This does not look like correct behavior to me.

Imagine a web application that alerts user about something and then keeps running to finish the job. If user disabled alerts because he was in a hurry and clicked fast on different message boxes, the script would not keep going but stop. And there is no way to revert that short of reloading the page (yikes!).

I found a workaround, I created a function called tryalert that wraps the alert in try..catch block. It looks like this:

function tryalert(message) 
{ 
    try { alert(message); } catch(e) {} 
}

This is a fine workaround. Now instead of alert() I call tryalert() and although the alert is not displayed anymore, the code keeps going as if user has been alerted.

The problem is introducing tryalert to ALL applications I’ve written so far. It’s impossible. I hope Firefox team changes this.

How to export data from QHI?

Sat, 29 Oct 2011 17:58:20 +0200

How to export data from QHI?:

Unfortunately, Quicken Home Inventory does not work on Windows 7, and you might have a hard time switching to another program because QHI does not have an option to export the data.

However, there’s a way to work around this. A program called Attic Manager, can import the data directly from Quicken database, even if you don’t have Quicken installed. It even works on 64bit Windows. You just need to have your QHI.MDF database backup file.

Once data is in Attic Manager you can export it to CSV format which can be imported into Excel, OpenOffice and almost all the other Home Inventory software. Or, maybe once you try it, you would stick to using Attic Manager.

How to use Quicken Home Inventory on Windows 7 [SOLVED]

Wed, 26 Oct 2011 16:20:10 +0200

If you are looking for a way to use all the data you have already entered on Windows 7 box, you came to the right place. Although the short answer is: “you really can’t do that with QHI”, there is an easy solution to this problem…

There is a nice inexpensive replacement called Attic Manager, which is able to load data from QHI even on Windows 7 computer without Quicken instalation.

It can load locations, categories, items and images (photos) of items.

Most importantly, it runs on all modern operating systems including Windows 7 and various Linux distributions.

If you don’t have access to your old copy of QHI or Quicken Classic, it does not really matter, because Attic Manager can load the data directly from QHI database.

P.S. If you decide to buy it, use the coupon code CNVRT4 to get 40% discount off the price.

Building wxWidgets 2.8.12 on old MinGW with GCC 3.2

Wed, 26 Oct 2011 12:48:00 +0200

I had a application using wxWidgets 2.8.0 and then 2.8.8 in production. There were some bugs in earlier wxWidgets versions on Linux, so printing was not working properly. I decided to upgrade wx and that fixed it. Now I wanted to use the same version for Windows version of my application. I originally used some (now old) MinGW version and just wanted to rebuild and be done. But, I got build errors instead. I don’t really last time wxWidgets failed to build so I asked at mailing list and finally dug into the source code myself.

It looks like wx code is all fine, but there are problems in MinGW headers. I particular, you need to edit the file C:\MinGW\include\winspool.h and change DocumentPropertiesW function’s signature from:

LONG WINAPI DocumentPropertiesW(HWND,HANDLE,LPWSTR,PDEVMODEA,PDEVMODEA,DWORD);

to:

LONG WINAPI  DocumentPropertiesW(HWND,HANDLE,LPWSTR,PDEVMODEW,PDEVMODEW,DWORD);

It seems to be already fixed in newer MinGW versions.

nginx hogs cpu when proxying large files

Thu, 06 Oct 2011 18:46:45 +0200

I have a server where nginx is used as frontend for Apache. nginx serves static content and Apache serves PHP pages. This is a common setup.

Today I migrated stuff to a new server and needed to copy a 7GB database file to another server. I figured HTTP would be fastest way to do it. Unfortunatelly, DNS change already went thought so I could not serve the file on the static domain nginx was configured for.

I thought “nevermind”, placed file under one of those domains handled by Apache and started the download. It was going fine at 11MB/s for some time. However, soon it started to crawl at 850KB/s. I suspected network problems, but everything else was running fine. I looked at process list and whoa, nginx using 99% of CPU. Because of this single download, the server was brought to its knees and no other client could even get a simple “Hello World” page.

I stopped the download at the client side and nginx soon recovered (not restart needed). Then I edited /etc/hosts and place the old IP address of static domain and continued the download (wget -c). It finished few minutes later with 11MB/s average.

Merging a huge git conflict

Mon, 03 Oct 2011 08:16:23 +0200

Both me and my colleague work separately working on same git tree while being offline for a couple of days. Result: following “git pull” I got a huge conflict spanning about 100 of files.

This meant that manual resolution is out of question. Enter “git mergetool” and “kdiff3”. I installed kdiff3 from linuxpackages.net (version is for old Slackware 11.0, and I had to symlink /opt/kde/kdiff3 to /usr/bin/kdiff3 so that git finds it).

git-mergetool calls kdiff3 for each file, you merge and save. Job done very quickly.

Google Apps problems

Thu, 22 Sep 2011 18:08:00 +0200

Seems to be a fine day at Google today, perhaps engineers are pulling hair.

This morning, I was looking at a spreadsheet in Google Docs and suddenly some 20 values simply vanished right before my eyes. I wasn’t even working anywhere near that part of the sheet. I was inserting new values at bottom and somewhere in top-right corner the values were gone. I tried undo and to scroll around (big sheet) and only when I switched to another sheet and came back the values showed up again. Phew. From now on I’m doing export and download to my computer every time I finish editing.

Few hours later, a new issue. Looking at a spreadsheet I selected Save from the menu. It said that it’s ok. I did some changes, clicked Save, got no error but the screen read “Last saved 2 minutes ago”. Ok, maybe it’s just a minor glitch. 15 minutes later I tried to save again. Once again, no errors, but it still says “saved 17 minutes ago”. At this point I was confused whether save is not possible or the message is simply wrong. I exported the document to xls format, checked in OpenOffice and then closed the browser tab.

Three strikes and blog post is out. I just had another issue, now with GMail, so I guess it’s time to make all this public. I wrote an e-mail message and it said “Your connection to GMail has expired. Please log in again.” Ok, it’s not like I haven’t seen that one before, but it’s been almost a month. I though they had it fixed. I logged out, logged back in and… it still does not allow me to send an e-mail. I can read messages fine, but as soon as I try to post, I get a warning that “Your connection to GMail has expired. Please log in again.”.

Oh well, I guess we get as much as we payed for it ;)

Stackoverflow.com scaling problem

Sun, 18 Sep 2011 13:33:00 +0200

I have been stackoverflow user almost from the very start of the website. I recall reading some Jeff Atwood’s blog posts and thinking how naive he is. He has a classic case of Microsoft fanboy-ism. He swears by .net and MSSQL server and spits on Linux, PHP and… well… entire LAMP stack.

When stack became popular the website started to get a lot of traffic and Jeff was all like “Oh we don’t need all the scaling technology that all the web companies have developed since web 1.0 till today. We’re smarter, we use the all-powerful Microsoft stack, we’ll just buy more RAM, more CPU and keep it all on single machine. Machines are so powerful these days and cost almost nothing”. How little did he know.

As more and more people use the website it seems that they reached the limit of what is possible. Stack website is now inaccessible for days. By inaccessible, I don’t mean that the site does not open. It just open waaaay too slow to be usable. I sometimes wait 5 minutes to get the home page.

What I really regret is all those dumb readers on Jeff’s codinghorror blog, and all those fanboys on stack website. Some people tried to tell Jeff that this would happen, but he would not listen. He was very arrogant and dismissed all that as LAMP-crap. All his followers blindly followed his thoughts as if they really wanted that to be true. Psychology of a herd, I’d say.

Oh well, too bad that public access to such valuable resource is now limited because of stubborn owners. Maybe it’s time for a real competitor to step up, with a simple slogan: “just like stackoverflow, except that it really works”.

Performance of PHP if, switch, arrays... using "B-tree-if" as solution

Thu, 01 Sep 2011 22:23:16 +0200

For the game I’m making I have a bunch of array which represent a puzzle player needs to solve. Ensuring puzzle is solvable is CPU intensive, so I pre-calculated a couple of thousands of puzzles and select one randomly. Since puzzles are static data which is not going to change, I decided not to burden the database with this because DBMS is always overloaded with other stuff anyway.

My first thought was to build a big array and fetch a random element. I found some benchmark showing that this is faster than “if” or “switch”, however benchmarks excluded time needed to parse/create the array itself. Since every player is a new HTTP request, this huge array would have to be constructed each time. I am using APC, but I failed to find if arrays in precompiled PHP source file are stored “structured” as well.

Dropping the array idea, I though about “switch”, foolishly thinking that it would use some kind of jump-table and run the desired return statement. Something like this:

function get_puzzle($id)
{
    switch ($id)
    {
        case 0: return array (...first puzzle...);
        case 1: return array (...second puzzle...);
        case 2: return array (...etc.

However, researching this I find out that switch performs similar to series of “if” statements… variable is compared with each “case”.

So I decided to roll my own solution using “if” statements, but not linear. I used a B-tree approach. Split by two until only one is left. This means it would take only 11 comparisons to reach a puzzle from a set of 2048. Here’s an example with set of 256 puzzles.

function get_puzzle_btree($id)
{
  if ($id < 128)
    if ($id < 64)
      if ($id < 32)
        if ($id < 16)
          if ($id < 8)
            if ($id < 4)
              if ($id < 2)
                if ($id < 1)
                  return array (...first puzzle...);
                else
                  return array (...second puzzle...); 
...etc.

Of course, I did not write this “if” behemoth by hand. A simple 20-line recursive function spits out the PHP code.

In the end, I wrote a simple comparison loop that tries to get all the puzzles and compares whether the old “switch” and new “btree” function return the same values.

Safe way to dual-boot Linux and Windows 7

Thu, 01 Sep 2011 11:44:45 +0200

I had a client’s machine installed with Windows 7 and some free hard disk space for Linux. I decided not to install the Linux boot loader because:

I did not have Windows install/rescue CD at hand
in case something goes wrong I could not boot into Windows
I had some experience in the past with Windows XP where it simply did not work

Since re-installing Windows or even fixing Windows if it became unbootable was not an option, I decided to play safe: use Windows’ boot loader to boot up Linux.

I did this in past with Windows XP. Basically, you save Linux boot loader into some file (it’s only 512 bytes) and then tell Windows’ boot loader to load it. On WindowsXP this means editing boot.ini file in C:. To create the linux boot loader file, install linux boot loader into root partition (for example, with LILO, if you installed Linux in /dev/sda4, then lilo.conf should read boot=/dev/sda4) and then read the first sector into a file:

dd if=/dev/sda4 of=linux.boot bs=512 count=1

This will create file named linux.boot which you need to copy to C:\ disk of your Windows machine (use the USB stick or network for this).

On Windows7 there is no boot.ini, you have to use Microsoft’s tool, named BCDEdit. BCD stands for Boot Configuration Data. You need to run BCDedit as administrator. Hit the Start button, then go to All programs and then to Accessories. Right-click the Command prompt and “Run as administrator”.

Now, we need to enter a couple of commands:

bcdedit /create /d "Linux" /application BOOTSECTOR

If will show something like

The entry {12345678-0000-1111-9999-112233445566} was successfully created.

That number is a unique identifier for boot menu entry. You need to use it in subsequent commands:

bcdedit /set {12345678-0000-1111-9999-112233445566} device boot
bcdedit /set {12345678-0000-1111-9999-112233445566} device partition=c:
bcdedit /set {12345678-0000-1111-9999-112233445566} PATH \linux.boot
bcdedit /displayorder {12345678-0000-1111-9999-112233445566} /addlast

You might need to prepend C: in the second line if it does not work this way.

Reboot and enjoy.