Directly load your IDB file from Quicken Home Inventory on any 64 bit Windows system. It works on 32 bit as well, of course. Today, a new version of Attic Manager is released, version 3.00. This version is able to load data directly from IDB files, there is no need to install any additional software. You don’t even have to have Quicken installed. This also means that you can run this option on 64 bit Windows 7 for example, or even on Linux.

Attic Manager can also load the inventory data from QHI and MDF files. QHI files are also loaded without any additional software.

For MDF files you need to have Microsoft SQL Server Express Edition installed. This is a freeware from Microsoft that comes with QHIM, so if you already have Quicken installed on the same computer, you don’t need to install anything.

In any case, Attic Manager is now unique software on the market, being able to load all Quicken Home Inventory formats and allowing you to keep track of your items on any PC.

There are even hints of Mac version coming soon.

For this particular application, I simply cannot afford it. MyISAM is able to cache index and data separately, using it I can keep the whole index in RAM and website works great. I tried to convert to InnoDB, the result was 4x larger database and 20 times worse performance, mostly due to the fast that index and data gets the same priority for caching so it was killed by disk I/O. If this website was earning enough money to buy at least 4 x more RAM it might not be a bad idea. However, it this case I’d rather use Firebird - it has a similar memory footprint as InnoDB, but has much more features (*real* stored procedures and triggers that work without problems, ability to use table aliases in delete statements, database events functionality, better resistance to system crashes, etc.)

I have been using MySQL for a very intensive read-write web application (averaging 102 queries per second) for more than two years. I had ups and downs with it, like crazy MyISAM behavior that readers can block writers AND OTHER READERS. Basically, a table level lock is issued for read. I have 100+ records in a table, so it takes a while to find anything that is not indexed. In the meantime, users are pondering (102qps, remember) and load goes up so much because of web server processes queuing like crazy. Ok, I learned not to do that anymore. I now use binary logging, restore to a different server and query there. Maybe a switch to InnoDB would be a good idea, but in this case I’d rather use a serious MVCC database like Firebird. Why, you might ask… well, here’s one of many reasons, the one that prompted my to write this:

In Firebird, I can happily do this:

delete from atable a1 
where exists ( 
select 1 from atable a2 where a1.data = a2.data and a1.id <> a2.id );

It just does it, and fast, because index on primary key field ID is used. In MySQL, to quote the manual:

“Currently, you cannot delete from a table and select from the same table in a subquery.”

Come on, this is one of the most basic database operation. So, what am I now to do? Waste my time dumping the list of IDs to delete to some temporary location, and then iterating that list to delete. :(

A few years ago I discovered screen, a nice Linux tool that enables you to detached from terminal with commands running and all in the background. You can even connect later from a different computer and continue where you left off. I initially used it for rtorrent, but now I also use it to administer remote computers, for example when I start to do something that might take more than a day, I can log back in tomorrow. Also loggin in from home/work to complete some task, etc. Another use is administering remote computers on dial-up (yes, there are some) or slow and unstable 3G connections. Even if connection breaks down, I can log in later and pick up where it stopped.

One of the annoying “problems” with screen is that shift+page up/down does not scroll the buffer. This is due to the fact that screen has its own buffers. To work with them you need to enter the “copy mode” using Ctrl+a followed by [. Since I use non-English keyboard that’s Ctrl+a, AltGr+f. Hard to remember when you don’t use it often.

I use Konsole, and I found a way to make it work by adding the following lines to .screenrc (in my home directory):

termcapinfo xterm|xterms|xs|rxvt ti@:te@

Beside being free (both as beer and also open source), you don’t need 24x7 DBA and there are generally less headaches. Here’s a nice example explained by Norman Dumbar in a mailing-list post. Norman administers over 600 Oracle databases and about 40 Firebird ones:

Oracle uses log files for REDO and has ROLLBACK_SEGMENTS or UNDO Segments (depending on Oracle version) for UNDO. It never uses log files for UNDO - and UNDO is what provides Read Consistency/MVCC in an Oracle database.

Changes are written to the LOG_BUFFER (n memory) and periodically - on commit, every 3 seconds max, or when the buffer is 33% full - flushed to the REDO logs. These REDO logs might be archived to disc when they fill up. That Depends on the database archive log mode though.

These logs are used when a database is restored and rolled forward (using the RECOVER DATABASE command, for example).

In order to roll back changes and to ensure read consistency, UNDO is used. These do live on disc - as tablespace files - but remain in memory in the buffer cache alongside data blocks etc.

When a SELECT is started, the data returned are the data from the data blocks. Each row in a block has an indicator that tells when it was last updated. If a pending update is taking place (currently uncommitted) or if a commit has taken place since this SELECT started then the data read from that data block has changed - and is not consistent with the start time of this SELECT transaction.

When this is detected, Oracle “rolls back” the changes to the start time of the SELECT taking place by looking for the UNDO block(s) associated with the transaction that made the changes. If that results in the correct (consistent) data, that’s what you get.

If it turns out that there were other transactions that also changed the data, they too will be detected and undone.

In this way you only ever see data that was consistent at the start of your own transaction.

As long as the DBA correctly sizes the UNDO tablespace and correctly sets the UNDO_RETENTION parameter to a decent enough value, data changes are able to be rolled back happily all the time.

If the DBA failed miserably in his/her duties, the ORA-01555 Snapshot too old” errors are the result. And are most irritating. Long running SELECTS - batch reports for example - tend to show up this error mostly.

Of course, you would never see such problems with Firebird, because the old record versions are stored in database and not the log files. You don’t have to care if system crashes - after reboot it simply works.

You might think that engineers who build Firebird are smarter than Oracle’s but sometimes I think Oracle is deliberately made so complicated to require DBA and also offer them job security. And also makes sure nobody can complain it’s too easy to use.

No, Quicken does not support 64bit Windows 7 yet. And there are no plans to do so. A few months back, GuacoSoft has released a new version of Attic Manager that is able to load data from Quicken directly. You can then export it into csv, excel, whatever OR simply use Attic Manager to manage the inventory.

Initial version of Attic Manager with this support (2.03) was only able to load data from .MDF files. However, a new version (2.50) is out now that supports .QHI files as well. It can load all data from .MDF. For files with .QHI extension, it loads all the data except image thumbnails. However, if you still keep your original images on the disk in same location where they were when you loaded them into QHIM, the Attic Manager will pick them up while importing and create thumbnails automatically. Not only that, but it will store a copy of each image into it’s database, so that you never lose it in the future.

So far, this is the only way to extract data from Quicken, and it’s really the only Home Inventory program on the market that enables you to transfer all your data before migrating to a new program.

Looking for a way to report a problem with YouTube software I found a link “Report a bug” a the bottom of the page. However, when I clicked it, I got redirected to:

http://www.google.com/tools/feedback/intl/en/error.html

Which says:

An error has occurred

We are sorry but we were not able to capture your feedback.

When logging from my laptop to remote SSH servers I had a strange problem. Whenever a big chunk of text needs to be returned, my SSH session would stuck and completely stop working. It would not disconnect, but just stay there doing nothing. I would have to log in again. By “big chunk” I mean something like 20+ lines. Output of “ps ax” for example.

This mad me so mad, because if was working on server for a few minutes making sure that I “head” and “tail” every command to reduce output and then I would forget that some command might output more. For example, using “vi” or “mcedit” was completely impossible.

My Internet connection goes through PPPoE. Websites work fine, HTTP works really well, but SSH… no go. The server on the other side is behind a firewall, so tunneling and port forwarding are here.

I searched around, and found that TCP/IP packet size might be the problem, so I tried different MTU values for my PPPoE connection, but without much luck. I was able to get a little bit more before it would stuck again.

And then I landed on this Debian bug report from 2005:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=296811

Apparently still valid. It looks like it only relates to some D-Link routers, although I have no clue what’s at the other end where the server is connected. The solution is to reduce MTU server-side. Luckily, I can still run a one-liner command, and so I did:

/sbin/ifconfig eth0 mtu 1000

Everything runs fine now. I just wonder if this would decrease server through-output on the local LAN where it runs.

Reducing dentry (slab) usage on machines with a lot of RAM

Recently I switched my main website from 2-core AMD 4GB RAM machine to 8-core 16GB RAM Intel i7 one. I also switched from CentOS 5 to CentOS 6. I set up everything the same, but suddenly the system was using much more RAM than before. And I’m not talking about filesystem cache here. I thought that increasing RAM would only increase filesystem cache, but something else was occupying RAM like crazy. Looking at output of “free”, “top” and “ps” I simply could not determine what eats RAM because running processes were fine.

So, I googled a little bit, and found that problem was in dentry cache used by Linux kernel. You can see the kernel memory usage with “slabtop” command, and my dentry was crazy, something like 5GB and growing. Googling even more, I found horror stories about servers going down, OOM killing vital processes like Apache or MySQL, etc. So I wanted to stop this.

Quick fix is to clear the cache manually. Some people even “solved” this problem by adding the command to cron job.

echo 2 > /proc/sys/vm/drop_caches

On the MRTG screenshot you can see the dentry cache size in megabytes marked as a blue line. 4000 means 4GB of cache. I have 16GB, remember. When you run the drop_caches command above, you get the effect marked by the red arrow.

I did not like the approach of adding this to crontab, so I investigated further, asked at mailing lists, learned that Linus himself says that “unused memory is dead memory” and that’s why kernel is hungry. Still, I decided to reduce the hunger and added this to /etc/sysctl.conf

vm.vfs_cache_pressure=10000

That did slow it down, but it was still growing. You can run sysctl -p to apply changes to the running kernel without restarting. Next I added these as well:

vm.overcommit_ratio=2
vm.dirty_background_ratio=5
vm.dirty_ratio=20

However, it was still growing, and I decided to leave it be and see what happens. Is my server going to crash, become unavailable, or something. 24 hours later, dentry was again going up like crazy and suddenly it dropped. By itself. See the blue arrow in the screenshot. It seems like kernel figure out that RAM is going to be exhausted, filesystem cache would be reduced, etc. After this point, everything went back to normal.

I tried this experiment again, about a week later, with same results. High-rise, drop and things going back to normal. So, if you’re worried your dentry cache is growing like crazy, don’t. Just tweak those settings in sysctl and wait for at least 48 hours before drawing any conclusions.

Today I learned about interesting issue with newer versions of Firefox (I use FF7). It has a nice web developer-friendly feature to disable alerts. This is really useful when you place alert() by mistake in some loop and you can’t get out because as soon as you click OK, you get another one.

New Firefox has a checkbox to disable future alerts. And this is great. So, what’s the problem? Once you disable alerts, and javascript code is executed that would display it, it does not keep running, but rather throws an exception. This does not look like correct behavior to me.

Imagine a web application that alerts user about something and then keeps running to finish the job. If user disabled alerts because he was in a hurry and clicked fast on different message boxes, the script would not keep going but stop. And there is no way to revert that short of reloading the page (yikes!).

I found a workaround, I created a function called tryalert that wraps the alert in try..catch block. It looks like this:

function tryalert(message) 
{ 
    try { alert(message); } catch(e) {} 
}

This is a fine workaround. Now instead of alert() I call tryalert() and although the alert is not displayed anymore, the code keeps going as if user has been alerted.

The problem is introducing tryalert to ALL applications I’ve written so far. It’s impossible. I hope Firefox team changes this.