Unfortunately, I need to get all the results, to find out which books are available, as the default search results page is to show only available books. Since the search index has ALL books, not just available ones, I need to get all the results anyway.

At any rate, this problem seems to be solved for now, as a 1GB in-memory cache seems to have had a great performance impact, and I’ve been running that for a few days, as the outcome of my various techniques found that to work best.

You make some very good points.

It’s true that nobody wants to look through 500,000 books with the word “science” in their title or topic, as a result of searching for “science”.

If there are only 20 books returned to a search, sorted results are nice.

However, I think that search results might be more interesting sorted by “relevancy”, which probably means moving the most-popular books to the top of the list, either by using the “number of times mooched” or “amazon sales ranking”.

This kind of sort would probably be more helpful for any list of books that goes on for more than 2 pages.

For inventories and small lists, yes, I want to see an author’s titles grouped together. If someone searches for a general term though are they really after an alphabeticalised list?

Alphabetical order is useful for finding a specific thing in a list, but if you want to find a specific thing among that many items then you can search for it using a specific term. A list of half a million items isn’t what a human wants to read even if it’s in order.

Yes, I know that I will eventually need to use those techniques, and the bookmooch architecture already supports it.

If you want to read about BerkeleyDB’s replication technology, take a look at http://www.oracle.com/technology/products/berkeley-db/feature-sets.html — this is replicated one-writer, multiple-reader, and will scale well as long as data-writing is kept under control. What I’m currently experimenting with, is what the effects of not using disk-based caching are to bookmooch performance, since writes are both slow, but also perform poorly in a replicated environment.

This replicated BerkeleyDB is currently in use by Google for their “single sign-on” feature.

I do know all about memcached, and its persistent variant, called memcachedb, actually uses BerkeleyDB as its engine http://memcachedb.org/

So do not worry, BookMooch can scale to very great heights. The issue for me now is to get the most I can out of one machine, so that costs will be minimized when I need to go to a multi-machine setup.

As BookMooch grows, adding more RAM and switching to Solid State Drives are really just temporary solutions that won’t scale very well, or for very long. If you put these in place, before long you’ll be right back in the same position, but eventually you won’t be able to add more RAM or more SSDs. You need a solid architecture redesign.

You’re getting into the area of needing to think about database sharding and whatnot. Also, memcached is a wonderful tool to aid in caching queries and dynamic pages. There are probably hundreds of areas where a little bit of cache will help speed things up, but as far as the database goes, you’re definitely going to need to start considering some type of partitioning and clustering.

I’ve used RAID in the past, and always have had data loss issues, as have other sysadmins I know. Google doesn’t use RAID, and neither does the Internet Archive, and I know the Internet Archive tried, but had RAID reliability issues as well.

Also, RAID5 is faster, but not 300x faster, like a solid-state-drive vs a traditional fixed-disk drive.