Okay, compare to other 3D applications, Blender is compact, fast, and runs on almost any computer made in the last 20 years. But is it running as best as it could on your system? Below is a few of my own observations on how to optimize settings to get the most speed out of Blender 2.5/2.6.

Enable VBO: VBO is a way to speed up the viewport performance by caching geometry data on the graphic card so that they can be drawn faster than relying on the CPU to send the data every frame.  On scenes with large polycount, VBO can easily quadruple the viewport frame rate. I believe the only reason why it’s off by default is for compatibility with really old Intel graphics card.  All modern Radeon and Geforce graphics should support VBO.  VBO can be turned on from the Preference Panel.

Disable double-sided object: On some graphic cards (older generations circa 2006 or earlier, and the latest Nvidia Fermi),doubled sided lighting makes the video card work a lot harder than normal.  So, to speed up display of high polygon object, disable doubled sided in the object panel should help a bit.  This will make the ‘back’ side of a polygon black, which might not be acceptable in certain cases.  But it makes hunting for reversed-normals even easier.

Enable Only Render: Eventually a scene gets cluttered up with all the lamps, empties, constraints and various other ‘helper’ lines that distracts you from the final look of the image.  Turning on Only Render from the view property panel will hide all the non-renderable objects.  Not only can this give you a clean look at the scene, but also speeds up the viewport performance by quite a bit.

Go 64bit: Almost all the computers out today is 64bit capable.  Running a 64bit of Blender will automagically give you a 20-30% speed boost over the same 32bit binary.   x64 also enables Blender to use more than 2GB of RAM without crashing, which is kinda mandatory for complex scenes.

Use Multi-core CPU: First there is hyper-threading, then dual core, then quad core, and now we have eight-core CPUs WITH hyperthreading.  Luckily, Blender’s main rendering engine is very good at taking advantage of multiple cores.  It will scale up to as many cores as you can buy for rendering.

Special effects calculation like particles, fluid, and smoke is partially multi-threaded, but admittedly they don’t work as well. Having a dual-core helps, but don’t expect astronomical performance on an eight-core CPU.

Operating Systems: The never-ending debate is really not an issue at all. From my personal experience, Windows, Mac and Linux are equally fast when it comes to running Blender.

Optimized Builds: Optimized builds are Blender versions compiled with more agressive optimization for a particular CPU, generally they are 10-20% faster than the official Blender. My experience with them is very positive, they have proven to be as stable as the official release. Graphicall.org hosts many of these builds.

Texture Limit:  When working with large textures in complex sets, setting the texture limit to 256 or 512 can also help view-port performance as it scales every texture down from say 4096 to the chosen limit. This setting can be found in the Preference Window. (Suggested by Daniel via comment)