It boils down to how much you use your browser and its features. The playing field is not level in terms of "add-ons" right now, but will be shortly... Explanation below.
Firefox has been around for quite some time. It is a very mature product. As a result, it is loaded with add-ons and plugins that extend the features and use of the browser. In regard to how it renders websites, handles downloads, tabs, pop-ups, etc. Firefox does a very good job, and I am most comfortable with it since I've been using it for years on both Mac/Win.
Chrome is a new player in the field. As such, the developer community is still ramping up ways to make it even more useful. Add-ons and such items are slowly coming to market. In order to use any existing add-ons or plug-ins you must use the beta version of Chrome (which is very stable in my findings). As far as the browser standard features, website rendering, download handling, etc, it is a bit more peppy than Firefox. I've only really been able to notice a drastic difference on my high end Quad-Core Mac. On slower machines, the speed really comes into play due to memory and processor handling being much better on Chrome.
You'll see a big shift in the next 12 months in how we use our browsers. Every bit of intelligence is pushing us closer to the Google ChromeOS way of using our computers. This model relies more on the "browser" being the core focal point of how we use computers and accomplish tasks. For another example, look at Apple's new iPad - it runs apps (yes), but the core use of these devices is in the tools we use today (e-mail, web, contacts, calendar, information).