Here's a how to I created for myself, as I had the same issue at first, trying to animate the camera through a scene, "fly through". When I got near a certain point in the scene, the camera slowed to a crippling crawl. This resolved it without having to constantly reset click onto another object each time it crawls.
CAMERA ANIMATION SOLUTION: NO ZOOM SLOW DOWN WHILE ZOOMING CAMERA VIEW
In regards to the debilitating CAMERA VIEW SLOWDOWN that occurs as the camera nears the center pivot of the most recent object it was on, It's a solution with a catch, but you CAN freely animate the camera without ANY slowdown throughout your scene provided the following is done:
1. Set your scene up as normal, with objects, buildings, etc.
2. Far outside the immediate range of your scene, set up a simple plane object, and call it "ZOOMPLANE", which will be your pivot point.
3. Next, click on the "Zoomplane" icon and press the camera ZOOM to zoom to it.
4. Now, with your back directly to your scene, start zooming away backwards from the ZOOMPLANE, keeping lined up with the scene so you don't go off track. At first it's slow, but once you finally reach your scene, you'll have total freedom without any slowdown.
5. Now select the camera ICON you wish to animate. Then begin animating the camera, but under NO circumstances, should you deselect the camera while animating. If you click on another object to animate it, the camera's most recent object PIVOT will be set to the new object, and you'll lose the freedom of movement without slowdown, forcing you to start step one if you wish to continue animating the camera freely.