You may have not implied this, but just in case...
There is not a specific part of your brain that only deals with reading maps, such that if you don't look at maps then that part of your brain turns off.
In the same way that there is not a specific part of your brain that only deals with reading history book in French, such that if you do not read history books in French that part of your brain turns off.
@Scott & Jeff
We live in a world with A LOT of mental stimulation (not less). If you live in a higher civilization, your brain is probably getting a lot more of a "workout" than it would if you lived in a hunter-gather society.
(As an aside this "workout" may be a driver of the Flynn effect.)