BEND, Ore. (CNN/Money) – Are two houses better than one? A growing number of second-home owners seem to think so.

The hottest segment of the real estate market has been in coastal towns, mountain towns and other vacation spots.

"Things have really picked up in 2004," said Lynda Traverso, a real estate agent on Sanibel Island, near Fort Myers, Fla. where home prices have risen noticeably over the past couple of years.

Most buyers, she said, are from the Midwest or the East Coast.

Property values on the Jersey Shore, meanwhile, have been bid up by buyers in New York, Philadelphia and Washington D.C. And in California, Lake Tahoe and Palm Springs are big second-home markets.

"Baby boomers are really fueling the purchase of second homes," said David Hehman, CEO of EscapeHomes.com, a site that matches buyers and sellers with real estate agents who specialize in the vacation segment. Traffic on the site increased 20 percent between February and March, in part because this is the time of year people dream of heading for the beach.

EscapeHomes.com recently unveiled its latest list of top 10 destinations for buying a second home, based on traffic on the site and inquiries – not transactions – over the past six months. (See Table)

The most popular destinations, said Hehman vary by season but water and lake property is always a favorite.

The second-home market is difficult to track because, on paper, second-home transactions are no different than those for primary residences, said Walter Molony, a spokesman for the National Association of Realtors (NAR).

To get at just how strong these markets are, David Stiff, a senior economist at Fiserv CSW, turned to Census Bureau statistics on seasonal housing units.

Stiff found 13 counties in which at least 10 percent of single-family homes are seasonal.

Among these counties, the median price appreciation was 49 percent between the fourth quarters of 2000 and 2003. In counties where seasonal homes represented less than 10 percent of housing, meanwhile, the median appreciation was only 32 percent.

In Cape May County in New Jersey, where seasonal housing accounts for 48 percent of the market, home prices increased 68 percent between the fourth quarters of 2000 and 2003. In Barnstable County Massachusetts, better known as Cape Code, 32 percent of all homes are seasonal, and prices are up 59 percent.

Vacation home havens Counties with a high proportion of vacation homes, as measured by Fiserv CSW, showed a median home-price increase of 49% over the past three years, vs. 32% for other markets. Market  Three-year gain  Cape May, NJ (Cape May County)   68%  Island Beach, NJ (Ocean County)   59%  Cape Cod, MA (Barnstable County)   59%  Lake Tahoe, CA (El Dorado County)   51%  Jupiter Island, FL (Martin County)   50%  Brigantine Island, NJ (Atlantic County)   49%  Boca Raton, FL (Palm Beach County)   49%  Gulf Coast, FL (Sarasota County)   42%  Punta Gorda, FL (Charlotte County)   42%  Naples, FL (Collier County)  40%  Sanibel Island, FL (Lee County)   40%  Berkshire Mts., MA (Berkshire County)   29%  Sedona, AZ (Coconino County)  25%   Gains from 4Q 2000 to 4Q 2003

Prices in the Lake Tahoe area appreciated about 51 percent, while prices in and around Sarasota and Fort Myers increased 42 percent and 40 percent respectively.

Said the NAR's Molony, who bought his second home -- a cabin on five acres in West Virginia -- in 1995: "The conventional wisdom when we bought was don't expect to make a lot of money on this and plan to hold onto it for decades." The land alone is now worth the 10 times what he paid.

Because many vacation destinations are adjacent to protected land or on the water, supply is particularly limited, said Hehman. This is one explanation for why properties in these areas have seen more than their share of appreciation.

Still, the primary reason owners of second homes buy, surveys say, is for pure pleasure. "Investment interest seem to come second," Hehman added.

When shopping for a second home, you'll want to take extra care to get to know the area. If it's not close to home, this will take more effort on your part. Plan to visit the spot at least three times and work with a real estate agent who knows the market.

Also keep in mind that if you're financing the property, you may have some extra hurdles than you did with your primary home.

Because lenders typically view second homes as a higher risk, you may pay a slightly higher interest rate, see higher fees and be required to hand over a larger down payment, said Mindy Neubauer, a spokesperson for LendingTree.com. "Many lenders require 20 percent or more down," she said.

According to NAR's Molony, financing a second home has gotten a little easier in recent years. "It used to be that lenders required shorter terms and wouldn't give you a fixed rate," he said.

There are also tax considerations. The interest you pay on your second home's mortgage is tax deductible, assuming you don't rent it out for more than two weeks a year. But, any profit you make on the property is subject to capital gains, unless of course you eventually sell your primary residence, move into your second home and live there for at least two years

Finally, there's homeowners insurance to consider. Insurance may be more expensive because you're not there full time or because your remote retreat is too far from the nearest fire station.

"Our cabin is worth less than half our primary residence yet we're paying a lot more," said Molony, adding that his own premium went up more than 50 percent this year.