On New York City's Fifth Avenue, the "richest avenue in the world," a tour bus announcer points out the boarded-up townhouse of "industrial wizard" Michael O'Connor (Charles Ruggles), the world's second richest man. As the bus passes, a middle-aged drifter named Aloysious T. McKeever (Victor Moore) and his dog Sam enter the O'Connor house through a loose board in the fence and a manhole, and spend the night.
Meanwhile, O'Connor evicts the tenants of one of his city apartment houses in order to erect an eighty-story building. One of his tenants, Jim Bullock (Don DeFore), an out-of-work veteran, refuses to leave. He is eventually thrown out, and while sleeping on a park bench, meets the drifter, McKeever, or "Mac” as he is lovingly referred to. Mac invites Jim to stay with him at O'Connor's townhouse, which he has occupied for the last three winters while O'Connor resides in Virginia, and Jim assumes that Mac is O'Connor.
Currently, O'Connor is preparing to buy Camp Kilson, a deserted army camp outside Manhattan, in order to build a massive air cargo network. He receives word that his daughter Trudy (Gale Storm) has run away from her finishing school.
When Trudy arrives at the townhouse, Jim concludes that she is a thief, but lets her stay. Trudy quickly falls in love with Jim, and is determined to keep her identity a secret so that he won't love her for her money. When the night patrol arrives to check the house, Mac makes everyone hide in the walk-in freezer and finally confesses to Jim and Trudy that he is an interloper.
Later, Jim runs into two friends from the service, Hank (Edward Ryan) and Whitey (Alan Hale, Jr.), and their wives and children, who are living in a car due to the postwar housing shortage, and invites them to stay at the townhouse, too. With Mac's help, Jim, Whitey and Hank are inspired to design a model to renovate vacant army barracks into housing projects, and decide to bid on Camp Kilson.
Soon O'Connor arrives in New York and finds Trudy leaving for her new job at a music shop. Although he orders her back to school, she insists that she has spent her life being lonely and now wants Jim. O'Connor wants to meet Jim and reluctantly agrees to pose as a drifter, after which Trudy and Jim convince Mac to let O'Connor become another "guest" at the mansion.
It is not long before O'Connor is fed up with his houseguests and threatens to call the police. Trudy sends for her mother, Mary (Ann Harding), who years before reluctantly divorced O'Connor because business was his first priority. Mary and O'Connor rekindle their love for each other, and on Christmas Eve, Mac encourages them to marry, unaware of their true relationship.
When Mary finds out that O'Connor outbid Jim on Camp Kilson, and tried to give him a job in Bolivia, to take him away from Trudy, she is determined to leave him for good this time. After Trudy gets furious with her father, O'Connor lets Jim buy the camp.
On New Year's Eve, the houseguests all celebrate the contract, and prepare to leave the townhouse. As they say their goodbyes, Trudy and Jim and Mary and O'Connor offer Mac a room, he assures them he has a place to stay, at Bubbling Springs, which is O'Connor's house in Virginia. As Mac walks away, O'Connor tells Mary that he has to board up the hole in the fence, because next November, Mac will be coming through the front door.