I think I've only ever been to one concert at a museum before (Wilco & My Morning Jacket at Buffalo's Albright Knox), but it was an amazing experience. The pairing of Oh! My Blackbird and the Folk Art Museum (2 Lincoln Square near West 66th St) is a great a match, and it should be an exciting show. It takes place Friday (9/28) and starts at 5:30. This is part of a free music series and also includes Rebecca Hart and Kwesi Kankam on the bill. Event link here.
Town Hall spent the summer playing throughout the country, both acoustically and with their full band. Their gig on Saturday (9/29) is the first in NYC of the fall season. I have seen them live before and highly recommend it. The show is at Shea Stadium (20 Meadow St in Brooklyn). The doors are at 9:00, the cost is $7, and the other bands are Plume Giant and the Due Diligence. You can get more details on the show, as well as the link to RSVP for the event on Facebook, here.