[Waterbury RR museum]
[They specialize in RR equipment from New England RR's and Conrail. I have never been to it but I have seen it from I84. It is connected to Metro-North's Waterbury branch. It is supposed to be off Exit 36 of Route 8. I got most of this info from the Steam Passenger Service directory.]
You couldn't have seen the museum from I-84, as it's about three or four miles north of there (off Exit 36 of Route 8, as you said). It's also about that far from the Metro North station, and the road going there, Thomaston Avenue, is definitely not pedestrian-friendly.
It may be a railroad museum, but unfortunately you can only get there by car :-(