Well, I don't know much about mandolin music. It seems to take a back seat or be completely absent in a lot of old-time compared to bluegrass which doesn't do much for me, or in Irish music, which I don't really listen to.
But for banjo, aside from bluegrass, there's a lot of great Dixieland jazz and ragtime that uses it, as well as old-time which is a personal favorite. In that area, check out Bascom Lamar Lunsford, Charlie Poole, Dock Boggs, Uncle Dave Macon, Kelly Harrell, Roscoe Holcomb, Clarence Ashley, as well as basically anything else played clawhammer, 2-finger, or pre-Scruggs 3-finger. Folk-style banjo like the kind Pete Seeger plays is good too. Scruggs style is fine if that's what you're into, and if it is you can just stick to bluegrass till the cows come home, but to me it just sounds soulless.