The best setup I've ever had with pepper spray was on a small piece of line hanging off my belt loop, with a toggle from a stuff sack cinching it down the line. (
http://www.therainshed.com/toggles.htm ) like on this page. When I needed the spray I could just grab it and the toggle would slide out down the line easily, It was just one step faster than having a pepper spray belt holster. When not in use I simply had the line (2 maybe 3 feet long) coiled up in my pocket, just like Highwayman would curl his throw-hook on the boat.
The reality is, half the time when an attack comes you have plenty of warning, half the time you have zero warning. I've been punched in the face so fast I didn't even see it coming. When someone has the element of surprise on you, you really need to have your shit together.
It takes a lot, but is quite possible, to recover from a blow and pull your weapon. I recommend practicing with your friends, so it isn't your first time fighting the first time you get attacked.
A really, really bad day on the street as a man is get robbed and beaten. A really, really bad day on the street as a woman is to get robbed, beaten, and raped.