Rare earth minerals aren't actually "rare" as in there's not many of them (https://cleantechnica.com/2018/02/01/renewable-energy-made-available-resources/), it's an archaic term that dates back centuries. The resources used in a solar panel and batteries are also recyclable. Nuclear, on the other hand, relies on a non-recyclable, actually rare resource. And requires a centralized power grid that gives control over your power to a large monopolistic corporation that has no responsibility to treat you like a customer and will inevitably fail in the smallest emergency. Plus, you're ignoring the wide variety of alternative energy sources like wind, geothermal, tidal, hydro, and the fact that solar panel technology is rapidly changing and improving, with development on panels that don't require rare earths.
The ideal, safe nuclear may be clean, but unfortunately we aren't ever going to get that. Managers cut corners, people make mistakes, unpredictable accidents happen, and when it breaks we all have to pay the cost. We can't just assume that the people designing, operating, and maintaining all these nuclear plants won't make any mistakes, and that there won't ever be any successful deliberate sabotage (like the way we used a computer virus to sabotage Iran's development of a nuclear plant), and that every possibility of failure is covered and protected against. Not when the stakes are so high.
I like what you've said here, I've always thought of nuclear power as one of those 'with great power comes great responsibility' types of situations, yes it can be extremely dangerous but on the other hand it is a huge achievement in the progress of energy production, technology, science, and engineering. Im glad that other forms of 'green' energy production are rapidly advancing and becoming more available, but I really think we lost out on advancements in nuclear energy because it's been dismissed as too dangerous- say what you will about the social, economic and political repercussions of that but with this many people on the planet pilfering resources and polluting we may need more than just 'green' energy to get off this rock to be able to continue and sustain this amount of growth and consumption. Or we fall prey to a massive J- curve in the population with some semblance of a near or total extinction event to return to natural balance.