This is reasonable, but what if the culture that created the artifacts already went extinct like Maya? Besides, we're not only talking about how it shouldn't have been done in the past, but also about what to do today with that past.
It's easy to say that everything bad of today is only because of wrongdoings of yesterday, but it is not useful and usually is only used as propaganda for something that has no justification except for the past being bad.
Edit: although, now that I think about it, coming from this viewpoint, that past is past and we should care about present, it's clear that you're right. If the culture bearer (or the inheritor, but this is grey zone for me) wants to destroy what is rightfully theirs, so be it. There is a bit of an issue with making those decisions by all eligible people, not a couple of extremists, though. Well, I think I found the contradiction that I had in me
It's not an empire if you call it a federation /s