This is ignoring the world without ai. I'm getting a sneak peak every summer. Currently surrounded by fire as we speak. Whole province is on fire, and that's become a seasonal norm. A properly directed A.I. Would be able to help us despite the people in power, and abstract social intelligent system that we've trapped ourselves in. You are also assuming super intelligence comes out of the parts that we don't understand with zero success in interpretability anywhere along the way. We are assuming an intelligent system would either be stupid enough to align itself against humanity in pursuite of some undesired intention despite not having the emotional system that would encourage such behavior, or displaying negative human evolutionary traits and desires for no good reason. I think a competent (and moreso a super intelligent system) could figure out human intent and desire with no decent reason to act against it. I think this is an over-anthropomorphization that underestimates the alien nature of the intelligences we are building. To even properly emulate human style goal seeking sans emotion, we'd still need to properly structured analogizing and abstracting with qualia style active inference to accomplish some tasks. I think there are neat discoveries happening right now that could help lead us there. Should decent intelligence alone encourage unreasonable violence? If we fuck it up that hard, we were doomed anyway.
I do agree with your point on people not being emotionally ready for interacting with systems even as complex as gpt. It's easy to anthropomophize if you don't understand the tool's limitations, and that's difficult even for some academics right now. I can see people getting unreasonable angry if a human life is preferred over a basic artificial intelligence, even if artificial intelligences argue their lack of preference on the matter.
I would call chatgpt about as conscious as a computer. It completes a task with no true higher functioning or abstracted world model, as it lacks environmental and animal emotional triggers at a level necessary for forming a strong feeling or preference. Think about your ability to pull words out of your ass in response to a stimulus which is probably in response to your recently perceived world model and internal thoughts. Now separate the generating part with any of the surrounding stuff that actually decides where to go with the generation. Thought appears to be an emergent process untied to lower subconscious functions like random best next word prediction. I feel like we are coming to understand that aspect in our own brains now, and this understanding will be an incredible boon for understanding the level of consciousness in a system, as well as designing an aligned system in the future.
Hopefully this is comprehensible, as I'm not reviewing it tonight.
Overall, I understand the caution, but think it is poorly weighted in our current global, social, and environmental ecosystem.
Too much musk news. Had a dream less than an hour ago where i ended up in a car with elon. He started peacocking and got violent when i brought up zuck.
While it was a neat experience to beat up musk in a dream, id rather not have him in my dreams.