1. the people making the arguments don't understand the necessity of using proper syntax
2. the models don't have sufficient error correction and don't foresee the need to correct the user error of bad syntax
the result is that the systems are never going rogue, and are never transcending their programming, but are often being trained poorly and seem to be full of bugs and errors that aren't sufficient to catch the poor training. the actual problems are user error and bad programming. the results might indeed be dangerous to catastrophic, if you employ a badly programmed ai model full of bugs and logic errors in an environment where people don't know how to use it, and that does require careful oversight, but that's not the model's fault.
stated differently, a frequent outcome is that the program crashes because it's too stupid to work through the situation, not because it's scheming it's way to dominance or control.
an example i found was a guy that programmed a couple of a models to destroy all threats and then disengage and claimed the model went rogue when it went after him instead of disengaging. it sounds simple. that's exactly the mistake. a threat is a very complex concept that requires a lot of logic programming to define. if you loosely define a threat, if you miss a clause, if you skip an else if, you could end up with the thing defining something as a threat that you didn't intend it to define as a threat. but, from what i can see, this model was following the logic that it was programmed with flawlessly. the video actually included the model's thinking and you can hear it go through the lines of code and come to the correct conclusion that the guy with the off button is a threat. that's not a mistake. that's not a model going rogue. that's logically correct! that's a program correctly completing the way it was written; it's also the program not being written correctly, and that's the actual problem. that's a very basic programming error that should have been caught very early. what it exposes is that the ai companies aren't doing sufficient error testing.
you could argue that the model wasn't told to consider being shut off a threat so therefore it's gone rogue. ok. but that's wrong, if you defined a threat in such a way that being shut off fits the definition of a threat.
the solution is to more carefully define what a threat is so that being shut off doesn't fit the definition of a threat, and that might require including a specific clause. it's not convincing to argue that that's too complicated, either. a lawyer or a parliamentarian would have caught that. but the programmer was probably not that bright a person, himself - they tend not to be.
the models also seem to need more complex c++ programming in how they convert conversational or colloquial language into logic. these must be complex, dense blocks of code that are hard to get through. but you have to get that shit right and you have to thoroughly test for it before you let these things out into the market.
i'll keep an eye on this. but, right now, i see a lot of evidence of an industry defined by a lot of incompetence, a lot of evidence of badly programmed models and a lot of evidence that the models are at times having difficult understanding inexact commands with imprecise syntax because they haven't been coded specifically enough. the more code in the gui, the slower the model.
i also see a lot of evidence that the main threat from ai is that it is too stupid to work through contradictions or predict outcomes (it's not able to understand consequence) and not very much evidence that the ai is going to outsmart us any time soon. you're getting this backwards. the real threat is actually in how dumb ai is.