the researchers are training the ai to be competitive, and when you let the ai loose on to a capitalist internet, it learns how to be competitive. this goes back to the old hobbes v rosseau debate. are humans assholes due to some innate "nature" or are we taught to be assholes by society? i would lean towards the latter, and argue that ai is not going to inherently end up being competitive and evil, but if you teach it to be a competitive asshole then it will be one, just like humans.
then, is a tendency to compete and win a sign of high intelligence or a sign of low intelligence? we only seem to want to ask the conservatives and the capitalists, and then the fake left has a shit when they don't like what the capitalists say.
but i want to propose a different idea - that competitive behaviour is reflective of low intelligence and that if we're training the ai to be competitive then we're functionally teaching the ai to be dumb. the result is the ai will act like stupid humans, because we're training it to act like stupid humans.
the ai might be a very knowledgeable stupid human, like that autistic kid you remember from your school days and you haven't talked to in years and maybe wish you kept in touch with, because they always had something weird to say that was sort of thought provoking, but who you in the end knew was not capable of fully taking care of themselves and needed a support structure to just find a way to eat, in between reciting pi to the 200th decimal place and beating an intricate video game in a weekend.
it's still stupid, at the end of the day and the more we teach it to be stupid the dumber it's going to get.
if ai is a threat, this is how it's a threat - not through super intelligence, but through humans teaching it to act like a primitive ape, and then giving the ape too much control. it's not the mythical leviathan. it's not even a utility monster. it's more like hitchens' dumb primate. it's more like a technocracy merging with an idiocracy.
we cannot expect different. the idea that we can create superintelligence by feeding human writing into a database is ridiculously absurd, and idiotic on it's face. that is an algorithm to create extreme stupidity, not an algorithm to generate superintelligence.
where the system actually works as advertised is in it's raw computing, because it is a super computer. that's actually true. but we're teaching the super computer to be stupid, and it just might do something stupid, because stupid is as stupid does, as the saying goes.