> My problem is with the first step: the existence of a super intelligent AI. Why are we sure it can exist?
It's difficult to prove that something that has never been done before is possible, until it has been done. I personally don't see any fundamental limitations that would limit non-biological intelligence to human-level intelligence, for your preferred definition of intelligence.
> And why we are we so sure GPT-x is the path there.
It may or may not be. Regardless, the capabilities of AIs (not just GPTs) are improving exponentially currently.
> To human level intelligence sure, but it's not obvious to me that it will enable superhuman AI
If you think GPTs can get to human-level intelligence, why would the improvement stop at that arbitrary point?
I think that the moment you scale up the hardware for intelligence (from the 20cm brain) to a datacenter scale (to several hundred meters) you start getting into some other kinds of issues, such as it's hard to get a consistent view across the whole "brain". Synchronizing things will take longer, limited by the speed of light: suddenly things on the far ends can communicate with a latency with a lower bound of 100ns. For GPT4 it doesn't matter yet, because all the matrix multiplications take much longer, but maybe once we get to much larger models, that can interact with their environment via something else than a static text box, this will become a limitation.
GPT4 is not an exponential improvement over GPT3. It's is better, but not exponentially so (as in f(x) = e^x).
> If you think GPTs can get to human-level intelligence, why would the improvement stop at that arbitrary point?
Because GPTs are trained on human generated data. They might have some ability to generalize, but to a limited extent. GPT4 is not a super human chess player. It's can play chess, but far from Magnus Carlssen. But we do have super human chess engines, but those are made in a completely different way from GPT.
It's difficult to prove that something that has never been done before is possible, until it has been done. I personally don't see any fundamental limitations that would limit non-biological intelligence to human-level intelligence, for your preferred definition of intelligence.
> And why we are we so sure GPT-x is the path there.
It may or may not be. Regardless, the capabilities of AIs (not just GPTs) are improving exponentially currently.
> To human level intelligence sure, but it's not obvious to me that it will enable superhuman AI
If you think GPTs can get to human-level intelligence, why would the improvement stop at that arbitrary point?