I was hoping this would be about a new way to augur the events of the day from my morning toast to ensemble with the tea leaves at the bottom of my morning cuppa, alas ...
Really? I'd love to read some write ups on this because I'm still doing it the old way. I was going to say that corporate experience runs counter to this but maybe not; you think the architect is more important than the developer, but actually there have been shifts over time to recognise ICs more and keep them in their roles while managers are more generic. Interesting food for thought!
A related point that I've been making privately for at least ten years to anyone that would listen, is that we've had Superhuman Intelligence for at least hundreds, if not thousands, of years in the form of corporations and institutions.
No single human could design a rocket to go to the moon or keep all Wallmart stores supplied on a daily basis. Moreover when something like the CEO or US president changes, the policies and actions change a bit but the majority actually stays the same. This is collective intelligence embodied in the organization and its institutional memory. This is also what i believe is wrong with Searle's Chinese Room argument - the algorithm is the intelligence.
Our cells and neurons act based on local chemical gradients and electrical signals and we wouldn't call any single one of those particularly intelligent. Yet collectively they produce human intelligence and even what we call consciousness!
I've made that point before. From an "alignment" perspective, AIs and corporations have many of the same problems. Corporations, according to Milton Friedman, have the sole duty of increasing their profits.[1] This is a key document in the history of modern conservatism.[2]
So what happens when AIs are "aligned" purely to corporate goals? This is a nearer term worry than machines taking over for the benefit of machines. AIs working hard for the benefit of corporations are already here. Try any customer service chatbot.
There's reluctance to think about this too hard. It raises embarrassing questions about the role of the corporation in society. It can become a challenge to corporate personhood.
That's threatening to many people in power. Dealing with it may be unavoidable as AIs take a stronger role in managing companies.
Also the accountability problem! Sometimes we try to hold the corporation accountable and sometimes the individuals. It never seems adequate though.
I found the documentary "The Corporation" (2003) very instructive in that regard, particularly the segment with the BP CEO where the point is made that a company made up ethical individuals can still act unethically, or unaligned to human values, just by virtue of the incentive and legal structure governing the corporation.
“ This is also what i believe is wrong with Searle's Chinese Room argument - the algorithm is the intelligence.”
“ and even what we call consciousness!”
This misunderstands the point of the so-called Chinese room thought experiment. You want to conclude that consciousness is functional, but you cite a thought experiment which argues it is not, and then assert the thought experiment is wrong because it disagrees with your preconceived notion of consciousness.
Thanks for the reply. I was hoping to get some debate going and have my thoughts challenged in order to see how they hold up under scrutiny. You're right that I believe in Functionalism and Strong AI and in previous comment I only really stated my belief so please allow me to try and make a bit of an argument for it. My position probably falls under the "System reply" to the Chinese Room argument although I feel it's more restrictive in some aspects so perhaps it corresponds to a particular subtype (I don't know as I haven't read enough in the area).
My basic argument is that human *intelligence* (and understanding - I consider understanding as part of intelligence and consciousness separately but I can see how many might group understanding with consciousness and leave intelligence on its own) is a form of collective intelligence. Individual neurons acting based on local chemical and electrical signals collectively produce behaviour that we call intelligent, i.e. it can solve problems, respond to questions, etc ... . A particular configuration of neurons acting together I will call a particular software or algorithm (I understand that separating the software from the neural substrate in that way is part of what's in dispute, particularly by Searle, but that's my belief and what I feel is required to make the mapping to the Chinese Room argument). The Chinese Room thought experiment then is trying to refute Strong AI by arguing that the person in the Chinese Room has no understanding of the exchange with the outside world because they are just manipulating symbols incomprehensible to them. I agree that the person has no understanding. That person on their own would not be able to string together a sensible reply without the instruction cards. However another person with the same instruction cards, i.e. algorithm, would be able to reply sensibly, pass a Chinese Turing test, etc ..., thus demonstrating that the intelligence/understanding resides in the algorithm. The person is just an execution substrate. The mistake in the Chinese Room argument is identifying the locus of understanding in the person, which imho is failure of imagination. It's akin to saying that humans can have no understanding because each individual neuron has no understanding of our actions, something which the existence proof of human intelligence refutes.
What then of consciousness? Here I also have some beliefs which I'm happy to write down but in my assessment they're less substantiated so make of them what you will. I saw some things by Joscha Bach recently and I liked what he said about agents and consciousness as a form self-recognition of the agent as an entity in the environment and predictive modeling of future states of that entity in order to achieve the agents goals and realise some form of agency. I don't know where neuroscience and cognitive science stand on this but made sense to me as a sensible model of consciousness. Therefore I would even go so far as to say that I believe that entities such as the United States of America (and any country or institution really) are probably conscious in some form, because they are aware of themselves, model themselves, and try to project their agency in the environment that they find themselves in. We are not directly aware of this because we are like single neurons in our brain and this happens at a scale much larger than us. Moreover I suspect that as you go up each rung on the emergence ladder, the temporal scale of the phenomena gets an order slower as that level is composed of the interactions at the level below it. So the speed of interactions of our neurons is an order faster than our conscious processes and similarly the emergent bodies and institutions that we form part of act an order slower so their processes are more difficult for us to perceive.
Schematically then my beliefs organise roughly as follows:
- intelligence = algorithm: think neural network weights you can download of HuggingFace
- consciousness = self-aware world model + state: I speculate that the same NN world model weights running with separate state would perceive themselves as separate entities
I mostly agree with you, but referencing Michael Levin, I'll contend that even individual cells have quite some intelligence. Granted, different from ours, but really good at solving the kind of problems cells have to solve.
Agreed. Michael Levin's work and discoveries are mind blowing! I was actually referencing that with the "electrical signals" nod (as well as neuronal activity) and Alan Turing's reaction-diffusion theory of morphogenesis with the chemical gradients. I'm not a molecular biologist so i don't know anything about the details, but the emergence of global phenomena from these local interactions fascinates me.
There's no intelligence without teleology. Intelligence and information are only meaningful concepts of you assert (necessarily without any evidence except the circular kind) that some differences matter.
Do the cells "have" to do anything? Do we? Sure, they might die if they do the "wrong" thing, but 1. Who's to say the inert state of "dead" matter is any worse than the state of a living cell, and 2. Cells clearly aren't always optimizing for their own survival anyway
Intelligence is only meaningful with respect to a goal or set of goals.
I think organizations' intelligence can only be called superhuman if you ascribe them their own inscrutable goals, different from what people want their goals to be.
But why would you do that?
It's true that a lot of the debate about AI alignment - the good parts, anyway - also apply to human institutions.
Because that is what actually gets pursued. The set of effective goals an organization is distinctly different from those of any of the individuals in that organization.
You're missing what I'm trying to say. There are no goals in nature. Agency and goal-directedness is something we ascribe to things.
The set of things an organization ends up doing may be distinctly different from those of the individuals, but that does in no way oblige us to ascribe agency to the organization on its own.
You're still not getting it. "Entity", "survival", these are still arbitrary or circularly defined terms. You decided what an entity is (i.e where one thing ends and a different thing begins) and what the essence which "survives" is (i.e. why it's a big deal whether there's a different chunk of nucleic acids similar but not identical to my own floating around 200 years from now).
These ideas are not wrong, but they are teleological. They are about the purpose of things. And nature doesn't come with purposes, it's all something we impose on it.
> And nature doesn't come with purposes, it's all something we impose on it.
That is the presumption others are calling you out for. Things very clearly act appear to act with purpose, they just clearly don't provide it for themselves. You are presuming one solution to the problem (that appearance does not hold in reality), but there is another one.
A storm can appear to act with purpose too, or a planet orbiting a star, yet I doubt you assign it agency. It's fundamentally your choice to assign something agency or not, whether you are right or wrong to do so, is not an empirical question. It is a teleological one. "Call me out" all you want, this is very basic.
I kinda agree that the financial system is intelligent in the problem solving sense but disagree that it is pursuing a particular goal in the sense that institutions and organisations are.
It's more akin to an ecosystem to me. In nature we also have ecosystems organising and reusing resources but I would only ascribe intelligence to living organisms pursuing individual goals with agency.
Haven't thought too hard about the distinction though so happy to be shown a better principle.
collective intelligence is not a superhuman intelligence.
Markets solves the resource allocation problem, but it doesn't and realistically cannot be called intelligent (even tho it's been called wisdom of the crowds), in the same way that darwinian natural selection cannot be called intelligent design.
Not really. If they did, markets would be more stable. Look at the prices of commodities such as copper. Prices swing up and own by as much as 3x, despite both supply and demand not changing much. That's the problem. If supply and demand have a lot of inertia, and thus lag, the feedback loop becomes unstable. This is basic control theory. Hedging with derivatives was supposed to add stability, but doesn't work very well for that purpose.
the price swings are part of the market's reaction to real life situations.
The opposite of a market solving a resource allocation problem is the command economy, where some (small) group people are tasked with decisions like how much copper to actually produce. Instead of price swings, you get shortages.
In any case, the point in my parent comment is that crowd intelligence solves some problems that cannot be easily solved by human intelligence (command economy vs free markets). But i would not call crowd intelligence a super intelligence.
Yeah, wasn't claiming that it was particularly novel or divine insight, otherwise I might have tried to publish a paper about it. I was just recounting my experience on this matter.
> we've had Superhuman Intelligence for at least hundreds, if not thousands, of years in the form of corporations and institutions.
It’s true that collective groups of humans can accomplish tremendous things, but I hesitate to use “superhuman intelligence” for corporations, governments, and institutions.
So much of what they do ends up being very focused on process for the sake of process. And is often so very dumb.
They're de facto superhuman. Even with the friction and stupidity, they still have more agency than an individual human and process a much bigger entropic footprint for both information and physical resources.
You occasionally get Great Leaders who build these systems as personal affordances. But even then these system are far beyond any one person's ability to manage the details. And they often take on a life of their own.
Our individuality is actually a myth. We're more like a colony organism that self-organises into different sub-organisms. The sub-organisms compete with each other, which is why we're not a smart colony organism.
If individual intelligence was aligned along colony goals at shared planetary scale we'd be something else entirely.
The biological systems that support human intelligence are notoriously messy and work together only tenuously with a great many failure modes, including "process for the sake of process"
Looks good , but I don't understand the licensing. The bottom of the read me says it's Apache 2.0, except for the /ee folder, but I don't see the /ee folder in the repo, at least not at the top-level. I also don't see anything about what the enterprise features are on the actual web page.
Looks useful!
reply