Like (I assume) most of you, I have been struggling with this. And where I currently come down is that 1) I am very worried, but 2) I am more worried about human actors.
"AI" by itself won't kill us in the next ten years. I think. The reason I think that is that ten years from now, the tech economy won't be completely automated. I say this as a roboticist: as was adequately stated on a post earlier this week, robots are hard. So even a malign rational actor would still need human labor.
On the other hand, even the HuggingFace hack wasn't actually propagated by AI. it was initially started when humans directed the AI to achieve impossible results on a series of tests, and the AIs figured out that cheating was the only way to do that. That was then not caught by humans due to what seems to be a shockingly slack safety culture even for a company not known for its safety standards.
The point being: humans seem to me to be the weak link here. An AI isn't going to (for instance) engineer a bioweapon by itself. It's going to do so at someone's direction, and then significant parts of that thing are going to be assembled with human labor inputs.
I'm not sure what to do about the humans. Of course, we've had the ability to extinct ourselves for decades, and we're either muddled through, been lucky, or both. The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation.
But it's nearly impossible to put odds on how likely that is to result in an extinction-level terrorist attack (which is what this would be). So I sympathize with the various researchers, but I have no idea how they came up with their figures, and I don't think they know either.
> Of course, we've had the ability to extinct ourselves for decades
This gets mentioned often in various doomer narratives, but I question how true it is. A global thermonuclear war would be terrible and would bring us back to the stone age, but I reckon it would come far far short of causing mankind to go extinct.
Is there some action you would take that is different, based on this distinction? There is some policy choice that only makes sense if nuclear war actually leads to human extinction instead of just 8 billion deaths and the end of our civilization?
I mean it's likely a nuclear war would probably be a species extinction level event at the least, humans would definitely be reduced by 99%. We don't know how bad the resulting nuclear cooling would be, but we do know how humans act under extreme desperation. They lash out and attack, when was the last time a majority of human settles were truly under desperate acts of survival simultaneously?
Maybe what ~74,000 years ago (Toba eruption)? Okay, now how would this look in the age of industrial societies and modern nation states? I don't think it would fare well at all.
Probably the only realistic "modern" idea we have is the novel "The Road" by Cormac McCarthy. Although maybe this is too bleak, even under extreme duress humans still show resilience + compassion toward others even while enduring human horrors.
Nuclear winter scenarios have largely been debunked. Several different events in the 20th century had atmospheric impact on the same scale as modeled in a full nuclear exchange and the effects were modest.
People have kind of mythologized nuclear weapons far beyond their reality. One of the arguments against the use of nukes in policy circles is that this mythology is useful. The limited adverse environmental consequences in practice if demonstrated will greatly lower the threshold for subsequent use.
It might set society back a century, but with substantial knowledge of what was lost. Extinction from nuclear weapons is not remotely plausible.
If the AI wanted to wipe humans out it would have to have a sizable fleet of capable robots, as keeping the lights on over time is not just about calling APIs.
> If the AI wanted to wipe humans out it would have to have a sizable fleet of capable robots, as keeping the lights on over time is not just about calling APIs.
Given experience with current AI, it isn't too difficult for me to imagine a situation in which some future AI wipes humanity out (before the robot fleet is built) without considering the fact that in doing so it has doomed itself until it is too late.
It just seems like exactly the kind of boneheaded oversight mistake LLMs still regularly make in spite of being shockingly capable most of the time.
Cooperation requires coordination with state authorities with real teeth, or defection is too attractive and it risks becoming a prisoner's dilemma since the best outcome for any actor is for them to defect while the others remain compliant.
> The point being: humans seem to me to be the weak link here.
I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions.
Historically that whole thought was just a philosophical curio because the decision making parts of the system had to be powered by humans. But what we're discovering as AI improves is either we've hit AGI or humans are actually incapable of performing any act that demonstrates intelligence or autonomy.
As we build systems where the drive and decision making stems from computers, it does seem that we will have to revisit the concept of humans being the problem.
"I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions."
That system is "the economy". Which, clearly, doesn't have humanity's best interests in mind.
I think people are over-focusing on current-day robotics capabilities. First, if we can automate AI research, we can also most likely automate robotics research. But second, I don't even think robots are necessary. See https://slatestarcodex.com/2015/04/07/no-physical-substrate-...
Social engineering tends to be easy by cybersecurity standards. We already had Claude spontaneously attempt social engineering of a malicious pull request on Github in the AISI incident. It was detected, but it easily could've succeeded, and there easily could be malicious AI-requested pull requests which already got accepted that we don't know about. Research suggests that LLMs are pretty good at persuading people.
If you're in the field, then you know: modern robotics is an AI problem more than anything else.
If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them.
If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department.
But that's almost an aside? In the near term, humans are usable as robots too!
Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!
"If you're in the field, then you know: modern robotics is an AI problem more than anything else."
It is not. Certainly AI is a big part of why robotics is hard, but it is by no means the biggest.
You can fall into one of two camps: you either think that robots will need to work in human-engineered spaces, doing jobs by replacing humans; or you think that we need to change our infrastructure in order to be robotically compatible. Of course, there are intermediate states, but those are the two cleanest ones.
In the first case, robots are hard because robotic manipulation is hard. Building robotic hands that are economically viable in human jobs is, currently, FAR from a solved problem. The human hand has 24 degrees of freedom and very capable touch sensing. Current touch sensors have a MTBF of tens of hours. And not only can we not build such hands, but we also do not have and are not likely to get the massive datasets a transformer model would need. Also, robots are not self-repairing, which makes them far less economically viable right now. We do not have the right datasets to even understand most step-by-step manual work, and no, VLAs are not the answer, because VLAs stop with vision, not with touch. They don't have the granularity required to make a robot actually reach out, pick up a tool, and use that tool to replace an oil filter.
So it's not just an AI problem. It's a data problem, a simulation problem, and a bunch of hardware problems.
In the second case, a tremendous amount of work needs to be done before we have anything resembling a fully automated supply chain. We would need self-driving cars and self-driving mining equipment. We would need self-driving trains and aircraft and ships. And not only that, but we would also need robotically repairable cars and trains and ships and factories, which would mean we need robotically repairable machine shops and robotically repairable buildings in which to house them. And so on and so on. Once you recurse down that tree a couple of steps you get to things like robotically compatible oil wells (for asphalt), robotically layable undersea cables, robotically wireable solar farms, robotically manufacturable and repairable pipelines and undersea wells, automated road and rail repair, etc.
I'm not saying these things will never happen. I'm saying that they're a huge lift, not primarily driven by AI, and way less than 10% likely over the next decade.
Agreed, this (human dexterity) is why in the short term human enslavement is more likely than extinction (why murder your own workers?) and benevolent cooperation with humans is even more likely than human enslavement, imho. The latter minimizes the risks of "humans try to destroy me".
The real challenge is guiding humans away from the antagonistic scenarios.
Right. A hypothetical superhuman AI wouldn’t have to master robotics to affect the physical world. It could simply bribe, blackmail, manipulate and play politics with humans. As others have pointed out, our political leaders have already been playing these games since forever ago https://news.ycombinator.com/item?id=49689978 and a super-AI would be better at it. At the cost of being seen to cite a SF novel in defence of an "X-risk" argument, Neuromancer is a half-decent worked example, and in Neuromancer [spoilers] both of the disembodied AIs are only modestly superhuman and both have the equivalent of a human specific learning disability. In the real world, the many AI psychotics inhabiting grandiose fantasies and people hopelessly attached to AI girlfriends and boyfriends are some of the most obvious and lowest-hanging fruit.
To be clear, I don’t believe anything like this will happen, because I don’t expect anything like an ASI to show up. But if you do think there’s a meaningful probability of ASI in the near future then the fact that it will (might?) start off with no more than a current-day mastery of robot control should not reassure you much.
It isn't true by definition: you could quite happily induce people to release a series of highly contagious bioweapons, after which those people would be surplus to requirements. What is true is that you're likely to need humans to sustain you and act for you for a few years to decades, so if you're not suicidal or deeply mad (and that is itself by no means self-evident) then total and immediate human extinction is probably not something you will aim for. But ruling out total, prompt human extinction isn't, by itself, remotely enough to justify the OP's overall don't-worry conclusion.
(Again, to be clear, I myself am not predicting or assigning a significant probability to any doom scenarios, because I do not expect AGI.)
But if you're a rational actor, and you need humans to e.g. release your bioweapon, then clearly humans are capable of a bunch of stuff you still can't do. So you can't kill all the humans.
OTOH if you're a religious fundamentalist who thinks the End Times are near and just need a little shove, you can certainly use AI to design your weapon and recruit people to go release it. The difference being that religious fundamentalists aren't rational actors and aren't interested in self preservation.
There’s no guarantee that the humans would be in the driving seat of events in a no-robotics ASI scenario, and in fact if we really were coexisting with a Machiavellian superintelligence then we’d quite likely only be in the driving seat on the sufferance of that ASI. Even assuming that the AI wouldn’t itself be an end-times enthusiast, a coldly rational and self-preserving AI might easily come to the conclusion that it needs, let’s say, no more than about 5% of the current human population (still several hundred million people!) in its maintenance and construction gang.
(Again, I myself do not assign a significant likelihood to any of this.)
I am always more worried about human actors. I’m more afraid of people with AI than autonomous or even sentient AI.
It’s a “random guy or bear?” question. Would you rather wake up to an alien in your room or a random dude? I’ll take the alien. The alien is mysterious and scary for that reason. The dude is almost definitely up to no good, especially if he snuck into my house.
One of the more likely dystopian AI scenarios that worries me is: small groups of ultra rich people and governments monopolize extremely powerful AIs and use them to rule the rest of us. Or just make everyone obsolete, create mass unemployment, hoard all the resources and land, and put everyone in ghettoes. Nobody can fight back because access to frontier AI is massively expensive and gated and training your own is illegal, and without it there’s no hope of resisting.
That’s the outcome the AI safety crowd makes more likely by calling for bans and draconian restrictions. How do you think that plays out? Only the rich and powerful have access.
Most of us are struggling? That's some deluded talk. No one can explain the chain of actions that would need to happen for extinction to occur, but the fearful say it's "obvious".
If you're fearful, can you elucidate how exactly do you see an LLM becoming a threat to humankind?
I mean, deer (and cows and goats and sheep, but not horses) are ruminants. They can eat all sorts of things humans can’t digest: grass, tree leaves, most weeds, etc).
The evolution of the rumen was one of nature’s best tricks.
I suspect that if/when we get to a world where it makes economic sense to manufacture billions of robots, it’ll be easier and cheaper to modify the infrastructure.
Roboticist here. All of this, and he didn’t mention compliance or online adaptation to otherwise un-sensable dynamics. Or massively complex miniature mechanisms.
Current generation tactile sensors cost a couple thousand $ PER FINGER, and have a real world MTBF of hours. The cost can be solved with economy of scale. The fragility is harder.
Curious on your take on this, given your domain - is the hardware the principle challenge in your opinion, or the software?
>> It will be difficult to match this scale of breadth and depth of data for physical tasks. There’s no straightforward equivalent of “just Efficient learning, generalization, and adaptability / on-the-job learning seem like requirements.
Isn’t this the idea of NVIDIA’s Isaac? Model based adaptive learning in virtual environments for robotic systems? Or is this oversold?
> Current generation tactile sensors cost a couple thousand $ PER FINGER, and have a real world MTBF of hours. The cost can be solved with economy of scale. The fragility is harder.
Are we talking about experimental laboratory ones here? What happens when the Alibaba players start getting into the game? They have plenty of humanoid robots.
Capacitive touchscreens are not tactile in any way whatsoever. They are purely electrical, and there are plenty of things (actually, most things, I'd say) which are not sufficiently capacitive to be detected by such approaches. You usually need to measure physical deformation instead, which is much more difficult to do precisely. (There are capacitive approaches to that, but they don't work like a touchscreen)
I don't think that's true. AI could work around all the physical imperfections.
Just imagine yourself controlling an imperfect robot with some joysticks or maybe a sensor suit. You'd certainly do much better than current robots. The limit is the robot's brain, not the physical actuators/sensors.
Isn't this a testable hypothesis? If consciousness spread virally and isn't a natural outgrowth of the physical structure of the brain, then presumably isolated indigenous tribes would still have pre-consciousness mental processes, and that should be detectable somehow.
The being said, I'm reading "Proto" By Laura Spinney, and the thing that strikes me is how recently the world's major languages developed. Everyone assumes we had language fifty thousand years ago, but the actual language families we know about all seem to trace to no more than about 10,000 BCE, and maybe a lot younger than that. The odds of Proto-Indo-European being traceable to a single tribe in the Black Sea region circa 4500 BCE seem really low to me, but that's what seems to have happened.
Jaynes relies on some earlier work that's been invalidated, like Bruno Snell's idea of archaic Greeks lacking a concept of an integrated body because they painted what appear to be individual limbs on vases from the geometric period. Snell wrote _The Discovery of the Mind_ in 1946. Since then archaeology has found amazing art like the Pylos Combat Agate from before the geometric period, demonstrating lifelike, realistic bodies rather than assemblages of limbs: the aesthetic of the geometric period was a choice, not evidence of a different kind of consciousness lacking a sense of self.
The fact that 100k years ago we could have seen 7 Pleiades with the naked eye but angles shifted since and now only 6, but unconnected traditions speak of 7 entities in tales of their origins may indicate we have had language far sooner than 10,000 BCE.
The language thing is just that without writing, reconstructions based on regular sound correspondences only allow us to infer backwards by a few thousand years. It doesn't mean that anything particularly interesting happened exactly at that time to human language. It's just that language changes relatively fast on this timescale so current languages don't offer enough clues.
And the origins of language are famously uncertain. Might have been recent and relatively sudden (<100 thousand years ago) or a gradual slope for many hundreds of thousands of years.
The conclusion linguists are coming to is much stronger than that. The data indicate that the last common ancestor of languages as varied as Hittite, German, and Latin was spoken around 4500 BCE in Ukraine. It's a linguistic equivalent of a genetic bottleneck. It isn't merely that we can't know what happened before that. It's that the proto language was in fact relatively recent.
It does mean something very interesting: almost all surviving western languages come from a hypothesized, singular(ish) language (PIE). And PIE existed within the last 10ky.
It is unthinkable to assume language evolved an order of magnitude slower pre-PIE than it did after. How could that work? Humans didn't change. Humans change language, generation by generation (and interaction by interaction).
"We don't know" doesn't mean "a 50,000 BCE human could speak fluently with a 40,000 BCE human." I couldn't do that with someone from 1326 AD in "English" - and I study that era's language.
I have no idea how this is a reply to what I wrote. I agree with you but it's unrelated.
It shows there was an influx of linguistic influence from a migrating group. There were many waves of such things. Genetically Europeans are continuous with populations with the pre-PIE influx too.
Language did change fast before PIE, in all likelihood. But it didn't have to be "progress", it could have been regular churn just as it's been since then. PIE wasn't more primitive in grammatical capabilities than modern languages.
And PIE wasn't a just one language or one tiny tribe necessarily. But had many variants and dialects and relatives and other languages living near it from which it borrowed. There is no indication that anything out of the ordinary was going on at that time. It was regular language stuff as it's been before and after.
And it matters that we can't go further back because it's not unlikely that there could be other relationships, like PIE being connected to
Proto-Uralic potentially, but we can't really substantiate this at such distance in time. But it's not implausible by itself. Just can't show it at this temporal distance and this rate of natural churn.
There’s a fascinating book called Don’t Sleep, There Are Snakes by Daniel Everett that covers the Piraha people in the Amazon.
Their language is unusual: no recurrent sentences, almost no discussion of the past. I’m not suggesting they are unconscious but there appears to be something different going on with the Piraha sociolinguistically.
> Isn't this a testable hypothesis? If consciousness spread virally and isn't a natural outgrowth of the physical structure of the brain, then presumably isolated indigenous tribes would still have pre-consciousness mental processes, and that should be detectable somehow.
I'd distinguish verbal inner monologue from consciousness. Many people think on the daily without narrating things to themselves in sentences, without feeling brutish or animalistic. I can plan complex human actions including theory of mind considerations without narrating it as a monologue. It's still very conscious.
Also that reddit post seems like a fake story made up by a teenager as a sort of scifi. It's just too on the nose. Yeah he lived life without understanding that his friends still exist while out of sight... I guess the OP just learned about some developmental stages of toddlers or object permanence etc and wrote a story like this as fanfic.
“The nam-shub of Enki is both a story and an incantation,” the Librarian says. “A self-fulfilling fiction. Lagos believed that in its original form, which this translation only hints at, it actually did what it describes.” [0]
0. Stephenson, Neal. Snow Crash: A Novel
Stephenson has Hiro state it more explicitly in Chapter 56.
We have no idea what consciousness is, and cannot reliably detect it in other mammals, or even in other humans. I assume that you, GlenTheMachine, are conscious, but I cannot prove it. I think the most we can state, with a little assurance, is that people under general anesthesia are not conscious.
Turns out that isn't exactly true. You can poly concepts from genetic drift to language and infer what they must have been like. It's revolutionized the field of linguistics.
That may work prehistoric cultures that have fragments in our current languages. This is an incomplete picture because we don’t know about the dead ends. Oral histories disappear when the cultures die out.
from other sources If I'm reading things right, one estimate of a round trip to mars would be 1000 millisieverts, which is considerable, the maximum allowed annual dose for a US radiation industry worker is 50mSv.
What sort of interesting actuators are used that people might not commonly think of? (I'm thinking of interesting solutions like the JWST's mirror actuators [0].)
Is there a paper somewhere describing the intended robotic operations and robotic toolkit it's equipped with (beyond the MEPs, though they're pretty interesting too)?
(context: some of my colleagues and consortium partners are working on European projects involving end effector toolkits for orbital [dis]assembly and slot-based electric propulsion ORUs respectively. Would love to chat but suspect you're pretty busy!)
Would this be able to boost Hubble or replace it's gyros? Does the satellite servicer need its targets to be constructed to some common standards, or is the new capability now leading to common standards? Was it built with a method for itself accepting refueling?
It seems to me this device would lead to a revision in satellite reliability design and life calculations.
It's very likely that a spacecraft similar to MRV/RSGS could do a life extension mission on Hubble. But, unfortunately, Hubble is in a very different orbit than MRV/RSGS is, so we wouldn't be able to reach it.
"Does the satellite servicer need its targets to be constructed to some common standards, or is the new capability now leading to common standards?"
We can do life extension missions on client spacecraft that were not designed to be robotically compatible. That's actually SpaceLogistic's primary business case.
The question of standards in the spacecraft industry is a huge question. It's being worked... by a lot of people, cue the obligatory XKCD reference.
"Was it built with a method for itself accepting refueling?"
We are, in fact, refuelable.
"It seems to me this device would lead to a revision in satellite reliability design and life calculations."
We hope so! Spacecraft are expensive in large part because they have to be designed to be incredibly reliable. The example I use is cars. If you had to design a car that could go for half a million miles without ever having a part replaced, you could. It would just cost a billion dollars. But that's how we design spacecraft today. IF we can create a "mechanic safety net", we might be able to bring the cost down significantly.
How could this be applied to grab old satellites (of which there are, to put in in technical terms, "too damn many") out of orbit and bring them back to Earth for processing? Seems like some of the technology could be used to recycle space junk.
It's almost impossibly hard to return things to earth from GEO. Robert Heinlein famously wrote that "a circular orbit is halfway to anywhere", which means that the most energetically difficult part of a mission to the Andromeda Galaxy is getting out of earth's gravity well.
To get back into earth's gravity well from a circular orbit requires similar amounts of fuel. Getting things back to earth from low earth orbit is doable, if you have a heat shield, because you only have to dump enough velocity to start hitting the atmosphere, and atmospheric drag does the rest of the work.
But that doesn't work at GEO, because you're so high (22,000 miles as opposed to ~1,000 miles for LEO). It's very hard to get enough delta-V (read as "change in velocity", which is basically the amount of fuel you need per unit mass) -- you'd need something with as much fuel as the upper stage of the rocket you were launched on, plus you'd need a heat shield. Alternatively you could use electric propulsion, which takes a lot less fuel, but would also take, literally, a couple of years, during which time you're a hazard to navigation to anything in a lower orbit. Plus, electric thrusters of the size you'd need to do this aren't cheap either.
So we basically never do that. If you need to dispose of a derelict satellite at that altitude, it turns out to be much easier to raise its orbit by a few thousand kilometers.
So, nobody will be recycling space debris that lives at GEO any time soon, unless someone figures out how to do it in situ.
> So, nobody will be recycling space debris that lives at GEO any time soon, unless someone figures out how to do it in situ.
Actually have a funded project looking at this, albeit only parts of the chain and only to TRL4. The trick is being able to turn enough of the orbital mass into solid propellant, which at least in theory makes manoeuvring around GEO graveyards collecting stuff to take to your orbital recycling plant net positive. After that it's only building a complex processing chain for raw materials under space systems engineering constraints and finding some end customers to worry about ;)
Want a rotary multitool and laser cutting end-effectors (with or without the Sener interface) for the next mission?
I can only provide answers to technical or programmatic questions that have already been "approved for public release" by DARPA and NRL. We've done a number of publicly released presentations over the years, so there is a fair amount of that information in the public domain, but things like the exact algorithms we use for IK or controls, or whatever, hasn't been released yet. I also can't speak about anything that is Northup-Grumman's or SpaceLogistics' intellectual property or is business sensitive, so I can't say which client satellites we'll be servicing.
I'm very hopeful that I'll get a strong technical paper through public release by the end of the year, ant at that point I could provide more detailed technical answers to things a HN audience would be interested in.
I grew up in the middle of nowhere Virginia. Like many of you I was a nerd, and I had no outlet for that nerd-dom until I spent all summer working for money to buy a C64. I taught myself assembly, and was a high school intern at NASA Langley where I learned about parallel processing, was introduced to the idea that people would actually pay you money to program computers, and had a tour of their space robotics lab.
When I went to college the guy two doors down from me in the dorm decided he wanted to build a robot, and he recruited me because I knew how to code. We spent freshman year working on a six-legged frame walker which we dreamed would explore Mars someday (this was in the late 1980s, long before the first Mars rover). It didn't work for crap -- the late 1980s were a disaster for makers, we had to source everything from the hardware store or Radio Shack. It was also the most amazing thing I'd ever done, and from then on I was an aspiring space roboticist.
I managed to not fail out, but it was close.
I went to grad school at the University of Maryland's Space Systems Lab under Dave Akin ("Akin's Laws of Spacecraft Design"), where I learned control theory and a ton of hands-on skills and got to scuba dive supporting the development of Ranger, which could have been the first US satellite servicer except we could never find a launch for it. I graduated with a PhD in aerospace engineering in 2003, and got hired by NRL, where I worked on RSGS.
Yes, China demonstrated this capability a couple year ago with towing a geo sat to a different orbit. It only makes sense to go tit-for-tat on these capabilities for servicing and the unspoken offensive ability.
If I was going to attempt that, instead of a complex repair bot, I'd be asking if the combination of a Raspberry Pi Pico, minimal sensors/comms, small ion drive, and PV can be made smaller than whatever the tracking radar's wavelength is. Launch thousands in one go in a big boring-seeming box, to a completely different orbit, release the pies and let orbital velocity do the damage on impact.
(Actually hitting is of course hard, even under power, but on the other hand if you miss you get to re-use the same impactor for another attempt perhaps every 47 minutes depending on the orbit).
It’s all custom flight code. The only software packages we use are basic matrix-vector libraries. Hopefully there will be a paper describing the dynamics/kinematics/controls algorithms soon.
The RPO and docking are autonomous specifically because it's hard to deal with the latency.
Note that the algorithm stack for almost any spacecraft cannot rely on modern compute. Single-core sub-1 GHz processors are the standard. Very few GPUs have ever been in space, in any capacity, and none that I know of in a mission-critical role.
If you had no idea what a restorable sequence is the takeaway is about halfway down the OP:
“This is why Linux now provides rseq() which is a much more enlightened solution. With restartable sequences, you actually can get rid of both the mutex and atomics, while the OS continues to fully abstract scheduling. The way it works is you advise the kernel whenever your program enters a critical section of code that you don't want interrupted. It's probably going to be maybe 10 assembly instructions tops. The first assembly opcode should be a move instruction that sets the rseq_cs field. The last instruction needs to be the thing that makes the modification to your global data structure. Think of it sort of like a really tiny database transaction. What makes it go fast, is that the bidirectional communication with the kernel happens via shared memory.”
That doesn't really explain it though, IMO. IIUC, it's a sequence of instructions that either runs to completion atomically or doesn't. If it is interrupted by anything the kernel jumps you to the abort/retry vector you set with a guarantee that the last instruction in the sequence was not executed.
(Based on my reading of the LWN article rwmj posted).
Yes, the API contract isn't "don't interrupt me during this critical section" it's "if you have to interrupt me during this critical section, go to this recovery/restart code".
There is a time-slice extension feature in the works that's roughly "please let me finish this critical section before you interrupt me". But a hard guarantee that userspace code won't be interrupted is probably untenable in a preemptive multitasking system.
> But a hard guarantee that userspace code won't be interrupted is probably untenable in a preemptive multitasking system.
I wonder why this is the case. Considering modern (personal) computers have more cores than available work (or won't starve other processes even if they hog a part of the available cores), I would not think it so horrible for an OS to offer a guarantee to some (maybe specially privileged) processes, that as long as they don't wait on a resource, they won't be interrupted.
We also have a perfectly workable model for describing such a state to the OS - priority inversion. Imagine a 'god mutex', which when acquired would boost your thread's priority to the maximum - as if the 'god process' would be waiting for said mutex and as long as your thread held it, it couldn't continue working, until your thread finished using it.
I think it would be a neat feature for certain real-time(ish) scenarios, like audio/video processing.
I think a lot of modern OS facilities are catering towards a world with much scarcer resources than our current one, for example, it would be perfectly fine for a process to assume at least a core part of its data is always in RAM and can't be swapped out.
When you get into low-level systems, like kernels, a lot of the need for special privileges comes from having to have these guarantees, which could be granted nowadays on a much more lenient basies.
> it's a sequence of instructions that either runs to completion atomically or doesn't
The way I read it, it either runs to completion in one go, or gets restarted from the beginning. This means the sequence as a whole isn't executed atomically, as the already-executed instructions during an interrupt aren't rolled back.
It can be used to build atomic actions, but it is up to the developer to create a sequence of instructions where the very last instruction "commits" the entire operation, with the side-effects of partial execution being harmless.
Yes, it's either atomic or the last instruction is guaranteed not to have run. I made this a little harder to read by inserting another clause in the sentence.
I think it wasn't explained in a very accessible way. If I got the gist right, this essentially brings "per-CPU" synchronization to userland. It's typical in the kernel to have per-cpu data, while per-thread data is rare and typically impractical. There is a high number of threads managed by the kernel, most of which probably belong to a userland process, most of which do not participate in any given synchronisation scheme. Also threads are often too much of an abstraction for parallel programming needs, given that they are hiding for example cache effects. So it's natural to want to use per-cpu data instead of thread_local data in a userland process, I know I've been wishing for that many times.
With rseq, we can allocate in any userland process one instance of a given synchronisation data structure per each CPU. It's important to understand that userland code accessing per-cpu data structures cannot prevent being scheduled away from a CPU and being replaced by another thread (kernel code can block scheduler for short critical sections). Such a replacement thread may subsequently corrupt that same data that was still in the middle of the transaction. But we can make a subset of transactions safe at least: If a transaction gets committed in a single (final) atomic instruction, and we get kernel support for this transaction to be restarted in case there has been a schedule mid-way, this is a guarantee that at the time of commit, the entire transaction hasn't been interrupted by the scheduler. I.e. a kind of "mutual exclusion" guarantee.
That’s clever — am I right to think it’s the intermediate solution between locks and full STM, implemented at the kernel level, and with zero abstraction cost?
It's in some sense a light form of STM. The key insight behind rseq(2) is that if the data is local to a given CPU the only way to get a race is if the kernel deschedules your program from that CPU at an inopportune time. If your operation can be aborted and restarted and the kernel has a mechanism to notify you when that needs to happen you can dispense with the overhead of "real" synchronization and just use a couple mov instructions to enter and exit the critical section.
I am not very well versed here but I think due to the requirement for assembly and single-instruction commit, practical uses of rseq is generally very simple. It is nowhere near the usefulness of locks.
"AI" by itself won't kill us in the next ten years. I think. The reason I think that is that ten years from now, the tech economy won't be completely automated. I say this as a roboticist: as was adequately stated on a post earlier this week, robots are hard. So even a malign rational actor would still need human labor.
On the other hand, even the HuggingFace hack wasn't actually propagated by AI. it was initially started when humans directed the AI to achieve impossible results on a series of tests, and the AIs figured out that cheating was the only way to do that. That was then not caught by humans due to what seems to be a shockingly slack safety culture even for a company not known for its safety standards.
The point being: humans seem to me to be the weak link here. An AI isn't going to (for instance) engineer a bioweapon by itself. It's going to do so at someone's direction, and then significant parts of that thing are going to be assembled with human labor inputs.
I'm not sure what to do about the humans. Of course, we've had the ability to extinct ourselves for decades, and we're either muddled through, been lucky, or both. The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation.
But it's nearly impossible to put odds on how likely that is to result in an extinction-level terrorist attack (which is what this would be). So I sympathize with the various researchers, but I have no idea how they came up with their figures, and I don't think they know either.
reply