
Loop. Swoop. Pull.
As I watch a series of videos capturing tiny robotic pincers deftly tying tiny surgical knots, the phrase typically associated with children learning to tie their shoes comes to mind. But the smoothness, while mesmerizing, isn’t the major feat being shown—it’s that the robot is performing the procedure autonomously, without a human toggling a joystick to control it.
“Current robotic surgeries are all teleoperated—there’s not a single autonomous surgical system out there,” Ji Woong (Brian) Kim, PhD, a robotics researcher at Stanford University, told Inside Precision Medicine. “There has been some research that addresses autonomous surgery, but they’re all using preprogrammed algorithms. For a robot, such as one that welds in a factory, the exact path it will take is predetermined. This is the first time we actually deployed an AI system that can learn from human surgical demonstrations.”
New research from Kim and colleagues describes a learning-based system developed for a surgical robot trained on videos of gall bladder removals (cholecystectomy), a procedure performed about 750,000 times annually in the United States, that autonomously executed the procedure. The system, called Surgical Robot Transformer-Hierarchy (SRT-H), was applied to one of Intuitive Surgical’s Da Vinci robots and is capable of adapting to individual anatomical features in real-time, making decisions on the fly, and self-correcting when things don’t go as expected. Published in Science Robotics, the article is a significant milestone toward clinical deployment of autonomous surgical systems, which offer the potential to improve outcomes, reduce costs, and expand access to high-quality care.
Joystick wizard
Precise and safe surgeries require the navigation of complex morphologies that vary between patients and from second to second as our organs move about. These living, unpredictable settings have created such difficulty for designing autonomous surgical robots that almost all procedures are teleoperated—controlled remotely by a human operator.
The closest that researchers have gotten to truly autonomous surgical robots has mostly been model-based, which are essentially robots that can do one thing: execute a task. Model-based autonomous robots have been successful in simulations and proxy settings, performing tabletop tasks (e.g., peg transfer, needle pickup, and deformable object manipulation). But for performing contact-rich manipulation tasks, such as suture knot tying, researchers have turned to learning-based systems.
In 2022, researchers from the lab of Axel Krieger, PhD, which is where Kim’s work was done, got within a hair of developing an autonomous surgical robot. Krieger and colleagues showed the Smart Tissue Autonomous Robot, STAR, performing the first autonomous robotic surgery on a live animal—a laparoscopic surgery on a pig. However, that robot required specially marked tissue, operated in a highly controlled environment, and followed a rigid, predetermined surgical plan.
To make the leap to complete autonomy, Kim and Krieger trained a model by integrating visual data and the movements of a surgeon on a joystick to use with Intuitive Surgical’s Da Vinci Robotic Surgical System.
“The surgeon will sit on the Da Vinci robot; there’s a teleoperation console where they can control the robot like a joystick, and they can look into the endoscope put into the human stomach,” said Kim. “We record the images and motion data, such as the joint angles of the joystick, and then we train an AI model, similar to the ChatGPT Transformer, to map images into robot actions coming from the joysticks that were basically imitating the surgeon. It’s a new paradigm.”
Kim explained that the algorithm worked just like ChatGPT does, using keystrokes to predict a sentence, but instead using images to predict robotic motion. To train SRT-H, a couple of researchers learned how the chosen surgical operation, the cholecystectomy, is performed on pig cadavers.
“We trained a student to learn the surgery from a surgical resident and basically learn how to do it on the pig,” said Kim. “We collected the demonstration data from the student to train our model, and then we deployed it on eight new, completely unseen pig organs, such as pig gallbladders; SRT-H was able to perform the actual surgery without any human intervention.”
The cholecystectomy was chosen because the operation is fairly common and involves some difficult steps that can be applied to many other procedures. The trickiest part of the surgery requires using two small robotic “arms” to work together to apply several clamps onto tubular structures and open up a tightly packed, tiny space for scissors to enter and excise the gallbladder. Not only is the area cramped, but part of the difficulty lies in there being several tubular structures, including an artery, that can be easily mistaken for the targets, which would lead to a failed procedure.

It took just 34 operations by the students to train SRT-H to perform eight successful operations on its own. To shake things up a bit and complicate things for SRT-H, Kim and colleagues added a substance similar to food coloring to simulate internal bleeding, which was something SRT-H had never seen before, and it still worked.
No HAL in the OR anytime soon
Before SRT-H enters operating rooms in a hospital near you, there is still a significant amount of work to be done. There still is a lot of preclinical work to be done, notably in living animals, which is a far more complicated scenario than the lifeless, still cadaver. In a live animal, where there is a lot more motion—the animals breathe, there can be blood, and there is also the potential variety of morphology that was never encountered in training, possibly requiring retraining of SRT-H on live animals.
SRT-H will also need to step up its game. In the Science Robotics article, there was no clear-cut “winner” when comparing SRT-H to a surgeon. The results show that the surgeon completed all tasks faster than SRT-H, but SRT-H navigated with shorter trajectory length and less mean jerk compared with the surgeon, generating smoother, shorter trajectories.
If and when SRT-H becomes an approved medical device, while it will be capable of performing fully autonomously, the surgical robots will not be left in rooms by themselves to perform operations unattended. Rather, the supervising surgeons will be working with the robot, even overriding it to drive learning and, ultimately, improved execution. Also, the robot can learn through feedback through language—that is, specific commands can be dictated or typed to the robot.
“You override what it’s trying to do, or you can give more specific feedback, directional controls, like ‘move the left arm to the left,’” said Kim. “That’s something that you can listen to on the fly. We wanted this system design flexibility because intervention may be needed at times. This language feedback is how the adaptation will happen in these sorts of novel scenarios. But if the scenario is completely novel, probably it’s going to be really hard.”
Kim stated that, because SRT-H includes a language component, adding a translation step for using a different language as input is a relatively simple task.



