Grand Challenges of Robotics

Last Update: 2020-08-13

And I started asking, “What are the important problems of your field?” And after a week or so, “What important problems are you working on?” And after some more time, I came in one day and said, “If what you are doing is not important, and if you don’t think it is going to lead to something important, why are you (at Bell Labs) working on it?” I wasn’t welcomed after that; I had to find somebody else to eat with! — Richard Hamming, You and your research

What grand challenges do we need to solve to push robotics (and AI) forward? When I thought hard about this question, I realized that I didn’t have a good answer. I’m actually not sure why I never asked myself this question before. Perhaps I’d converged to the “stationary distribution” of traversing only “relevant” research questions. Or maybe it’s a function of the sheer diversity in the different fields of inquiry that robotics supports. This coupled with the fast pace of the field makes it hard to identify key challenges and track progress. Nonetheless, I believe this is an important question to reflect on, not just for personal gratification, but more importantly, to understand how research contributions in one field can ripple across the space of research and affect questions in another field.

I expect this to be a living (and evolving) document as I distill information to reach a clearer understanding. For now, I’m bootstrapping most of the content from [1] and also using the paper as a scaffolding for organization. However, I expect the structure and content of this post to evolve as I read, synthesize, and figure out potentially better ways to organize the information.

While providing links to references + further reading, I’ve made the conscious decision of linking to sources with a high “fan-out”, to enable an easy launching point to read about further research.

I’d appreciate suggestions on content to add, general feedback, and comments!

---

Table of Contents

AI for Robotics

Biohybrid and bioinspired robots

Brain-computer interfaces

Robot Swarms

Navigation and Exploration

New materials and fabrication schemes

Power and energy

Medical robotics

Social interaction

Ethics and security

General References

---

[a] AI for Robotics

Goal: learning how to learn, combining advanced pattern recognition and model-based reasoning, and developing intelligence with common sense.

Challenges:

• go beyond statistical correlation and begin to reason about system dynamics + causation
• meta-learning with limited data
• learn on the fly, adapting to dynamic + uncertain environments
• systems that know their own limitations and know how to seek help
• learn across heterogeneous tasks and domains
• systems that can deeply comprehend and synthesize information across complex texts and narratives
• perform deep moral and social reasoning about real-world problems

References + Readings

[a1] To Build Truly Intelligent Machines, Teach Them Cause and Effect

[a2] The Book of Why

[a3] Meta-Learning: Learning to Learn Fast

---

[b] Biohybrid and bioinspired robots

Goal: to translate fundamental biological principles into engineering design rules or integrate living components into synthetic structures to create robots that perform like natural systems.

Challenges:

• battery to match metabolic conversion
• muscle-like actuators
• self-healing material that manufactures itself
• autonomy in any environment
• human-like perception
• computation + reasoning
• actuation and energy (compared with performance as that seen in animals)
• electromagnetic motors are adequate actuators for large robots but inefficient at small scales (or in soft systems)
• artificial muscles, currently made of shape memory materials and electroactive polymers, lack robustness, efficiency, and energy and power density
• effectively integrating components to perform system-level behavior
• matching the dexterity of human hands
• framework for terradynamics is required to understand how to effectively interact with the ground (similar to first principle models such as the Navier Stokes equation for fluid dynamics)

References + Readings

[b1] Wyss Institute

[b2] Shadow Hand

---

[c] Brain-computer interfaces

Goal: the seamless control of peripheral neuroprostheses, functional electric stimulation devices, and exoskeletons.

Challenges:

• sensing and data acquisition because current modalities are expensive and cumbersome
• data processing and dealing with artifacts of non-cerebral origin (especially for wearable approaches)
• cortex folding differs between individuals, as do relevant functional maps
• sensor locations may differ across different recording sessions, and brain dynamics can be intrinsically non-stationary
• current methods involve extended periods of training, calibration, learning, and adaptation, thus prohibitive for general use
• unclear whether BCI will always outperform simpler techniques (such as eye-tracking or muscle-based devices)
• dealing with tasks /w high degrees of freedom. current multiclass BCI classification generalizes poorly across individuals + tasks

The future:

• implantable sensing with new microfabrication, packaging, and flexible electronics, combined /w ultralow-power local processing + wireless data paths. untethered implants.

References + Readings

[c1] Neuralink

[c2] CTRL Labs

[c3] OpenBCI

---

[d] Robot swarms

Goal: to allow simpler, less expensive, modular units to be reconfigured into a team depending on the task that needs to be performed while being as effective as a larger, task-specific, monolithic robot.

Challenges:

• develop stochastic models for predicting collective behavior in large scale swarms (since interactions between individuals grow combinatorially)

  • we lack models of flock/herd-like groups that elude enumeration in small scale groups yet do not justify ensemble-averaged models seen in large scale swarms

• no systematic approach to designing multidimensional perception-action-communication feedback loops across large groups

• lack of tools to create teams that can be responsive to human commands can adapt to changing conditions are robust to disturbances, resilient to adversarial, disruptive changes caused by large scale failures, or damage to swarm infrastructure.

---

[e] Navigation and exploration

Goal: to navigate an explore extreme environments that are not only unmapped but also poorly understood, with abilities to adapt, to learn, and to recover and handle failures.

Challenges:

• operating in environments that are not only unmapped, but their very nature is not well understood:

• tunnels or mines:

  • rough terrain, narrow passages, degraded perception

• nuclear decommissioning:

  • radiation and restricted access

• infrastructure construction:

  • resist chemicals and materials used in the construction process
  • resist dirt, dust, and large impact forces

• undersea robots

  • radio does not penetrate. usual forms of communication + navigation disappear
  • untethered; hence must be autonomous

• space robots

  • long latency + low bandwidth reduces productivity and puts the survival of the robot at risk

• how to learn, forget, and associate memories of scene content both qualitatively and semantically (similar to human perception)

• surpass purely geometric maps to have a semantic understanding of scenes

• evolve through online/lifelong learning

• handle failures and learning to adapt + recover

• develop abilities to make and recognize new discoveries

• complex self-monitoring, self-reconfiguration, and repair such that there’s no single point of complete failure

The future:

• SLAM will go beyond current rigid + static assumptions, to effectively deal with time-varying, dynamic environments /w deformable objects

References + Readings

[e1] Probabilistic Robotics

[e2] DARPA SubT Challenge (Tunnels + Mines)

[e3] CMU RadPiper (1), (2) (Nuclear Decommissioning)

[e4] robosub (Underwater Robotics)

[e5] NASA JPL/Caltech (Space Robotics)

---

[f] New materials and fabrication schemes

Goal: to develop a new generation of robots that are multifunctional, power-efficient, compliant, and autonomous in ways akin to biological organisms.

Examples include artificial muscles, compliant materials for soft robotics, and emerging advanced manufacturing and assembly strategies.

Challenges:

• improved portable energy storage and harvesting
• designing new materials with tunable properties
• fabrication and assembly is typically a serial process that is slow and difficult to scale to very large or very small scales.

The future:

• new fabrication strategies that embody these functional materials, including the robot building and repairing itself.

• materials that integrate molecular machines [f1], or other force-generating molecules or biological motor proteins, into hierarchical materials.

• the convergence of additive and subtractive methods, with emerging technologies involving 2D to 3D transformations to generate new architectures that can simplify the need for specialized hardware and enhance the robot’s function.

• multiphoton lithography [f2] that can create features and structures over nine orders of magnitude in size.

• materials that are able to adapt and heal over time

References + Readings

[f1] There’s Plenty of Room at the Bottom

[f2] Multiphoton Lithography

[f3] Stuff Matters (I’ve written a short review of the book here!)

---

[g] Power and energy

Goal: create new power sources, battery technologies, and energy-harvesting schemes for the long-lasting operation of mobile robots.

Challenges:

• irreversible phase transitions of active materials
• unstable electrode / electrolyte solution interfaces
• liquid electrolytes often suffer from electrochemical + thermal instabilities and low ion selectivity due to high ion conductivity and good surface wetting properties
• mobile robots lack a standard fuel source, storage, and distribution system

The future:

• battery systems based on solid electrolytes (advantages due to their safety, excellent stability, long cycle lives, and low cost)

• flow-based lithium-ion, lithium-sulfur, lithium-organic batteries

• radioisotope power systems (already used in space exploration)

• extracting energy from surroundings (solar light, vibration, and mechanical movement)

• electromagnetic + electrostatic generators, piezoelectric nanogenerators, and triboelectric nanogenerators

References + Readings

[g1] The complete guide to the battery revolution, Quartz Magazine

---

[h] Medical robotics

Goal: increasing levels of autonomy but with due consideration of legal, ethical, and technical challenges, as well as micro-robotics tackling real demands in medicine.

Challenges:

• move towards systems that exhibit increasingly higher degrees of autonomy
• creation of fully implantable robots that replace, restore, or enhance physiological processes
• realization of micro + nanorobotic devices of clinical relevance
• enable one surgeon to supervise a set of robots that can perform routine procedure steps autonomously, and only call on the surgeon to take control during critical, patient-specific steps
• anticipate, detect, and respond to all possible failure modes
• biocompatibility, reliability, adaptability, security, and power for implanted bionic systems

The future:

• micro and nanorobotics

References + Readings

[h1] Intuitive Surgical

---

[i] Social interaction

Goal: to understand human social dynamics and moral norms and that can be truly integrated with our social life showing empathy and natural social behaviors.

Challenges:

• we have very few comprehensive, quantitative analyses of human social responses

• modeling social dynamics

• learning social + moral norms

• building a robotic theory of mind

• significant perceptual demands:

  • social cues, such as gaze direction, facial expressions, or vocal intonation, are often extremely detailed, rapid, nuanced signals that are embedded within other activity

• we still lack systems that can operate under the diverse natural conditions and real-world time constraints that social interactions demand

• social signals are context and culture-dependent

• tools required to model the expectation of people around the robot and expanding the robot’s understanding of the consequences of its own actions

• build and maintain complex models of people (knowledge, beliefs, goals, desires, and emotions)

• need models that allow robots to have rich, usable models of human mental states

• most current robots have been designed for short term interaction, but human interactions span months, years, and even decades

The future:

• elements from multiple input modalities combined to form predictive models of social intentions
• integration of models of episodic memory, hierarchical models of tasks and goals, robust models of emotion to create detailed cognitive models
• generate interaction content rather than relying on prescripted components, and adjust responses based on time scales of interaction

References + Readings

[i1] HARP Lab, CMU

[i2] InterACT Lab, UC Berkeley

---

[j] Ethics and security

Goal: to foster responsible innovation in robotics.

Challenges:

• excessive reliance on robotics + AI may lead to the delegation of sensitive tasks to autonomous systems that should remain partly subject to human supervision
• robotics + AI may remove responsibility from people whenever an autonomous system could be blamed for a failure
• unemployment is an ethical problem since robotics + AI could change the workforce structure, cause a shift in the skills base, and potentially facilitate a complete de-skilling of the workforce
• AI may erode human freedom, due to unplanned and unwelcome changes in human behaviors to accommodate the routines that make automation work and people’s lives easier.
• robotics + AI can refine strategies and launch more aggressive counteroperations (security), which may snowball into an intensification of attacks and responses, which could threaten the key infrastructure of our society.

References + Readings

[j1] AI Safety Needs Social Scientists

[j2] Concrete Problems in AI Safety

[j3] Future of Humanity Institute

[j4] Center for Human Compatible AI

[j5] Future of Life Institute

[j6] Humans Need Not Apply

---

General References

[1] The Grand Challenges of Robotics, Yang et al.

[2] robonation