Teaching robots to build simulations of themselves (2025)

References

Afzal, A., Katz, D. S., Le Goues, C. & Timperley, C. S. Proc. 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (IEEE, 2021).
Choi, H. et al. On the use of simulation in robotics: ppportunities, challenges, and suggestions for moving forward. Proc. Natl Acad. Sci. USA 118, e1907856118 (2021).
Google Scholar
Liu, C. K. & Negrut, D. The role of physics-based simulators in robotics. Annu. Rev. Control Robot. Auton. Syst. 4, 35–58 (2021).
MATH Google Scholar
Žlajpah, L. Simulation in robotics. Math. Comput. Simul. 79, 879–897 (2008).
MathSciNet MATH Google Scholar
Howard, D. et al. Evolving embodied intelligence from materials to machines. Nat. Mach. Intell. 1, 12–19 (2019).
MATH Google Scholar
Gallup, G. G. Self recognition in primates: a comparative approach to the bidirectional properties of consciousness. Am. Psychol. 32, 329–338 (1977).
MATH Google Scholar
See Also
Researchers simulate an entire fly brain on a laptop. Is a human brain next? - Berkeley News A Countdown to a Digital Simulation of Every Last Neuron in the Human Brain
Gallup, G. G. Jr Self-awareness and the emergence of mind in primates. Am. J. Primatol. 2, 237–248 (1982).
MATH Google Scholar
Cash, T. F. Body Image (Oxford Univ. Press, 2000).
Chiel, H. J. & Beer, R. D. The brain has a body: adaptive behavior emerges from interactions of nervous system, body and environment. Trends Neurosci. 20, 553–557 (1997).
MATH Google Scholar
Agnew, W. et al. Amodal 3D reconstruction for robotic manipulation via stability and connectivity. In Proc. 2020 Conference on Robot Learning (eds Kober, J. et al.) 1498–1508 (2021).
Huang, W. et al. VoxPoser: composable 3D value maps for robotic manipulation with language models. In Proc. 7th Conference on Robot Learning (eds Tan, J. et al.) 540–562 (PMLR, 2023).
Papachristos, C., Khattak, S., Mascarich, F., Dang, T. & Alexis, K. Proc. 2019 International Conference on Unmanned Aircraft Systems (ICUAS) (IEEE, 2019).
Xu, Z., He, Z., Wu, J. & Song, S. Learning 3D dynamic scene representations for robot manipulation. In Proc. 2020 Conference on Robot Learning (eds Kober, J. et al.) 126–142 (PMLR, 2021).
Steels, L. & Spranger, M. The robot in the mirror. Connect. Sci. 20, 337–358 (2008).
MATH Google Scholar
Kwiatkowski, R. & Lipson, H. Task-agnostic self-modeling machines. Sci. Robot. 4, eaau9354 (2019).
MATH Google Scholar
Kwiatkowski, R., Hu, Y., Chen, B. & Lipson, H. On the origins of self-modeling. Preprint at https://arxiv.org/abs/2209.02010 (2022).
Vaughan, R. & Zuluaga, M. Use your illusion: sensorimotor self-simulation allows complex agents to plan with incomplete self-knowledge. In Proc. 9th International Conference on from Animals to Animats: Simulation of Adaptive Behavior (eds Nolfi, S. et al.) 298–309 (Springer, 2006).
Wittmeier, S. et al. Toward anthropomimetic robotics: development, simulation, and control of a musculoskeletal torso. Artif. Life 19, 171–193 (2013).
Google Scholar
Blum, C., Winfield, A. F. T. & Hafner, V. V. Simulation-based internal models for safer robots. Front. Robot. AI https://doi.org/10.3389/frobt.2017.00074 (2018).
Chen, B., Kwiatkowski, R., Vondrick, C. & Lipson, H. Fully body visual self-modeling of robot morphologies. Sci. Robot. 7, eabn1944 (2022).
Google Scholar
Barron, J. T. et al. Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (IEEE, 2021).
Pumarola, A., Corona, E., Pons-Moll, G. & Moreno-Noguer, F. Proc. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2021).
Reiser, C., Peng, S., Liao, Y. & Geiger, A. Proc. IEEE/CVF International Conference on Computer Vision (ICCV) (IEEE, 2021).
Mildenhall, B. et al. NeRF: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 99–106 (2021).
Hu, B., Huang, J., Liu, Y., Tai, Y.-W. & Tang, C.-K. Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2023).
Lazova, V., Guzov, V., Olszewski, K., Tulyakov, S. & Pons-Moll, G. Proc. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (IEEE, 2023).
Xu, C. et al. Proc. IEEE/CVF International Conference on Computer Vision (ICCV) (IEEE, 2023).
Zhong, E. D., Bepler, T., Davis, J. H. & Berger, B. Proc. 8th International Conference on Learning Representations (ICLR, 2020).
Xu, J. et al. Proc. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (IEEE, 2023).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).
LaValle, S. M. & Kuffner, J. J. in Algorithmic and Computational Robotics (ed. Rus, D.) 303–307 (A K Peters/CRC Press, 2001).
Bongard, J., Zykov, V. & Lipson, H. Resilient machines through continuous self-modeling. Science 314, 1118–1121 (2006).
MATH Google Scholar
Kucuk, S. & Bingul, Z. Robot Linematics: Forward and Inverse Kinematics (INTECH Open Access Publisher, 2006).
Coumans, E. ACM SIGGRAPH 2015 Courses (Association for Computing Machinery, 2015).
Paszke, A. et al. PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (eds Wallach, H. et al.) 8024–8035 (Curran Associates, Inc., 2019).
Agarap, A. F. Deep learning using rectified linear units (ReLU). Preprint at https://arxiv.org/abs/1803.08375 (2018).
Tancik, M. et al. Fourier features let networks learn high frequency functions in low dimensional domains. Adv. Neural Inf. Process. Syst. 33, 7537–7547 (2020).
MATH Google Scholar
Buda, M., Maki, A. & Mazurowski, M. A. A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw. 106, 249–259 (2018).
MATH Google Scholar
Lin, T.-Y., Goyal, P., Girshick, R., He, K. & Dollár, P. Proc. 2017 IEEE International Conference on Computer Vision (ICCV) (IEEE, 2017).
Hu, Y. & Lin, J. Teaching robots to build simulations of themselves. Zenodo https://doi.org/10.5281/zenodo.14539921 (2024).

Download references