Jacky Liang

Hello!

I’m a research scientist at Google DeepMind, and my research focus is on foundation models for robotics. I received my PhD from Carnegie Mellon University’s Robotics Institute, co-advised by Prof. Oliver Kroemer and Prof. Maxim Likhachev. My PhD thesis is on Learning with Structured Priors for Robust Robot Manipulation. A recording of my PhD defense is here.

Previously, I interned at NVIDIA for the Isaac project, applying GPU-accelerated physics simulations for robot learning. During my time at UC Berkeley, I did research in robot grasping and manipulation with Prof. Ken Goldberg.

In my spare time I write for and co-run Skynet Today and the Last Week in AI newsletter. Our goal is to demystify and disseminate AI news without hype. Check us out and subscribe if you like what you see :)

Research

Conference Papers

Code as Policies: Language Model Programs for Embodied Control

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng. International Conference on Robotics and Automation (ICRA). June 2023. Outstanding Paper in Robot Learning. [arXiv] [Website]

Learning Preconditions of Hybrid Force-Velocity Controllers for Contact-Rich Manipulation

Jacky Liang, Xianyi Cheng, Oliver Kroemer. Conference on Robot Learning (CoRL). Dec 2022. [arXiv] [Website]

Inner Monologue: Embodied Reasoning through Planning with Language Models

Wenlong Huang*, Fei Xia*, Ted Xiao*, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter. Conference on Robot Learning (CoRL). Dec 2022. [arXiv] [Website]

Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation

Jacky Liang*, Mohit Sharma*, Alex LaGrassa, Shivam Vats, Saumya Saxena, Oliver Kroemer. International Conference on Robotics and Automation (ICRA). May 2022. [arXiv] [Website]

Visual Identification of Articulated Object Parts

Vicky Zeng, Timothy E. Lee*, Jacky Liang*, Oliver Kroemer. International Conference on Intelligent Robots and Systems (IROS). September 2021. [arXiv] [Website]

Contact Localization for Robot Arms in Motion without Torque Sensing

Jacky Liang, Oliver Kroemer. International Conference on Robotics and Automation (ICRA). May 2021. [arXiv] [Website]

Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation

Mohit Sharma*, Jacky Liang*, Jialiang Zhao, Alex LaGrassa, Oliver Kroemer. Conference on Robot Learning (CoRL). November 2020. Plenary Presentation. [arXiv] [Website]

Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap

Jacky Liang, Saumya Saxena, Oliver Kroemer. Robotics: Science and Systems (RSS). July 2020. [arXiv] [Website]

In-Hand Object Pose Tracking via Contact Feedback and GPU-Accelerated Robotic Simulation

Jacky Liang, Ankur Handa, Karl Van Wyk, Viktor Makoviychuk, Oliver Kroemer, Dieter Fox. International Conference on Robotics and Automation (ICRA). May 2020. [arXiv] [Website]

DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System

Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox. International Conference on Robotics and Automation (ICRA). May 2020. [arXiv] [Website]

Homography-Based Deep Visual Servoing Methods for Planar Grasps

Austin S. Wang, Wuming Zhang, Daniel Troniak, Jacky Liang, Oliver Kroemer. International Conference on Intelligent Robots and Systems (IROS). November 2019. [Paper]

Towards Precise Robotic Grasping by Probabilistic Post-grasp Displacement Estimation

Jialiang Zhao, Jacky Liang, Oliver Kroemer. Field and Service Robotics (FSR). August 2019. [arXiv]

GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning

Jacky Liang*, Viktor Makoviychuk*, Ankur Handa*, Nuttapong Chentanez, Miles Macklin, Dieter Fox. Conference on Robot Learning (CoRL). October 2018. [arXiv] [Website]

Using dVRK Teleoperation to Facilitate Deep Learning of Automation Tasks for an Industrial Robot

Jacky Liang, Jeffrey Mahler, Michael Laskey, Pusong Li, Ken Goldberg. Conference on Automation Science and Engineering (CASE). August 2017. Finalist, Best Student Paper Award. [PDF] [Website]

Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, Ken Goldberg. Robotics: Science and Systems (RSS). Cambridge, MA. July 2017. [PDF] [Website]

Design of Parallel-Jaw Gripper Tip Surfaces for Robust Grasping

Menglong Guo, David V. Gealy, Jacky Liang, Jeffrey Mahler, Aimee Goncalves, Stephen McKinley, Ken Goldberg. International Conference on Robotics and Automation (ICRA). Singapore, May 2017. [PDF]

Undergraduate-Led Survey Class to Improve CS Education for New Students

Nathan Zhang*, Jacky Liang*, Amanda Tomlinson*, Frank Boensch, Anant Sahai. SIGCSE ‘20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education. February 2020. [Paper]

Technical Reports

A Modular Robotic Arm Control Stack for Research: Franka-Interface and FrankaPy

Kevin Zhang*, Mohit Sharma*, Jacky Liang*, Oliver Kroemer. November 2020. [arXiv]

* Equal Contribution.

Teaching

16-350 Planning Techniques for Robotics

Spring 2021 Teaching Assistant

16-662 Robot Autonomy

Spring 2020 Teaching Assistant

CS188 Introduction to Artificial Intelligence

Spring 2017 Discussion Teaching Assistant

EE16A Design of Information Devices and Systems

Fall 2016, 2017 Discussion Teaching Assistant

Education

Carnegie Mellon University. Robotics Institute. PhD

University of California Berkeley. Electrical Engineering and Computer Science. Bachelor of Science, Class of 2018

Open-Source Libraries

Language Model Programs - Interactive colab notebook that implements language model programs for a simulated table-top manipulation domain.
FrankaPy - Python library for controlling the Franka Emika Panda robot arm.
IsaacGym-Utils - Python library that provides simpler APIs for the Nvidia Isaac Gym robot simulator
Simple ZMQ - Python library that uses zmq to send arbtirary objects over a network
Async Savers - Python library for asynchronously saving data in shards
Data Learning Boilerplate - Boilerplate Python code for collecting data and training neural networks using W&B and PyTorch Lightning