publications

2024

2024

1. 

   CEAR: Comprehensive Event Camera Dataset for Rapid Perception of Agile Quadruped Robots

   Shifan Zhu, Zixun Xiong, and Donghyun Kim

   RAL, 2024

   Abs Bib HTML

   When legged robots perform agile movements, traditional RGB cameras often produce blurred images, posing a challenge for rapid perception. Event cameras have emerged as a promising solution for capturing rapid perception and coping with challenging lighting conditions thanks to their low latency, high temporal resolution, and high dynamic range. However, integrating event cameras into agile-legged robots is still largely unexplored. To bridge this gap, we introduce CEAR, a dataset comprising data from an event camera, an RGB-D camera, an IMU, a LiDAR, and joint encoders, all mounted on a dynamic quadruped, Mini Cheetah robot. This comprehensive dataset features more than 100 sequences from real-world environments, encompassing various indoor and outdoor environments, different lighting conditions, a range of robot gaits (e.g., trotting, bounding, pronking), as well as acrobatic movements like backflip. To our knowledge, this is the first event camera dataset capturing the dynamic and diverse quadruped robot motions under various setups, developed to advance research in rapid perception for quadruped robots. The CEAR dataset is available at https://daroslab.github.io/cear/ .

   @article{zhu2024cear,
     title = {CEAR: Comprehensive Event Camera Dataset for Rapid Perception of Agile Quadruped Robots},
     author = {Zhu, Shifan and Xiong, Zixun and Kim, Donghyun},
     journal = {RAL},
     year = {2024},
     video = {},
   }

2. 

   Impedance Matching: Enabling an RL-Based Running Jump in a Quadruped Robot

   Neil Guan, Shangqun Yu, Shifan Zhu, and 1 more author

   Ubiquitous Robots, 2024

   Abs Bib HTML

   Replicating the remarkable athleticism seen in animals has long been a challenge in robotics control. Although Reinforcement Learning (RL) has demonstrated significant progress in dynamic legged locomotion control, the substantial sim-to-real gap often hinders the real-world demonstration of truly dynamic movements. We propose a new framework to mitigate this gap through frequency-domain analysis-based impedance matching between simulated and real robots. Our framework offers a structured guideline for parameter selection and the range for dynamics randomization in simulation, thus facilitating a safe sim-to-real transfer. The learned policy using our framework enabled jumps across distances of 55 cm and heights of 38 cm. The results are, to the best of our knowledge, one of the highest and longest running jumps demonstrated by an RL-based control policy in a real quadruped robot. Note that the achieved jumping height is approximately 85% of that obtained from a state-of-the-art trajectory optimization method, which can be seen as the physical limit for the given robot hardware. In addition, our control policy accomplished stable walking at speeds up to 2 m/s in the forward and backward directions, and 1 m/s in the sideway direction.

   @article{guan2024impedance,
     title = {Impedance Matching: Enabling an RL-Based Running Jump in a Quadruped Robot},
     author = {Guan, Neil and Yu, Shangqun and Zhu, Shifan and Kim, Donghyun},
     journal = {Ubiquitous Robots},
     year = {2024},
     video = {},
   }

2023

2023

1. 

   Event camera-based visual odometry for dynamic motion tracking of a legged robot using adaptive time surface

   Shifan Zhu, Xinyu Zhang, Shichun Guo, and 2 more authors

   IROS, 2023

   Abs Bib HTML

   Our paper proposes a direct sparse visual odometry method that combines event and RGBD data to estimate the pose of agile-legged robots during dynamic locomotion and acrobatic behaviors. Event cameras offer high temporal resolution and dynamic range, which can eliminate the issue of blurred RGB images during fast movements. This unique strength holds a potential for accurate pose estimation of agile- legged robots, which has been a challenging problem to tackle. Our framework leverages the benefits of both RGBD and event cameras to achieve robust and accurate pose estimation, even during dynamic maneuvers such as jumping and landing a quadruped robot, the Mini-Cheetah. Our major contributions are threefold: Firstly, we introduce an adaptive time surface (ATS) method that addresses the whiteout and blackout issue in conventional time surfaces by formulating pixel-wise decay rates based on scene complexity and motion speed. Secondly, we develop an effective pixel selection method that directly samples from event data and applies sample filtering through ATS, enabling us to pick pixels on distinct features. Lastly, we propose a nonlinear pose optimization formula that simultaneously performs 3D-2D alignment on both RGB-based and event-based maps and images, allowing the algorithm to fully exploit the benefits of both data streams. We extensively evaluate the performance of our framework on both the public dataset and our own quadruped robot dataset, demonstrating its effectiveness in accurately estimating the pose of agile robots during dynamic movements. Supplemental video: https://youtu.be/-5ieQShOg3M

   @article{zhu2021event,
     title = {Event camera-based visual odometry for dynamic motion tracking of a legged robot using adaptive time surface},
     author = {Zhu, Shifan and Zhang, Xinyu and Guo, Shichun and Li, Jun and Liu, Huaping},
     journal = {IROS},
     year = {2023},
     video = {https://youtu.be/-5ieQShOg3M},
   }

2. 

   Dynamic Object Avoidance using Event-Data for a Quadruped Robot

   Shifan Zhu, Nisal Perera, Shangqun Yu, and 2 more authors

   IROS IPPC Workshop, 2023

   Abs Bib HTML

   As robots increase in agility and encounter fast- moving objects, dynamic object detection and avoidance become notably challenging. Traditional RGB cameras, burdened by motion blur and high latency, often act as the bottleneck. Event cameras have recently emerged as a promising solution for the challenges related to rapid movement. In this paper, we introduce a dynamic object avoidance framework that integrates both event and RGBD cameras. Specifically, this framework first estimates and compensates for the event’s motion to detect dynamic objects. Subsequently, depth data is combined to derive a 3D trajectory. When initiating from a static state, the robot adjusts its height based on the predicted collision point to avoid the dynamic obstacle. Through real-world experiments with the Mini-Cheetah, our approach successfully circumvents dynamic objects at speeds up to 5 m/s, achieving an 83% success rate. Supplemental video:

   @article{zhu2023dynamic,
     title = {Dynamic Object Avoidance using Event-Data for a Quadruped Robot},
     author = {Zhu, Shifan and Perera, Nisal and Yu, Shangqun and Hwang, Hochul and Kim, Donghyun},
     journal = {IROS IPPC Workshop},
     year = {2023},
     video = {https://www.youtube.com/watch?v=wEPvynkVlLA},
   }

2021

2021

1. 

   Lifelong Localization in Semi-Dynamic Environment

   Shifan Zhu, Xinyu Zhang, Shichun Guo, and 2 more authors

   ICRA, 2021

   Abs Bib HTML

   Mapping and localization in non-static environments are fundamental problems in robotics. Most of previous methods mainly focus on static and highly dynamic objects in the environment, which may suffer from localization failure in semi-dynamic scenarios without considering objects with lower dynamics, such as parked cars and stopped pedestrians. In this paper, we introduce semantic mapping and lifelong localization approaches to recognize semi-dynamic objects in non-static environments. We also propose a generic framework that can integrate mainstream object detection algorithms with mapping and localization algorithms. The mapping method combines an object detection algorithm and a SLAM algorithm to detect semi-dynamic objects and constructs a semantic map that only contains semi-dynamic objects in the environment. During navigation, the localization method can classify observation corresponding to static and non-static objects respectively and evaluate whether those semi-dynamic objects have moved, to reduce the weight of invalid observation and localization fluctuation. Real-world experiments show that the proposed method can improve the localization accuracy of mobile robots in non-static scenarios.

   @article{zhu2021life,
     author = {Zhu, Shifan and Zhang, Xinyu and Guo, Shichun and Li, Jun and Liu, Huaping},
     title = {Lifelong Localization in Semi-Dynamic Environment},
     journal = {ICRA},
     year = {2021},
     video = {},
   }

2. 

   Line-based Automatic Extrinsic Calibration of LiDAR and Camera

   Xinyu Zhang, Shifan Zhu, Shichun Guo, and 2 more authors

   ICRA, 2021

   Abs Bib HTML

   Reliable real-time extrinsic parameters of 3D Light Detection and Ranging (LiDAR) and camera are a key component of multi-modal perception systems. However, extrinsic transformation may drift gradually during operation, which can result in decreased accuracy of perception system. To solve this problem, we propose a line-based method that enables automatic online extrinsic calibration of LiDAR and camera in real-world scenes. Herein, the line feature is selected to constrain the extrinsic parameters for its ubiquity. Initially, the line features are extracted and filtered from point clouds and images. Afterwards, an adaptive optimization is utilized to provide accurate extrinsic parameters. We demonstrate that line features are robust geometric features that can be extracted from point clouds and images, thus contributing to the extrinsic calibration. To demonstrate the benefits of this method, we evaluate it on KITTI benchmark with ground truth value. The experiments verify the accuracy of the calibration approach. In online experiments on hundreds of frames, our approach automatically corrects miscalibration errors and achieves an accuracy of 0.2 degrees, which verifies its applicability in various scenarios. This work can provide basis for perception systems and further improve the performance of other algorithms that utilize these sensors.

   @article{zhang2021line,
     title = {Line-based Automatic Extrinsic Calibration of LiDAR and Camera},
     author = {Zhang, Xinyu and Zhu, Shifan and Guo, Shichun and Li, Jun and Liu, Huaping},
     journal = {ICRA},
     year = {2021},
     video = {},
   }