NVIDIA Announces New AI Perception Coming to ROS Developers
September 22, 2021
News
NVIDIA and Open Robotics have entered into an agreement to accelerate ROS performance on NVIDIA’s Jetson edge AI platform and GPU-based systems. These initiatives are designed to reduce development time and improve performance for developers seeking to incorporate computer vision and AI/machine learning functionality into their ROS-based applications.
According to the company, Open Robotics will enhance ROS 2 to enable efficient management of data flow and shared memory across GPU and other processors present on the NVIDIA Jetson edge AI platform. This is designed to improve the performance of applications that have to process high-bandwidth data from sensors such as cameras and lidars in real time.
In addition, Open Robotics and NVIDIA are working to enable seamless simulation interoperability between Open Robotics’s Ignition Gazebo and NVIDIA Isaac Sim on Omniverse. Isaac Sim already supports ROS 1 and 2 out of the box and features an ecosystem of 3D content with its connection to applications, such as Blender and Unreal Engine 4.
Ignition Gazebo brings a decades-long track record of widespread use throughout the robotics community, including in high-profile competition events such as the ongoing DARPA Subterranean Challenge.
With the two simulators connected, ROS developers can move their robots and environments between Ignition Gazebo and Isaac Sim to run large-scale simulation and take advantage of each simulator’s advanced features such as high-fidelity dynamics, accurate sensor models, and photorealistic rendering to generate synthetic data for training and testing of AI models.
Software resulting from this collaboration is expected to be released in the spring of 2022.
Isaac GEMs for ROS are hardware accelerated packages designed to make it easier for ROS developers to build high-performance solutions on the Jetson platform. The focus of these GEMs is on improving throughput on image processing and on DNN-based perception models that are of growing importance to roboticists. These packages reduce the load on the host CPU while providing significant performance gain.
The new Isaac GEMs for ROS include:
- SGM Stereo Disparity and Point Cloud
- Color Space Conversion and Lens Distortion Correction
- AprilTags Detection
Figure 1. Stereo camera support in ROS depicted here with the left and right camera view in the ROS Rviz tool. Both RGB and depth images are shown in RViz.
The latest release of Isaac Sim includes support for the ROS developer community. Some examples of this are the ROS2 Navigation stack and the MoveIt Motion Planning Framework. These examples are available today and can be found in the Isaac Sim documentation here.
List of ROS Examples in Isaac Sim
- ROS April Tag
- ROS Stereo Camera
- ROS Navigation
- ROS TurtleBot3 Sample
- ROS Manipulation and Camera Sample
- ROS Services
- MoveIt Motion Planning Framework
- Native Python ROS Usage
- ROS2 Navigation
Figure 2. Functional block diagram of Isaac Sim on Omniverse showing three inputs to the simulator: the robot model, environment model, and 3D assets. The simulator can be controlled from both ROS and python. Outputs of the simulator include the digital twin and the synthetic data which can be used to train perception models.
In addition to being a robotic simulator, Isaac Sim contains a set of capabilities that generate synthetic data to train and test perception models. These capabilities will be more important as roboticists incorporate more perception features into their platforms. It's clear that the better that a robot can perceive its environment the more autonomous it can be, thereby requiring less human intervention.
Once Isaac Sim generates synthetic datasets, they can be fed directly into NVIDIA’s TAO, an AI model adaptation platform, to adapt perception models for a robot’s specific working environment. The task of ensuring that a robot’s perception stack is going to perform in a given working environment can be started well before any real-data is ever collected from the target surroundings.
Roboticists have long faced challenges in connecting and integrating the classic robotic tasks like navigation to AI-based perception stacks. Isaac Sim addresses this workflow challenge by being simultaneously a full robotics and synthetic data generation tool with streamlined integration with the TAO training platform.
NVIDIA is gearing up for ROS World on Oct 21-22, 2021. "We are planning to release more new GEMs for Jetson developers including several popular DNNs. We will also announce features in Isaac Sim to support the ROS developer community. Be sure to stop by our virtual booth, attend our NVIDIA ROS roundtable, watch the technical presentation on Isaac Sim, and more."
NVIDIA has a great lineup of speakers, talks, and content at the upcoming GTC scheduled for Nov. 8-11. "We have a track for robotics developers including a presentation by Brian Gerkey, CEO of Open Robotics. Additionally, we have talks covering NVIDIA Jetson, Isaac ROS, Isaac Sim, and Isaac GYM."
Developers interested in getting started today on adding NVIDIA AI Perception to their products should leverage the links below:
Download Isaac GEMs for ROS here
More Info on Isaac Sim here
Tutorials on Synthetic Data Generation with Isaac Sim here
More Info on Accelerating ML Training with TAO Toolkit here
NVIDIA Jetson & JetPack – details here
NVIDIA DeepStream – details here
NVIDIA Fleet Command – details here
NVIDIA TRITON Inference Server - details here
NVIDIA RIVA – details here
For more information, visit: https://developer.nvidia.com/blog/nvidia-ai-perception-coming-to-ros-developers/
