UTIAS

��TV team takes top spot in self-driving car challenge for 6th time in 7 years

Christopher.Sorensen — Wed, 07 Aug 2024 17:42:55 +0000

��TV team takes top spot in self-driving car challenge for 6th time in 7 years

Christopher.Sorensen Wed, 08/07/2024 - 13:42

Cutline

As part of the competition, the ��TV team’s autonomous vehicle had to react to obstacles such as a fake deer moving across the road (photo courtesy of aUToronto)

Safa Jinje

Topic

Our Community

Alumni

Artificial Intelligence

Faculty of Applied Science & Engineering

Graduate Students

Research & Innovation

Self-Driving Cars

Undergraduate Students

UTIAS

Subheadline

"Each time we saw an obstacle – a stop sign, a red light, the railroad bar coming down – and the car reacted by stopping and then continuing, we let out a big cheer or a sigh of relief"

A team from the University of Toronto has placed first for sixth time in seven years in a North American self-driving car competition.

After finishing in second place last year, the aUToronto team returned to the top spot at the 2024 SAE AutoDrive Challenge II, which was held in June at the Mcity Test Facility in Ann Arbor, Mich.

The aUToronto team competed against nine other teams from across Canada and the United States.

“Through the AutoDrive Challenge, we are preparing the next generation of engineers to head into the industry, to keep pushing towards the challenging goal of reaching Level 4 autonomous driving,” says Tim Barfoot, a professor at the ��TV Institute for Aerospace Studies (UTIAS) in the Faculty of Applied Science & Engineering and one of the team’s academic advisers.

“The team did another excellent job this year.”

The team approached the competition by going back to first principles to ensure they had a reliable and robust system, says Kelvin Cui, a ��TV Engineering alumnus and the team’s principal.

He joined aUToronto last fall after five years with the University of Toronto Formula Racing team, where he founded the “driverless” division.

“We looked at what was going to get us the most points at competition and made sure that we were not overbuilding our system and adding too much complexity,” he says.

This meant pushing for additional testing time at UTIAS and achieving more than 900 kilometres of system testing prior to the competition.

The team placed first out of 10 teams from institutions across the United States and Canada (photo courtesy of aUToronto)

A partnership with the AutoDrive team from Queen’s University was instrumental to aUToronto’s preparation. The aUToronto team drove Artemis, their autonomous vehicle, to Kingston, Ont. to assess the system at Queen’s testing facility, which features intersections and electronic streetlights.

“We added radar to our vehicle as a new sensor, so we needed to be aware of all the sensor failure modes,” says third-year Engineering Science student Robert Ren.

“A lot of our testing time went into making sure that including radar didn’t break anything else in our system, and that it could handle any sensor failure cases.”

Including radar sensors in the vehicle’s perception system allowed it to measure the motion of objects directly, which is not possible with light detection and ranging (LiDAR) sensors. 

“Radar can help with adverse weather object detections,” adds Ren. “So, if the vehicle is operating under heavy rain or fog, the LiDAR is going to be limited, but the radio waves from radar can help the vehicle see what objects are in front and what objects are moving. This enables it to make good decisions when driving in uncertain scenarios.” 

View this post on Instagram

A post shared by aUToronto (@autoronto_uoft)

In an event where both LiDAR and radar sensors fail, the aUToronto system can still rely on visual cameras to perform object tracking. This made the team’s object tracker much more robust compared to last year when the team experienced sensor failure during a dynamic event.

Brian Cheong, a ��TV Engineering master’s student who has been a member of aUToronto since 2021, acted as technical director of the autonomy team this year – part of a new leadership structure introduced by Cui.  

“In the past, it was a lot of work for our team’s principal to keep track of all the systems,” Cheong says. “So instead of having to work directly with all 15 sub teams, Kelvin created groups of sub teams that we called stacks, and each stack had a director.”

The restructuring and technical innovations paid off, with aUToronto completing its first clean sweep in the AutoDrive Challenge II, placing first in all static and dynamic events, including the concept design presentation and intersection challenge.

“The intersection challenge was a big highlight for us,” says Cheong. “Kelvin and Robert were in the car, and I was on the sidelines watching with the rest of the team. Each time we saw an obstacle – a stop sign, a red light, the railroad bar coming down – and the car reacted by stopping and then continuing, we let out a big cheer or a sigh of relief.

“And then we were all silent as the car approached the final obstacle, which was a deer. We watched as Artemis slowed down to a stop and the deer moved by. Then we screamed and cheered, and we could hear cheering from inside the car.”

“Our success is entirely a team effort,” adds Cui. “It was not smooth sailing before the competition. The only reason we won is because everybody put in so much effort to test our vehicle every day.

“That’s how we were able to get this reliable system across the line.” 

Start@UTIAS

laurie.bulchak — Fri, 26 Jul 2024 01:33:52 +0000

Start@UTIAS laurie.bulchak Thu, 07/25/2024 - 21:33

URL

https://www.utias.utoronto.ca/startutias-entrepreneurship-program/

��TV researchers enhance object-tracking abilities of self-driving cars

rahul.kalvapalle — Wed, 29 May 2024 14:59:42 +0000

��TV researchers enhance object-tracking abilities of self-driving cars

rahul.kalvapalle Wed, 05/29/2024 - 10:59

Cutline

Sandro Papais, a PhD student, is the co-author of a new paper that introduces a graph-based optimization method to improve object tracking for self-driving cars (photo courtesy of aUToronto)

Safa Jinje

Topic

Breaking Research

Artificial Intelligence

Faculty of Applied Science & Engineering

Subheadline

The new tools could help robotic systems of autonomous vehicles better track the position and motion of vehicles, pedestrians and cyclists

Researchers at the University of Toronto Institute for Aerospace Studies (UTIAS) have introduced a pair of high-tech tools that could improve the safety and reliability of autonomous vehicles by enhancing the reasoning ability of their robotic systems.

The innovations address multi-object tracking, a process used by robotic systems to track the position and motion of objects – including vehicles, pedestrians and cyclists – to plan the path of self-driving cars in densely populated areas.

Tracking information is collected from computer vision sensors (2D camera images and 3D LIDAR scans) and filtered at each time stamp, 10 times a second, to predict the future movement of moving objects.

“Once processed, it allows the robot to develop some reasoning about its environment. For example, there is a human crossing the street at the intersection, or a cyclist changing lanes up ahead,” says Sandro Papais, a PhD student in UTIAS in the Faculty of Applied Science & Engineering. "At each time stamp, the robot’s software tries to link the current detections with objects it saw in the past, but it can only go back so far in time.”

In a new paper presented at the 2024 International Conference on Robotics and Automation in Yokohama, Japan, Papais and co-authors Robert Ren, a third-year engineering science student, and Professor Steven Waslander, director of UTIAS’s Toronto Robotics and AI Laboratory, introduce Sliding Window Tracker (SWTrack) – a graph-based optimization method that uses additional temporal information to prevent missed objects.

The tool is designed to improve the performance of tracking methods, particularly when objects are occluded from the robot’s point of view.

A visualization of a nuScenes dataset used by the researchers. The image is a mosaic of the six different camera views around the car with the object bounding boxes rendered overtop of the images (image courtesy of the Toronto Robotics and AI Laboratory)

“SWTrack widens how far into the past a robot considers when planning,” says Papais. “So instead of being limited by what it just saw one frame ago and what is happening now, it can look over the past five seconds and then try to reason through all the different things it has seen.” 

The team tested, trained and validated their algorithm on field data obtained through nuScenes, a public, large-scale dataset for autonomous driving vehicles that have operated on roads in cities around the world. The data includes human annotations that the team used to benchmark the performance of SWTrack.

They found that each time they extended the temporal window, to a maximum of five seconds, the tracking performance got better. But past five seconds, the algorithm’s performance was slowed by computation time.

“Most tracking algorithms would have a tough time reasoning over some of these temporal gaps. But in our case, we were able to validate that we can track over these longer periods of time and maintain more consistent tracking for dynamic objects around us,” says Papais.

Papais says he’s looking forward to building on the idea of improving robot memory and extending it to other areas of robotics infrastructure. “This is just the beginning,” he says. “We’re working on the tracking problem, but also other robot problems, where we can incorporate more temporal information to enhance perception and robotic reasoning.”

Another paper, co-authored by master’s student Chang Won (John) Lee and Waslander, introduces UncertaintyTrack, a collection of extensions for 2D tracking-by-detection methods that leverages probabilistic object detection.

“Probabilistic object detection quantifies the uncertainty estimates of object detection,” explains Lee. “The key thing here is that for safety-critical tasks, you want to be able to know when the predicted detections are likely to cause errors in downstream tasks such as multi-object tracking. These errors can occur because of low-lighting conditions or heavy object occlusion.

“Uncertainty estimates give us an idea of when the model is in doubt, that is, when it is highly likely to give errors in predictions. But there’s this gap because probabilistic object detectors aren’t currently used in multi-tracking object tracking.” 

Lee worked on the paper as part of his undergraduate thesis in engineering science. Now a master’s student in Waslander’s lab, he is researching visual anomaly detection for the Canadarm3, Canada’s contribution to the U.S.-led Gateway lunar outpost. “In my current research, we are aiming to come up with a deep-learning-based method that detects objects floating in space that pose a potential risk to the robotic arm,” Lee says.

Waslander says the advancements outlined in the two papers build on work that his lab has been focusing on for a number of years.

“[The Toronto Robotics and AI Laboratory] has been working on assessing perception uncertainty and expanding temporal reasoning for robotics for multiple years now, as they are the key roadblocks to deploying robots in the open world more broadly,” Waslander says.

“We desperately need AI methods that can understand the persistence of objects over time, and ones that are aware of their own limitations and will stop and reason when something new or unexpected appears in their path. This is what our research aims to do.”

With the launch of its first satellite, student team charts a course to new knowledge

rahul.kalvapalle — Fri, 19 Jan 2024 17:44:03 +0000

With the launch of its first satellite, student team charts a course to new knowledge

rahul.kalvapalle Fri, 01/19/2024 - 12:44

Cutline

A Falcon 9 rocket lifts off from Vandenberg Space Force Base on Nov. 11, 2023, carrying a satellite designed and built by the University of Toronto Aerospace Team (photo courtesy of SpaceX)

Safa Jinje

Topic

Our Community

Aerospace

Electrical & Computer Engineering

Faculty of Applied Science & Engineering

Graduate Students

Mechanical & Industrial Engineering

Space

Undergraduate Students

UTIAS

Subheadline

“We worked on this project for so long with such a narrow focus that actually seeing it deployed was very rewarding”

Students in the University of Toronto’s Faculty of Applied Science & Engineering recently gathered in the basement of the Sandford Fleming Building – known to many as “The Pit” – to witness the deployment of HERON Mk. II into space.

The 3U CubeSat satellite, built and operated by the space systems division of the University of Toronto Aerospace Team (UTAT), was launched into orbit on a Falcon 9 rocket on Nov. 11, 2023 as part of SpaceX’s Transporter-9 rideshare mission that lifted off from the Vandenberg Space Force Base near Lompoc, Calif.

The feat was entirely student funded with support from ��TV Engineering through student levies and UTAT-led fundraising efforts.

“The experience of the launch was very surreal,” says master’s degree student Benjamin Nero, HERON’s current mission manger.

“We worked on this project for so long with such a narrow focus that actually seeing it deployed was very rewarding.”

“There are any number of things that could go wrong that might prevent a satellite from deploying,” adds Zachary Teper, a fellow master’s degree candidate who is part of the technical development team working on HERON’s ground station.

“So, watching each of the call outs coming out of the SpaceX mission control, seeing the rocket go up and meet every one of its mission objectives and then finally seeing our satellite get ejected out of the dispenser in the correct trajectory was a big relief – because we knew that it was finally in space and on the right path.”

Members of the UTAT space systems division gather on the sixth-floor roof of the Bahen Centre for Information Technology with the fully assembled ground station (photo by UTAT Space Systems)

Launching HERON – short for High frequency Educational Radio communications On a Nanosatellite – was the culmination of years of teamwork that brought together the efforts of more than 100 students.

HERON Mk. II, the second iteration of UTAT’s spacecraft, was originally designed and built between 2016 and 2018 for the fourth edition of the Canadian Satellite Design Challenge. Since space systems division was formed in 2014, many of the students who worked on the initial HERON design and build have since graduated. But the current operations team continued to develop the satellite and renew the student levy that allowed them to secure their space launch.

“The original objective for HERON was to conduct a biology experiment in space,” says Nero, who joined the team in 2019 during his second year of undergraduate studies. “But because of delays in the licensing process, we were unable to continue that mission objective. So, we re-scoped and shifted our focus to amateur radio communication and knowledge building.”

From left to right: HERON Mk. I (2016), HERON Mk. II Prototype (2018), HERON Mk. II Softstack (2020), HERON Mk. II Flight Model (2021) (photos by UTAT Space Systems)

Once the satellite’s final assembly was completed in 2021, the team began flight model testing and assembling a ground station, while also managing the logistics of the regulatory approvals needed to complete the launch.

“It’s difficult to put something in space, both technically and bureaucratically,” says Nero. “There are a lot of different governments that care about what you’re doing and want to know when and how you’re doing it.”

Getting to space was a significant milestone for the team, but it’s still only the beginning of their work.

“The goal for us as a design team is to start gathering institutional knowledge that we didn’t have before,” says Reid Sox-Harris, an undergraduate student who is HERON’s ground station manager and the electrical lead for UTAT’s next space mission, FINCH (Field Imaging Nanosatellite for Crop residue Hyperspectral mapping).

“We’ve never operated a satellite. So, we’re taking a lot of lessons learned with us through this process.” 

For example, when a satellite is deployed for the first time, the ground control team only has a rough idea of its movement and eventual location. They must simulate the launch to figure out exactly where it is before they can establish a connection. And when they receive new positional data, they must rerun their simulation.

“We have to take into account effects such as air resistance, or the sun’s solar cycles and the gravitational effects of the sun, the moon and the Earth – it’s a fairly complicated simulation,” Sox-Harris says.

Nero adds: “Part of the difficulty with a simulation is that a model is only useful for a certain period. An old estimate could result in as much as a few kilometres of drift from the satellite’s actual position per day.”

HERON’s ground station on the roof of the Bahen Centre (photo by UTAT Space Systems)

The team was not only tasked with designing a ground station capable of communicating with a satellite more than 500 kilometres away, but one that can survive a frigid and snowy Canadian winter.

“For any project, the most important thing you should be doing is testing,” says second-year student Swarnava Ghosh, who primarily works on the ground station software. “One challenge with our ground station currently is that there are too many variables that are not fully tested – and everything needs to be perfect in the chain for the communication to work. If the ground station is not pointing in the right direction, we won’t get a signal and we won’t establish communication. And if the amplifier is not working, then we won’t establish communication.” 

The team is confident that they will ultimately resolve any outstanding issues and establish communications with HERON. More importantly, they will be able to take what they’ve learned and apply it to the next mission.

“With FINCH, we want to make sure the ground station software and satellite can communicate on the ground,” says Sox-Harris. “Right now, there are over 500 kilometres between the satellite and ground station, so we can’t fly up there and test whether a command has worked.”

FINCH is set to launch in late 2025 on a rideshare rocket flight. Its current mission objective is to generate hyperspectral imaging maps of crop residue on farm fields in Manitoba from a low-Earth orbit.

There are many technical developments that are new to FINCH that weren’t applicable to HERON, the team says, including a novel optic system for remote sensing that is being developed by students.

“The risks associated with FINCH are mitigated by the work that is being performed by HERON right now. We’re learning many lessons that will be directly applicable to our next mission, and we’ll continue to learn from HERON for at least another year or more,” says Sox-Harris.

“This means the FINCH mission can be more complicated, it can move faster and ultimately we can have better reliability, which is something that we always strive for in aerospace.”

��TV researchers partner with Siemens Energy to tackle sustainable energy production

Christopher.Sorensen — Wed, 10 Jan 2024 19:50:56 +0000

��TV researchers partner with Siemens Energy to tackle sustainable energy production

Christopher.Sorensen Wed, 01/10/2024 - 14:50

Cutline

PhD student Yazdan Naderzadeh (left) investigates flames with lasers in the Propulsion and Energy Conversion Lab at UTIAS (photo by Neil Ta)

Selah Katona

Topic

Our Community

Industry Partnerships

Faculty of Applied Science & Engineering

Graduate Students

Sustainability

UTIAS

Subheadline

'Together, we hope to unravel the complexities of hydrogen combustion, paving the way for cleaner and more efficient engines'

Researchers in the University of Toronto’s Faculty of Applied Science & Engineering have partnered with Siemens Energy to tackle a key challenge in the energy sector: sustainable energy conversion for propulsion and power generation – such as developing gas turbine engines that can run on sustainable energy sources like hydrogen.

Led by Associate Professor Swetaprovo Chaudhuri from the ��TV Institute of Aerospace Studies (UTIAS), the initiative aims to rethink traditional gas turbine engines to reduce carbon emissions from both aviation and land-based fuel consumption.

Chaudhuri’s team is exploring hydrogen combustion as a viable option since it can be burned without producing carbon dioxide.

However, the transition is not without its challenges. For one, hydrogen is a small, highly reactive molecule, causing flames to move five to ten times faster than those of natural gas. This makes existing combustors and engines that run on natural gas incapable of handling pure hydrogen.

Another key challenge is the lack of infrastructure available to transport hydrogen in the way pipelines are used to move natural gas. Until such infrastructure is developed, Chaudhuri’s team is researching how to build reliable fuel-flex gas turbine engines that can work on both fuels.

“Hydrogen and natural gas are vastly different - it’s like comparing a Bugatti Veyron to a public bus in both speed and size,” says Chaudhuri, who leads the Propulsion & Energy Conversion Laboratory at UTIAS. “The critical question is: ‘how can engines be designed to accommodate both fuels seamlessly?’”

The team is led by Chaudhuri in collaboration with Associate Professor Jeff Bergthorson at McGill University, Professor Étienne Robert and Assistant Professor Bruno Savard at Polytechnique Montréal, Patrizio Vena at National Research Council Canada and engineers from Siemens Energy Canada in Montreal.

The project received an Alliance Mission Grant from the Natural Sciences and Engineering Research Council (NSERC) to build a comprehensive understanding that will guide the creation of fuel-flex gas turbine engines.

PhD candidate Yazdan Naderzadeh (left) and master’s student Scott Watson from the Propulsion and Energy Conversion Lab work with a swirling hydrogen flame (photo Praful Kumar)

The researchers have constructed a model lab-scale combustor at the Propulsion and Energy Conversion Laboratory at UTIAS, to study the behaviour of natural gas and hydrogen flames within engines. These experiments aim to understand the intricacies of hydrogen combustion to establish engineering principles and guidelines for future engine development.

While practical applications are on the horizon, the immediate goal is to establish a robust knowledge base that will be essential for designing engines that can efficiently and safely use hydrogen as a fuel source.

“Currently, long-range aircrafts cannot, even theoretically, fly on batteries. We need to make significant strides towards combustion engines that use hydrogen or other carbon-neutral fuels to substantially reduce carbon emissions in these critical sectors,” says Chaudhuri.

In a different, stand-alone project, Chaudhuri and his research group are developing a self-decarbonizing combustor, which separates hydrogen and carbon from natural gas within the combustor. This process not only allows for hydrogen to be used for fuel but could also allow the carbon byproduct to be used to offset the additional cost associated with decarbonization.

“Our collaboration with Siemens Energy marks an exciting synergy between academia and industry,” says Chaudhuri. “Siemens Energy’s gas turbines for generating power have historically used natural gas, so this partnership represents a significant step towards a greener future.

“Together, we hope to unravel the complexities of hydrogen combustion, paving the way for cleaner and more efficient engines.”

The development and commissioning of the fuel-flex combustor, capable of safely stabilizing both hydrogen and natural gas flames, presents numerous research opportunities for students.

Yazdan Naderzadeh and Scott Watson, a PhD candidate and master’s student respectively in Chaudhuri’s lab, are working on the project. “I am so excited to work on the ongoing fuel-flex combustor project, addressing concerns related to clean emissions and compatibility with conventional gas turbine burners,” says Naderzadeh. “This endeavor allows for a thorough study and understanding of the challenges associated with hydrogen as a prospective fuel in the aviation industry and gas power plants.”

Learn more about industry partnerships at ��TV

AI algorithm improves predictive models of complex dynamical systems

Christopher.Sorensen — Wed, 15 Nov 2023 14:20:37 +0000

AI algorithm improves predictive models of complex dynamical systems

Christopher.Sorensen Wed, 11/15/2023 - 09:20

Cutline

From left to right: Professor Prasanth Nair and PhD student Kevin Course are the authors of a new paper in Nature that introduces a new machine learning algorithm that addresses the challenge of imperfect knowledge about system dynamics (supplied images)

Safa Jinje

Topic

Breaking Research

Artificial Intelligence

Faculty of Applied Science & Engineering

Research & Innovation

UTIAS

Subheadline

Developed by ��TV researchers, the new approach could have applications ranging from predicting the performance of aircraft engines to forecasting climate change

Researchers at the University of Toronto have made a significant step towards enabling reliable predictions of complex dynamical systems when there are many uncertainties in the available data or missing information.

In a recent paper published in Nature, Prasanth B. Nair, a professor at the ��TV Institute of Aerospace Studies (UTIAS) in the Faculty of Applied Science & Engineering, and UTIAS PhD candidate Kevin Course introduced a new machine learning algorithm that surmounts the real-world challenge of imperfect knowledge about system dynamics. The computer-based mathematical modelling approach is used for problem solving and better decision making in complex systems, where many components interact with each other.

The researchers say the work could have numerous applications ranging from predicting the performance of aircraft engines to forecasting changes in global climate or the spread of viruses.

“For the first time, we are able to apply state estimation to problems where we don’t know the governing equations, or the governing equations have a lot of missing terms,” says Course, who is the paper’s first author.

“In contrast to standard techniques, which usually require a state estimate to infer the governing equations and vice-versa, our method learns the missing terms in the mathematical model and a state estimate simultaneously.”

State estimation, also known as data assimilation, refers to the process of combining observational data with computer models to estimate the current state of a system. Traditionally, it requires strong assumptions about the type of uncertainties that exist in a mathematical model.

“For example, let’s say you have constructed a computer model that predicts the weather and at the same time, you have access to real-time data from weather stations providing actual temperature readings,” says Nair. “Due to the model’s inherent limitations and simplifications – which is often unavoidable when dealing with complex real-world systems – the model predictions may not match the actual observed temperature you are seeing.

“State estimation combines the model’s prediction with the actual observations to provide a corrected or better-calibrated estimate of the current temperature. It effectively assimilates the data into the model to correct its state.”

However, it has been previously difficult to estimate the underlying state of complex dynamical systems in situations where the governing equations are completely or partially unknown. The new algorithm provides a rigorous statistical framework to address this long-standing problem.

“This problem is akin to deciphering the ‘laws’ that a system obeys without having explicit knowledge about them,” says Nair, whose research group is developing algorithms for mathematical modelling of systems and phenomena that are encountered in various areas of engineering and science.

A byproduct of Course and Nair’s algorithm is that it also helps to characterize missing terms or even the entirety of the governing equations, which determine how the values of unknown variables change when one or more of the known variables change.

The main innovation underpinning the work is a reparametrization trick for stochastic variational inference with Markov Gaussian processes that enables an approximate Bayesian approach to solve such problems. This new development allows researchers to deduce the equations that govern the dynamics of complex systems and arrive at a state estimate using indirect and “noisy” measurements.

“Our approach is computationally attractive since it leverages stochastic – that is randomly determined – approximations that can be efficiently computed in parallel and, in addition, it does not rely on computationally expensive forward solvers in training,” says Course.

While Course and Nair approached their research from a theoretical viewpoint, they were able to demonstrate practical impact by applying their algorithm to problems ranging from modelling fluid flow to predicting the motion of black holes.

“Our work is relevant to several branches of sciences, engineering and finance as researchers from these fields often interact with systems where first-principles models are difficult to construct or existing models are insufficient to explain system behaviour,” says Nair.

“We believe this work will open the door for practitioners in these fields to better intuit the systems they study,” adds Course. “Even in situations where high-fidelity mathematical models are available, this work can be used for probabilistic model calibration and to discover missing physics in existing models.

“We have also been able to successfully use our approach to efficiently train neural stochastic differential equations, which is a type of machine learning model that has shown promising performance for time-series datasets.”

While the paper primarily addresses challenges in state estimation and governing equation discovery, the researchers say it provides a general groundwork for robust data-driven techniques in computational science and engineering.

“As an example, our research group is currently using this framework to construct probabilistic reduced-order models of complex systems. We hope to expedite decision-making processes integral to the optimal design, operation and control of real-world systems,” says Nair.

“Additionally, we are also studying how the inference methods stemming from our research may offer deeper statistical insights into stochastic differential equation-based generative models that are now widely used in many artificial intelligence applications.”

Researchers help robots navigate crowded spaces with new visual perception method

Christopher.Sorensen — Wed, 09 Nov 2022 20:10:52 +0000

Researchers help robots navigate crowded spaces with new visual perception method

Christopher.Sorensen Wed, 11/09/2022 - 15:10

Cutline

Researchers from the ��TV Institute for Aerospace Studies have developed a system that improves how robots stitch together a set of images taken from a moving camera to build a 3D model of their environments (photo by iStock/LeoPatrizi)

Safa Jinje

Topic

Breaking Research

Alumni

Faculty of Applied Science & Engineering

Graduate Students

Research & Innovation

Robotics

UTIAS

A team of researchers at the University of Toronto has found a way to enhance the visual perception of robotic systems by coupling two different types of neural networks.

The innovation could help autonomous vehicles navigate busy streets or enable medical robots to work effectively in crowded hospital hallways.

“What tends to happen in our field is that when systems don’t perform as expected, the designers make the networks bigger – they add more parameters,” says Jonathan Kelly, an assistant professor at the University of Toronto Institute for Aerospace Studies in the Faculty of Applied Science & Engineering.

“What we’ve done instead is to carefully study how the pieces should fit together. Specifically, we investigated how two pieces of the motion estimation problem – accurate perception of depth and motion – can be joined together in a robust way.”

Researchers in Kelly’s Space and Terrestrial Autonomous Robotic Systems lab aim to build reliable systems that can help humans accomplish a variety of tasks. For example, they’ve designed an electric wheelchair that can automate some common tasks such as navigating through doorways.

More recently, they’ve focused on techniques that will help robots move out of the carefully controlled environments in which they are commonly used today and into the less predictable world humans are accustomed to navigating.

“Ultimately, we are looking to develop situational awareness for highly dynamic environments where people operate, whether it’s a crowded hospital hallway, a busy public square or a city street full of traffic and pedestrians,” says Kelly.

One challenging problem that robots must solve in all of these spaces is known to the robotics community as “structure from motion.” This is the process by which robots stitch together a set of images taken from a moving camera to build a 3D model of the environment they are in. The process is analogous to the way humans use their eyes to perceive the world around them.

In today’s robotic systems, structure from motion is typically achieved in two steps, each of which uses different information from a set of monocular images. One is depth perception, which tells the robot how far away the objects in its field of vision are. The other, known as egomotion, describes the 3D movement of the robot in relation to its environment.

“Any robot navigating within a space needs to know how far static and dynamic objects are in relation to itself, as well as how its motion changes a scene,” says Kelly. “For example, when a train moves along a track, a passenger looking out a window can observe that objects at a distance appear to move slowly, while objects nearby zoom past.”

The challenge is that in many current systems, depth estimation is separated from motion estimation – there is no explicit sharing of information between the two neural networks. Joining depth and motion estimation together ensures that each is consistent with the other.

“There are constraints on depth that are defined by motion, and there are constraints on motion that are defined by depth,” says Kelly. “If the system doesn’t couple these two neural network components, then the end result is an inaccurate estimate of where everything is in the world and where the robot is in relation.”

In a recent study, two of Kelly’s students – Brandon Wagstaff, a PhD candidate, and former PhD student Valentin Peretroukhin – investigated and improved on existing structure from motion methods.

Their new system makes the egomotion prediction a function of depth, increasing the system’s overall accuracy and reliability. They recently presented their work at the International Conference on Intelligent Robots and Systems (IROS) in Kyoto, Japan.

“Compared with existing learning-based methods, our new system was able to reduce the motion estimation error by approximately 50 per cent,” says Wagstaff.

“This improvement in motion estimation accuracy was demonstrated not only on data similar to that used to train the network, but also on significantly different forms of data, indicating that the proposed method was able to generalize across many different environments.”

Maintaining accuracy when operating within novel environments is challenging for neural networks. The team has since expanded their research beyond visual motion estimation to include inertial sensing – an extra sensor that is akin to the vestibular system in the human ear.

“We are now working on robotic applications that can mimic a human’s eyes and inner ears, which provides information about balance, motion and acceleration,” says Kelly.

“This will enable even more accurate motion estimation to handle situations like dramatic scene changes — such as an environment suddenly getting darker when a car enters a tunnel, or a camera failing when it looks directly into the sun.”

The potential applications for such new approaches are diverse, from improving the handling of self-driving vehicles to enabling aerial drones to fly safely through crowded environments to deliver goods or carry out environmental monitoring.

“We are not building machines that are left in cages,” says Kelly. “We want to design robust robots that can move safely around people and environments.”

��TV student team takes first place at International Small Wind Turbine Contest

Christopher.Sorensen — Mon, 04 Jul 2022 15:24:48 +0000

��TV student team takes first place at International Small Wind Turbine Contest

Christopher.Sorensen Mon, 07/04/2022 - 11:24

Cutline

The UTWind team stands next to their winning prototype turbine at the Open Jet Facility wind tunnel at Delft University of Technology (photo by Niels Adema/Hanze University of Applied Sciences)

Tyler Irving

Topic

Global Lens

Alumni

Faculty of Applied Science & Engineering

Global

Graduate Students

Sustainability

Undergraduate Students

UTIAS

In their first-ever competition, UTWind – a team of undergraduate and graduate students from the University of Toronto’s Faculty of Applied Science & Engineering – has taken the top prize in an international challenge to design and build a small-scale wind turbine.

“While we always strived to be a competitive team from the beginning and knew that we had a strong design, we definitely didn’t expect to win first place,” says David Petriw, a third-year materials science and engineering student who is a member of UTWind.

“The morale of the team is at an all-time high, and we are going to celebrate this win in a big way.”

The International Small Wind Turbine Contest (ISWTC) is hosted annually by Hanze University of Applied Sciences in Groningen, Netherlands. To clinch first place, UTWind edged out teams from Denmark, Germany, Poland and Egypt.

“The goal of ISWTC is to build and demonstrate a wind turbine designed for rural regions in Sub-Saharan Africa,” says Andrew Ilersich, a PhD candidate at the ��TV Institute for Aerospace Studies (UTIAS) and aerodynamics lead for UTWind.

“Every aspect of our design had to be tailored to, or at least compatible with, the region it would be sold and operated in. We also had to show that our design was sustainable, being made from recyclable, low-cost, and locally available materials.”

Unlike the large turbines used in commercial wind farms, which can rise to over 100 metres and generate megawatts of power each, small wind turbines (SWTs) are designed for generation on scales from a few hundred watts to a few kilowatts.

To win the contest, teams must demonstrate top-of-class performance across a number of criteria, including power generation, cut-in speed, estimated annual energy production and coefficient of power, which is a measure of the turbine’s efficiency.

Performance was measured at the Open Jet Facility wind tunnel at Delft University of Technology. After that, the teams headed to the Science of Making Torque Conference in Delft, to present their business case.

View this post on Instagram

A post shared by UTWind (@utwindclub)

The process of creating the prototype took more than a year from start to finish.

“We began the design phase in the beginning of 2021 and the whole assembly was built in winter semester 2022,” says Ashley Best, a third-year student in materials science and engineering who is media team lead for UTWind.

“Our turbine is made from wood and 3D-printed plastics. A few parts were outsourced to our sponsoring machine shop, Protocase, but the majority of the fabrication was done in house by our team – 3D printing, laser cutting, drill pressing, lathing, milling and assembly.”

“One of the things that set our team apart was our high coefficient of power, even when operating at very low wind speeds,” says Suraj Bansal, UTWind co-president and technical adviser and a PhD candidate at UTIAS.

“In addition, we had a very modular, low-cost and sustainable construction, as well as a self-starting wind-turbine design thanks to our active pitch control system. We are currently creating a mobile app to control and monitor the wind turbine performance right from our mobile devices.”

UTWind is one of ��TV Engineering’s newest design teams, co-founded in January 2021 by Bansal and UTIAS alumnus Ben Gibson.

“I was a member of a similar wind turbine design team at the University of Manitoba, while Suraj had prior experience from his master’s research work in the U.S. to design extreme-scale wind turbines,” Gibson says.

“We wanted to pass as much of that knowledge on as we could, while both having fun and pushing ourselves to the maximum. And so far, it has worked out great.”

��TV's aUToronto team wins first competition of AutoDrive Challenge sequel

Christopher.Sorensen — Tue, 14 Jun 2022 19:49:13 +0000

��TV's aUToronto team wins first competition of AutoDrive Challenge sequel

Christopher.Sorensen Tue, 06/14/2022 - 15:49

Cutline

The aUToronto team, made up mostly of ��TV undergraduate students, won the first phase of the AutoDrive Challenge II, which took place earlier this month in Ann Arbor, Mich. (photo courtesy aUToronto)

Tyler Irving

Topic

Our Community

Faculty of Applied Science & Engineering

Graduate Students

Self-Driving Cars

Undergraduate Students

UTIAS

A self-driving vehicle team from the University of Toronto’s Faculty of Applied Science & Engineering has taken the top spot overall in the first competition of the four-year AutoDrive Challenge II.

The achievement by aUToronto continues an impressive winning streak for the team, which consistently placed first throughout the original four-year AutoDrive Challenge.

In the current contest, the intercollegiate competition’s original concept has been expanded with more teams and more sophisticated tasks as participants develop and demonstrate an autonomous vehicle (AV) that can navigate urban driving courses.

“This year was a fresh new start for us,” says Frank (Chude) Qian, a master’s candidate at the ��TV Institute for Aerospace Studies (UTIAS) and team principal for aUToronto. “We have a very young team, with nearly 90 per cent of the students new to the competition.”

Approximately 85 per cent of these students are undergraduates from across ��TV Engineering’s departments and divisions. The remainder are graduate students or undergraduates from other parts of ��TV, including the department of computer science in the Faculty of Arts & Science.

Throughout the fall of 2021 and winter of 2022, the aUToronto team spent hours designing, training and testing their perception cart, pictured here at the ��TV Institute for Aerospace Studies. (photo courtesy of aUToronto)

A total of 10 institutions from across North America sent teams to AutoDrive Challenge II. They assembled earlier this month at Mcity in Ann Arbor, Mich., a unique purpose-built proving ground for testing the performance and safety of connected and automated vehicles.

“This is the first year of the second round of the AutoDrive competition, so the team was required to design, build and code everything from scratch,” says Steven Waslander, an associate professor at UTIAS who advises the team along with fellow faculty members Tim Barfoot, Jonathan Kelly and Angela Schoellig.

“It was a monumental effort and one that shows the true depth of talent and dedication of all the members of this amazing group of students.”

In the first phase of the four-year competition, the teams were using what are known as perception carts.

“We use these to validate the design of our perception system, which we will incorporate onto a real vehicle for next year’s competition,” says Qian. “Our brand-new sensor suite is based on a new solid-state LiDAR modality.”

LiDAR is a sensing technology that works in a similar way to radar, except that it uses laser light instead of sound. It is a key component of the sensor suite – which also includes traditional radar and visual cameras similar to those found in smartphones – that enables a self-driving vehicle to build up a 3D representation of its surroundings.

Members of the aUToronto team with their winning perception cart at MCity in Ann Arbor, Mich (photo courtesy of aUToronto)

In addition to being declared the overall winner, the ��TV Engineering team received top marks in a wide range of categories, including the concept design event, the traffic light challenge and mobility innovation.

One of the most dramatic parts of the competition was the dynamic object detection challenge, during which the cart had to detect and avoid a mannequin of a deer.

“In the final testing session, the team realized their segmentation code was failing due to a change to the deer mannequin being used,” says Waslander.

“Having planned for just such an event, they immediately switched to high gear. They took and labelled over 2,000 images of the new deer, then spent the whole night training and tweaking a brand-new detector. It worked brilliantly in competition the next day, securing first place.”

Qian says he is very proud of all that the team has accomplished.

“This year with so many newer members, training became very important,” he says. “We also lost some precious development time due to challenges associated with COVID-19. I cannot give enough credit to the aUToronto flight team members who really took one for the team and pulled through together.”

Researchers design 'socially aware' robots that can anticipate – and safely avoid – people on the move

Christopher.Sorensen — Tue, 17 May 2022 16:54:39 +0000

Researchers design 'socially aware' robots that can anticipate – and safely avoid – people on the move

Christopher.Sorensen Tue, 05/17/2022 - 12:54

Cutline

Hugues Thomas and his collaborators at the ��TV Institute for Aerospace Studies created a new method for robot navigation based on self-supervised deep learning (photo by Safa Jinje)

Safa Jinje

Topic

Our Community

Artificial Intelligence

Faculty of Applied Science & Engineering

machine learning

Research & Innovation

Robotics

UTIAS

A team of researchers led by University of Toronto Professor Tim Barfoot is using a new strategy that allows robots to avoid colliding with people by predicting the future locations of dynamic obstacles in their path.

The project, which is supported by Apple Machine Learning, will be presented at the International Conference on Robotics and Automation in Philadelphia at the end of May.

The results from a simulation, which are not yet peer-reviewed, are available on the arXiv preprint service.

“The principle of our work is to have a robot predict what people are going to do in the immediate future,” says Hugues Thomas, a post-doctoral researcher in Barfoot’s lab at the ��TV Institute for Aerospace Studies in Faculty of Applied Science & Engineering. “This allows the robot to anticipate the movement of people it encounters rather than react once confronted with those obstacles.”

To decide where to move, the robot makes use of Spatiotemporal Occupancy Grid Maps (SOGM). These are 3D grid maps maintained in the robot’s processor, with each 2D grid cell containing predicted information about the activity in that space at a specific time. The robot choses its future actions by processing these maps through existing trajectory-planning algorithms.

Another key tool used by the team is light detection and ranging (lidar), a remote sensing technology similar to radar except that it uses light instead of sound. Each ping of the lidar creates a point stored in the robot’s memory. Previous work by the team has focused on labeling these points based on their dynamic properties. This helps the robot recognize different types of objects within its surroundings.

The team’s SOGM network is currently able to recognize four lidar point categories: the ground; permanent fixtures, such as walls; things that are moveable but motionless, such as chairs and tables; and dynamic obstacles, such as people. No human labelling of the data is needed.

“With this work, we hope to enable robots to navigate through crowded indoor spaces in a more socially aware manner,” says Barfoot. “By predicting where people and other objects will go, we can plan paths that anticipate what dynamic elements will do.”

In the paper, the team reports successful results from the algorithm carried out in simulation. The next challenge is to show similar performance in real-world settings, where human actions can be difficult to predict. As part of this effort, the team has tested their design on the first floor of ��TV’s Myhal Centre for Engineering Innovation & Entrepreneurship, where the robot was able to move past busy students.

“When we do experiment in simulation, we have agents that are encoded to a certain behaviour and they will go to a certain point by following the best trajectory to get there,” says Thomas. “But that’s not what people do in real life.”

When people move through spaces, they may hurry or stop abruptly to talk to someone else or turn in a completely different direction. To deal with this kind of behaviour, the network employs a machine learning technique known as self-supervised learning.

Self-supervised learning contrasts with other machine-learning techniques, such as reinforced learning, where the algorithm learns to perform a task by maximizing a notion of reward in a trial-and-error manner. While this approach works well for some tasks – for example, a computer learning to play a game such as chess or Go – it is not ideal for this type of navigation.

“With reinforcement learning, you create a black box that makes it difficult to understand the connection between the input – what the robot sees – and the output, or the robot does,” says Thomas. “It would also require the robot to fail many times before it learns the right calls, and we didn’t want our robot to learn by crashing into people.”

By contrast, self-supervised learning is simple and comprehensible, meaning that it’s easier to see how the robot is making its decisions. This approach is also point-centric rather than object-centric, which means the network has a closer interpretation of the raw sensor data, allowing for multimodal predictions.

“Many traditional methods detect people as individual objects and create trajectories for them. But since our model is point-centric, our algorithm does not quantify people as individual objects, but recognizes areas where people should be. And if you have a larger group of people, the area gets bigger,” says Thomas.

“This research offers a promising direction that could have positive implications in areas such as autonomous driving and robot delivery, where an environment is not entirely predictable.”

In the future, the team wants to see if they can scale up their network to learn more subtle cues from dynamic elements in a scene.

“This will take a lot more training data,” says Barfoot. “But it should be possible because we’ve set ourselves up to generate the data in more automatic way: where the robot can gather more data itself while navigating, train better predictive models when not in operation and then use these the next time it navigates a space.”

UTIAS

����TV team takes top spot in self-driving car challenge for 6th time in 7 years

Start@UTIAS

Tags

����TV researchers enhance object-tracking abilities of self-driving cars

With the launch of its first satellite, student team charts a course to new knowledge

����TV researchers partner with Siemens Energy to tackle sustainable energy production

Learn more about industry partnerships at ����TV

AI algorithm improves predictive models of complex dynamical systems

Researchers help robots navigate crowded spaces with new visual perception method

����TV student team takes first place at International Small Wind Turbine Contest

����TV's aUToronto team wins first competition of AutoDrive Challenge sequel

Researchers design 'socially aware' robots that can anticipate – and safely avoid – people on the move

��TV team takes top spot in self-driving car challenge for 6th time in 7 years

��TV researchers enhance object-tracking abilities of self-driving cars

��TV researchers partner with Siemens Energy to tackle sustainable energy production

Learn more about industry partnerships at ��TV

��TV student team takes first place at International Small Wind Turbine Contest

��TV's aUToronto team wins first competition of AutoDrive Challenge sequel