control and planning algorithms for UAVs, whose execution is interfaced with a human operator.
An autonomous robot is a robot that is capable to act and perform its task without any supervision from humans or other agents. What are the capabilitis that the robot needs to be autonomous depends on the task it must perform and the context. In general, the most restricted and structured the environment, the easier it is to achieve autonomy. However, achieving autonomy at large, for complex tasks and in very dynamic and unpredictable environments is extremely challenging. For this reason, this endeavour come in incremental steps, commonly called levels of autonomy. The levels of autonomy were defined by Sheridan and Verplanck (1978) in the context of human-computer interactions. These levels mark a scale in which at one extreme the human executes the task and at the other extreme the computer does the job alone. This concepts translates to robotics, where at the lowest level of autonomy a robot is directly maneuvered by a human who acts as a pilot (direct control), whereas at the highest level of autonomy the robot is fully autonomous and the human operator may only be present for supervision (supervisory control). In between these two extremes, lies the shared control paradigm, in which the intelligence governing the execution of the task comes from a co-participation of a human and some automatic algorithm. Shared control is a promising compromise to enable operation of robots in situations that are still far too complex for full autonomy. One such instance is the application to UAVs in unrestricted environments, where besides the complexity of the system there are may also be mandates from the regulator that impose the presence of a human teleoperator to either pilot the robot or supervise it being ready to take over control in critical situations. At the Max Planck Institute for Biological Cybernetics I researched shared control solutions for several tasks with UAVs. In particular, I developed various blending functions to combine the different control inputs provided by the two systems –human operator and automatic control algorithm– so that they do not interfere with each other.
Figure 1 Different control paradigms for various levels of autonomy. Shared control lies between direct control, where the human manualy steers the robot, and supervisory control, where the robot autonomously executes the task while only receiving high-level directives from the operator. Image from my PhD dissertation, titled Planning and control for robotic tasks with a human-in-the-loop.
In this research I investigated the problem f developing a shared control solution that would allow a single user to steer a formation of UAVs defined in terms of their relative bearing angles. The goal of this solution is to make the interface for the user easy, while guaranteeing asymptotic stability of the formation. The proposed solution was developed on the idea of splitting the task int two orthogonal components:
Control the relative bearing angles among the UAVs to achieve and maintain the desired formation.
Control the motions of the UAVs that are in the orthogonal space to the formation, i.e. that do not alter the relaive bearing angles.
Based on this intuition, the proposed solutio npartitions the commands to the UAVs such that the automatic formation control algorithms regulates the relative bearing angles, whereas the human operator steers the free motions that do not alter the formation. The two orthogonal control inputs can thus be added together to provide the overal control input. Figure 4-left shows what are the motions that are orthogonal to the bearing formation and that are controlled by the human operator. Figure 4-right shows the experimental setup with a 3-UAVs formation.
I have also applied the paradigm of shared control to the case study of a formation of UAVs transporting a payload suspended by cables (see Fig. 4). In this case study the operator is in charge of steering the payload, without having to worry about controlling the individual UAVs. The shared control algorithm implements filtering techniques to generate a smooth reference trajectory for the system and to ensure that the formation does not reach a singular configuration. A geometric decomposition of the motions of the formation brings to light a series of internal motions (motions that do not change the formation) that are used for auxiliary tasks controlled either by the human (formation shrinking/expanding) or by an algorithm (formation control). The loop with the user is closed via a suitable force feedback.
A shared control framework as the one described above effectively reduces the burden required to the human operator, who does not need to control that the formation is maintained but must only steer the collective group of robots. However, if the operator must still provide commands to the robots at all times, because without his/her input the robot won’t move. The idea of shared planning is to move the action of the person to a higher level in the navigation stack, placing him/her in charge of modifying a planned trajectory for the robot, rather than controlling the movement of the robot itself. This way, the robot will still move according the the current planned trajectory even when the operator provides no inputs.
In the spirit of sharing the planning effort, much like in the shared control, part of the trajectory modification is deputed to an automatic algorithm that is in charge of guaranteeing that the planned path has some properties, i.e. that it does not cross obstacles, that it has some geometric regularity and that it automatically passes through points of interest. This is achieved using artificial forces that are applied on the path, which is modeled as a B-spline. However, online modifications of the planend path may cause it to move away from the robot and result in a discontinuity in the tracking error. Thus, the proposed blending function is implemented via a null-space projection operator that imposes the invariance of the local geometric properties of the path at the current location of the robot regardless of the global changes brought by human and automatic inputs.
Figure 6. Left: blending function based on the null-space projection, which preserves the local geometric properties of the path at the location of the robot. Right: example of the effect of the obstacle avoidance on the path. Images from the PhD dissertation titled Planning and control for robotic tasks with a human-in-the-loop.
The shared planning framework was demonstrated with physical experiments, as shown in Fig. 7.
This paper presents a novel bilateral shared framework for a cooperative aerial transportation and manipulation system composed by a team of micro aerial vehicles with a cable-suspended payload. The human operator is in charge of steering the payload and he/she can also change online the desired shape of the formation of robots. At the same time, an obstacle avoidance algorithm is in charge of avoiding collisions with the static environment. The signals from the user and from the obstacle avoidance are blended together in the trajectory generation module, by means of a tracking controller and a filter called dynamic input boundary (DIB). The DIB filters out the directions of motions that would bring the system too close to singularities, according to a suitable metric. The loop with the user is finally closed with a force feedback that is informative of the mismatch between the operator’s commands and the trajectory of the payload. This feedback intuitively increases the user’s awareness of obstacles or configurations of the system that are close to singularities. The proposed framework is validated by means of realistic hardware-in-the-loop simulations with a person operating the system via a force-feedback haptic interface.
@article{Masone-2021-sharedtransport,author={Masone, C. and Stegagno, P.},journal={Journal of Intelligent & Robotic Systems},title={Shared Control of an Aerial Cooperative Transportation System with a Cable-suspended Payload},year={2021},volume={103},number={40},pages={},doi={10.1007/s10846-021-01457-4},keywords={shared control, aerial robotics, aerial cable robot},}
2018
Journal
Shared planning and control for mobile robots with integral haptic feedback
C. Masone, M. Mohammadi, P. Robuffo Giordano, and A. Franchi
The International Journal of Robotics Research, 2018
This paper presents a novel bilateral shared framework for online trajectory generation for mobile robots. The robot navigates along a dynamic path, represented as a B-spline, whose parameters are jointly controlled by a human supervisor and an autonomous algorithm. The human steers the reference (ideal) path by acting on the path parameters that are also affected, at the same time, by the autonomous algorithm to ensure: (i) collision avoidance, (ii) path regularity, and (iii) proximity to some points of interest. These goals are achieved by combining a gradient descent-like control action with an automatic algorithm that re-initializes the traveled path (replanning) in cluttered environments to mitigate the effects of local minima. The control actions of both the human and the autonomous algorithm are fused via a filter that preserves a set of local geometrical properties of the path to ease the tracking task of the mobile robot. The bilateral component of the interaction is implemented via a force feedback that accounts for both human and autonomous control actions along the whole path, thus providing information about the mismatch between the reference and traveled path in an integral sense. The proposed framework is validated by means of realistic simulations and actual experiments deploying a quadrotor unmanned aerial vehicle (UAV) supervised by a human operator acting via a force-feedback haptic interface. Finally, a user study is presented to validate the effectiveness of the proposed framework and the usefulness of the provided force cues.
@article{Masone-2018-sharedplanning,author={Masone, C. and Mohammadi, M. and {Robuffo Giordano}, P. and Franchi, A.},title={Shared planning and control for mobile robots with integral haptic feedback},journal={The International Journal of Robotics Research},volume={37},number={11},pages={1395-1420},year={2018},doi={10.1177/0278364918802006},keywords={shared control, aerial robotics},}
2014
Conference
Semi-autonomous trajectory generation for mobile robots with integral haptic shared control
C. Masone, P. Robuffo Giordano, H. H. Bülthoff, and A. Franchi
In IEEE International Conference on Robotics and Automation (ICRA), 2014
A new framework for semi-autonomous path planning for mobile robots that extends the classical paradigm of bilateral shared control is presented. The path is represented as a B-spline and the human operator can modify its shape by controlling the motion of a finite number of control points. An autonomous algorithm corrects in real time the human directives in order to facilitate path tracking for the mobile robot and ensures i) collision avoidance, ii) path regularity, and iii) attraction to nearby points of interest. A haptic feedback algorithm processes both human’s and autonomous control terms, and their integrals, to provide an information of the mismatch between the path specified by the operator and the one corrected by the autonomous algorithm. The framework is validated with extensive experiments using a quadrotor UAV and a human in the loop with two haptic interfaces.
@inproceedings{Masone-2014-trajectory,author={Masone, C. and {Robuffo Giordano}, P. and Bülthoff, H. H. and Franchi, A.},booktitle={IEEE International Conference on Robotics and Automation (ICRA)},title={Semi-autonomous trajectory generation for mobile robots with integral haptic shared control},year={2014},volume={},number={},pages={6468-6475},doi={10.1109/ICRA.2014.6907814},keywords={shared control, aerial robotics},}
2012
Journal
Modeling and Control of UAV Bearing Formations with Bilateral High-level Steering
A. Franchi, C. Masone, V. Grabe, M. Ryll, H. H. Bülthoff, and P. Robuffo Giordano
The International Journal of Robotics Research, 2012
In this paper we address the problem of controlling the motion of a group of unmanned aerial vehicles (UAVs) bound to keep a formation defined in terms of only relative angles (i.e. a bearing formation). This problem can naturally arise within the context of several multi-robot applications such as, e.g. exploration, coverage, and surveillance. First, we introduce and thoroughly analyze the concept and properties of bearing formations, and provide a class of minimally linear sets of bearings sufficient to uniquely define such formations. We then propose a bearing-only formation controller requiring only bearing measurements, converging almost globally, and maintaining bounded inter-agent distances despite the lack of direct metric information.The controller still leaves the possibility of imposing group motions tangent to the current bearing formation. These can be either autonomously chosen by the robots because of any additional task (e.g. exploration), or exploited by an assisting human co-operator. For this latter ’human-in-the-loop’ case, we propose a multi-master/multi-slave bilateral shared control system providing the co-operator with some suitable force cues informative of the UAV performance. The proposed theoretical framework is extensively validated by means of simulations and experiments with quadrotor UAVs equipped with onboard cameras. Practical limitations, e.g. limited field-of-view, are also considered.
@article{Franchi-2012-bearing,author={Franchi, A. and Masone, C. and Grabe, V. and Ryll, M. and Bülthoff, H. H. and {Robuffo Giordano}, P.},title={Modeling and Control of {UAV} Bearing Formations with Bilateral High-level Steering},journal={The International Journal of Robotics Research},volume={31},number={12},pages={1504-1525},year={2012},doi={10.1177/0278364912462493},keywords={shared control, aerial robotics},}
Conference
Interactive planning of persistent trajectories for human-assisted navigation of mobile robots
C. Masone, A. Franchi, H. H. Bülthoff, and P. Robuffo Giordano
In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012
This work extends the framework of bilateral shared control of mobile robots with the aim of increasing the robot autonomy and decreasing the operator commitment. We consider persistent autonomous behaviors where a cyclic motion must be executed by the robot. The human operator is in charge of modifying online some geometric properties of the desired path. This is then autonomously processed by the robot in order to produce an actual path guaranteeing: i) tracking feasibility, ii) collision avoidance with obstacles, iii) closeness to the desired path set by the human operator, and iv) proximity to some points of interest. A force feedback is implemented to inform the human operator of the global deformation of the path rather than using the classical mismatch between desired and executed motion commands. Physically-based simulations, with human/hardware-in-the-loop and a quadrotor UAV as robotic platform, demonstrate the feasibility of the method.
@inproceedings{Masone-2012-interactive,author={Masone, C. and Franchi, A. and Bülthoff, H. H. and {Robuffo Giordano}, P.},booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},title={Interactive planning of persistent trajectories for human-assisted navigation of mobile robots},year={2012},volume={},number={},pages={2641-2648},doi={10.1109/IROS.2012.6386171},keywords={shared control, aerial robotics}}
2011
Conference
Bilateral teleoperation of multiple UAVs with decentralized bearing-only formation control
A. Franchi, C. Masone, H. H. Bülthoff, and P. Robuffo Giordano
In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2011
We present a decentralized system for the bilateral teleoperation of groups of UAVs which only relies on relative bearing measurements, i.e., without the need of distance information or global localization. The properties of a 3D bearing-formation are analyzed, and a minimal set of bearings needed for its definition is provided. We also design a novel decentralized formation control almost globally convergent and able to maintain bounded and non-vanishing inter-distances among the agents despite the absence of direct distance measurements. Furthermore, we develop a multi-master/ multi-slave teleoperation setup in order to control the overall behavior of the group and to convey to the human operator suitable force cues, while ensuring stability in presence of delays and packet losses over the master-slave communication channel. The theoretical framework is validated by means of extensive human/hardware in-the-loop simulations using two force-feedback devices and a group of quadrotors.
@inproceedings{Franchi-2011-bearing,author={Franchi, A. and Masone, C. and Bülthoff, H. H. and {Robuffo Giordano}, P.},booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},title={Bilateral teleoperation of multiple {UAVs} with decentralized bearing-only formation control},year={2011},volume={},number={},pages={2215-2222},doi={10.1109/IROS.2011.6094525},keywords={shared control, aerial robotics},}
