PROJECT TITLE: Improving GRAsping Movements by predictions based on Observation (IGRAMO)
UKF GRANT - DURATION OF THE PROJECT (months): 24 months
NAME OF THE PROJECT AND CO - APPLICANT: Danica Kragić, Bojan Jerbić

INVOLVED ORGANISATIONS:
Faculty of Mechanical Engineering and Naval Architecture,
Department of Robotics and Production System Automation
Ivana Lučića 5, 10000 Zagreb, Croatia
Contact person: Prof. dr. sc. Bojan Jerbić
Ova e-mail adresa je zaštićena od spam robota, nije vidljiva ako ste isključili JavaScript ; tel: +385 1 61 68 356

Kungliga Tekniska Hogskolan (KTH, Royal Institute of Technology),
Valhallavagen 79, 10044 Stockholm, Sweden
Contact person: Prof. dr. Danica Kragic
CSC-CVAP/CAS, KTH, 10044 Stockholm, Sweden
Ova e-mail adresa je zaštićena od spam robota, nije vidljiva ako ste isključili JavaScript ; tel: +468 790 6729

AMOUNT: 300.000 HRK (Private sector) + 458.934 HRK (FOND)

SUMMARY:
For the future, we want to build robots that can in and easy and flexible way learn how to solve tasks in unknown environments. Traditional automation approaches are cost ineffective, inflexible and unreliable, because the world is dynamic and unpredictable. However, inexperienced, ordinary users cannot use these classical approaches. Recent scientific achievements in cognitive science and robotics enable modeling of different, nondeterministic approaches to automation of production systems. It is expected from such approaches to provide adaptability and high level of intelligence and autonomy enabling the machines to work in unstructured environment, learn and improve and cooperate with other agents including humans. The adaptability and autonomy imply hardware and software evolvable reconfigurability bringing up an issue of a higher level of system control and wider abstraction aspects at the autonomous learning level. In humans, learning involves changes in behavior that arise from interaction with the environment. The principles and the inspiration for this project are adopted from the human psychology and physiology. This project concentrates on grasping and manipulation of known and new objects that is currently recognized as one of the most important open problems in the field of robotics. How to provide suitable models and develop flexible and safe systems is a topic of several ongoing EU projects (CoSy, PACO-PLUS, GRASP, DexSmart, CogX). Grasping is motor skill based on perception, interaction with environment and neural/control machinery. Although much of our motor repertoire is acquired during our lifetime, we do not start life as tabula rasa. Evolutionary processes drive an innate pattern of behavior to hardwire motor skills into the brain before birth and support subsequent learning. Motor learning is a consequence of the co-adaptation of the neural machinery and structural anatomy. In the project, we will use human movement data to derive the „innate“ motor primitives and used these to build complex robot-hand movements. Here we will explore the need for motor learning, what is learned, how it is represented, and the mechanisms of learning. The perception system will interpret the environment and the objects, mapping the knowledge about the world with the set of primitive motor behaviors the agent can initially execute and from which all the more complex behaviors derive. The evaluation of the proposed methodology will be evaluated both on actual physical systems and in simulation.

PROJECT TEAM:
Prof. dr.sc. Danica Kragić (project leader)
Prof. dr.sc. Bojan Jerbić (co-applicant)
Petar Ćurković, dipl. ing.
Tomislav Stipančić dipl. ing.
Dr Patric Jensfelt
Javier Martinez
Denis Bašić

PUBLICATIONS:
- Getsure database

ACTIVITIES:
- Organisation of the International Summer School on Robotics - www.roboschool.fsb.hr

RESULTS:
Month 1-6:

The objectives of the project for the first year are focused on the experimental system setup. This involves acquiring, mounting, connecting of the manipulators. At the University of Zagreb, three FANUC robots are installed and fully operational at this point. Necessary equipment including two-finger grippers, sensors, wiring etc. is designed, wired and fully operational. Cameras for image acquisition are mounted and basic image acquisition and processing are performed.

New control  paradigm based on Multi-Agent System concept for the robot-hand multi-finger coordination in grasping and moving the manipulation object is developed. We model fingers and global-view camera as independent rational collaborative autonomous agents working on their movement coordination and motion planning in a discrete state space. The fingers  through different coordination and collaboration techniques (e.g. coordination graphs, negotiation and conflict solution methods), collaborate with the rest of the fingers in the gripper and with the global-view camera in the realization of the gripper’s task: bringing the convex manipulation object from some initial to some target position by surrounding it, grasping it, and transporting it in a coordinated and collaborative manner, and in the same time avoiding the obstacles that come in a way. The improvement of efficiency of the gripper’s task execution is noticed.  This MAS paradigm promises certain benefits to implementation not only to the gripper fingers but also to the whole manufacturing system made of robots and manipulators. Each robot can be considered as one agent in interaction with the other agents (robots, manipulators, AGV, FMS etc.) and at the same time as a Multi-Agent System where each its active component is seen as an agent interacting with the other agents in that specific Robot.
To further improve the efficiency and security of the two manipulators moving in the workspace, partially autonomously, presenting dynamic obstacles to each other, an optimization control frame based on co-evolutionary approach is proposed. Each manipulator (finger, or manipulator as whole) is considered autonomous and controlled by independent control algorithm based on evolutionary algorithm. To optimize concurrently in terms of number of collisions, velocity profile, rotation angle, we co-evolve each agent, as to enable concurrent optimization of conflicting parameters. This way, complex motion problems are decomposed to parallel processing units and better efficiency is achieved. Additional effort is needed to implement the framework to the real robots, since no physical characteristic of the robots are included in the model in this phase.

To further improve and customize the vision tools for extracting of the data from vision systems, several vision platforms were considered. At the University of Zagreb, Fanuc, DVT, Basler – NI and Point Gray platforms were considered, whereas at the KTH, open-source vision library was researched. At the University of Zagreb, the main accent was given to determine vision platforms characteristics with respect to the initial experimental system setup. Such an approach provides a better look to platforms integration capabilities and identifies possible vision applications. During the research time, Fanuc and DVT are used for 2D monochromatic vision, Basler – NI is used for 2D color vision and Point Gray is used for 3D vision. At the KTH, special effort was focused in research of OpenCV Vision library. This open-source library is very interesting for the project because it is powerful, free, based on c++, relatively easy to use and there is a whole Internet community ready to help if it is needed. Source code availability enables free library development and better vision application understanding overall. This vision library is tending to be used for custom vision applications which are related to the project.  

Regarding the work on activity modeling and recognition, we have developed statistical models for single hand activities. In addition, we have concentrated on automatic hand tracking without the use of any markers. We have developed a vision based method for estimation of the pose of human hands in interaction with objects. Despite the fact that most robotics applications of human hand tracking involve grasping and manipulation of objects, the majority of methods in the literature assume a free hand, isolated from the surrounding environment. Grasping hand reconstruction is a more challenging problem than the reconstruction of hands in isolation, since the object often occludes large parts of the hand. Our hand tracking method is non-parametric, performing a nearest neighbor search in a large database (100000 entries) of hand poses with and without grasped objects. The system operates in real time, robust to self occlusions, object occlusions and segmentation errors, and provides full hand pose reconstruction from markerless video. Temporal consistency in hand pose is taken into account, without explicitly tracking the hand in the high dimensional pose space.

http://www.csc.kth.se/~danik/gesture_database/

Month 7-12:

During the second period of the project and related to activity recognition, a database of magnetic data has been put on the web:
http://www.csc.kth.se/~danik/gesture_database.
Preliminary work on data modeling and recognition has been performed. The work on vision based hand posture recognition has also been continued and more research has been done in terms of spatio-temporal processing.

LECTURES:
- RISS2010 materials