Planning robotic agent actions using semantic knowledge for a home environment

2.1. Semantic maps

The daily challenges have drive the research for automated and autonomous solutions to enable mobile robots to operate in highly dynamic environments. For this purpose, mobile robots need to create and maintain an internal representation of their environment, commonly referred to as a map. Robotic systems rely on different types of maps depending on their goals. Different map typology’s have been developed such as metric and topological maps, which are generally 2D representations of the environment^[16], or hybrids (a combination of the previous two)^[17,18]. There are also maps with 3D representation (sparse map, semi-dense map, and dense map). Metric and topological maps only contain spatial information^[19]. A fundamental requirement for the successful construction of maps is to deal with uncertainty arising, from errors in robot perception (limited field of view and sensor range, noisy measurements, etc.), from inaccurate models and algorithms, etc.

To get around this limitation, semantic maps were developed to add additional information, such as instances, categories, and attributes of various constituent elements of the environment (objects, rooms, etc.)^[5–7]. These provide robots with the ability to understand beyond the spatial aspects of the environment, the meaning of each element, and how humans interact with them (features, events, relationships, etc.). Semantic maps deal with meta information that models the properties and relationships of relevant concepts in the domain in question, encoded in a knowledge base (KB).

2.2. Ontologies for semantic maps

One of the tasks to be solved in mobile robot navigation is the acquisition of information from the environment. In the field of semantic navigation, information includes concepts such as objects, utilities, or room types. The robot needs to learn the relationships that exist between the concepts included in the knowledge representation model. Semantic maps add to classical robotic maps spatially grounded object instances anchored in a suitable way for knowledge representation and reasoning. The classification of instances through the analysis of the data collected by sensors is one of the biggest challenges in the creation of semantic maps (i.e., to give a richer semantic meaning to the sensor data)^[20,21].

In the last decade, several papers have appeared in the literature contributing different representations of semantic maps. Kostavelis et al.^[5] summarized the significant progress made on a broad range of mapping approaches and applications for semantic maps, including task planning, localization, navigation, and human– robot interaction. Semantics has been used in a diverse range of applications. Lim et al.^[22] presented an approach for unified robot knowledge for service robots in indoor environments. Rusu et al.^[23] developed a map called Semantic Object Maps (SOM), which encodes spatial information about indoor household environments, in particular kitchens, but in addition it also enriches the information content with encyclopedic and common sense knowledge about objects, as well as includes knowledge derived from observations. Galindo et al.^[24] proposed an approach for robotic agents to correct situations in the world that do not conform to the semantic model by generating appropriate goals for the robot. In short, it combines the use of a semantic map with planning techniques in Planning Domain Description Language (PDDL) that converts the goals into actions (moving the robot, picking and dropping an object, etc.). Wang et al.^[25] employed the relationships among objects to represent the spatial layout. Object recognition and region inference are implemented by using stereo image data. Vasudevan et al.^[26] created a hierarchical probabilistic concept-oriented representation of space, based on objects. Diab M et al.^[27] introduced an interpretation ontology to identify possible failures which occur during automatic planning and the execution phase; this ontology aims to improve planning and allow automatic replanning after error. Balakirsky et al.^[28] proposed an ontology-based framework that allows a robotic system to automatically recognize and adapt to changes that occur in its workflow and dynamically change the details of task assignment, increasing process flexibility by allowing plans to adapt to production errors and task changes.

In short, semantic maps enable a robot to solve reasoning problems of geometric, topological, ontological, and logical nature, in addition to localization and path planning^[29]. Formal conceptualization of the robotics domain is an essential requirement for the future of robotics, in order to be able to design robots that can autonomously perform a wide variety of tasks in a wide variety of environments.

2.3. Applications in robotic systems

Different research groups have used semantic knowledge in the area of robotics. Semantic knowledge allows a clear dialog between all stakeholders involved in the life cycle of a robotic system and enables the efficient integration and communication of heterogeneous robotic systems. These facilitate communication and knowledge exchange between groups from different fields, without actually forcing them to align their research with the particular view of a particular research group^[30].

One of the most recent advances in the field of robotics can be denoted by analyzing the KnowRob project, where researchers aimed to enable a robot to answer different types of questions about possible interactions with its environment, using semantic knowledge^[31,32]. For example, they developed an ontology that allows the robot to start an assembly activity, with incomplete knowledge. It identifies the missing parts, having the ability to reason about how the missing information can be obtained^[32]. KnowRob employs the DUL foundational ontology, which is a slim version of the Descriptive Ontology for Linguistic and Cognitive Engineering (DOLCE). DUL and DOLCE have a clear cognitive bias, and they are both well established in the knowledge engineering community as foundational ontologies. However, DUL does not define very specific concepts such as fork or dish. These concepts are needed for our robots that do everyday activities^[8]. There are also other relevant works that aim at the standardization of knowledge representation in the robotics domain, such as IEEE-ORA^[33], ROSETTA^[34], CARESSES^[35], RoboEarth^[36], RoboBrain^[37], RehabRobo-Onto^[38], and OROSU^[39].

All of the above work already represents promising advances in the use of semantics in robotic systems; however, it lacks the ability to perform advanced reasoning and relies heavily on ad hoc reasoning solutions, significantly limiting its scope. A general standardized framework for working with ontologies is needed, natively supporting symbolic logic and advanced reasoning paradigms.

The next sections present the reasoning frameworks, with particular emphasis on the domain specific ontology, home environment. The proposed ontology is designed to be easily reusable in different environments of a house, as well as by different robotic agents.

3.1. Knowledge engine

Figure 1 depicts the global knowledge engine conceptual framework, designed to achieve the main objective of the paper: the integration of a domain specific home environment ontology, with a task planner (ROSPlan), transforming the goals coming from the reasoning into executable actions by the robotic agent. The framework have three main parts: reasoning (which includes the ontologies), planning, and the robot. These parts are presented in the remainder of the paper.

Figure 1. Global System, containing a Reasoning section that is based on an ontology, a Planning section that is based on the Planning Domain Definition Language (PDDL), a relational database (MongoDB) that is queried using the ontology and ROSPlan, and a Robot section that is based on robot controller, as well as its sensing and acting devices.

A domain specific home environment ontology, aligned with a MongoDB database, encapsulate the important concepts of the domain to be considered (space of a house, objects, etc.). Indeed, the main benefit of a domain ontology is to set standard definitions of shared concepts identified in the requirement phase and to define appropriate relations between the concepts and their properties^[40]. The ontology contain concepts of Core Ontology for Robotics and Automation (CORA), with the representation of fundamental concepts of robotics and automation^[41].

The ontology model is based on the concepts and relationships between different entities, and then aligned with the MongoDB database. The basic concept of the reasoning process is based on the premises that: a relational database contains both the entities of the conceptual hierarchy and the instances of the physical hierarchy, this information is stored in lists, and these lists are related to each other, as in the entity–relationship model of the environment^[42,43].

The domain specific ontology home environment is defined with the Protégé software. Protégé version 5.5.0 was used^[44]. The the domain ontology was verified through version 1.4.3 of HermiT Reasoner to ensure that it is free of inconsistencies^[45]. Protégé is a free, open-source editor for developing the ontologies produced by Stanford university. It is a java-based application (multi-platform), with plugins such as onto Viz to visualize the ontologies. The backbone of protégé is that it supports the tool builders, domain specialists, and knowledge engineers.

Domain specific ontology home environment

Figure 2 shows the main relationships of the developed domain-specific home environment ontology, which was designed for an agent to interpret and interact with its surrounding environment. In this case, the environment is a house, with special focus on the internal environment.

Figure 2. Snapshot of the main ontology (relations is-a).

The developed model is subdivided into three main classes: Home_lab, Information, and Objects. The class Home_lab is subdivided into class Environment, which is subdivided into internal and external environment. The Indoor_environment is further subdivided into Hall, Corridor, and Rooms. The class Rooms is subdivided into the possible rooms types of an house (bathroom, bedroom, kitchen, etc.). The class Information is subdivided into Information_Object. The Objects class is subdivided into Types_Devices that are subdivided into Sensors and Actuators. The main class Objects is further subdivided into internal and external objects that contemplate the objects that can be found in a home environment. Finally, the Objects class is subdivided into PhysicalAgent, which is subdivided into the agents that can appear in the environment as human or robot, the latter further subdivided into the different types of robots.

Figure 2 presents the hierarchical class where the main concepts defined in the ontology are visible as:

• Environment: The surroundings or conditions in which an agent, person, animal, or plant lives or operates.

• Indoor environment: Environment situated inside of a house or other building.

• Corridor: A long passage in a building from which doors lead into rooms.

• Hall: The room or space just inside the front entrance of a house or flat.

• Rooms: Space that can be occupied or where something can be done (kitchen, bedroom, etc).

• Outdoor Objects: Used to describe objects that exist or appear outside a home.

• Objects: Any physical, social, or mental object, or a substance. Following DOLCE, objects are always participating in some event (at least their own life), and are spatially located (defined by: http://www.ontologydesignpatterns.org/ont/dul/DUL.owl).

• Outdoor environment: Environment situated outside of a house or other building.

• Information: Facts provided or learned about something or someone.

• Information_Object: They are messages performed by some entity^[46]. They are ordered (expressed in accordance with) by some information encoding system (e.g., sensors present in the agent). They can express a description (the ontological equivalent of a meaning/conceptualization), they can be about any entity, and they can be interpreted by an agent.

• Indoor Objects: Used to describe objects that exist or appear inside a home.

• Outdoor Objects: Used to describe objects that exist or appear outside a home.

• Physical Agent: Any agentive Object, either physical (e.g., a whale, a robot, or an oak tree) or social (e.g., a corporation, an institution, or a community) (defined by: http://www.ontologydesignpatterns.org/ont/dul/DUL.owl).

• Mobile robot: Robot that is able to move in the surrounding (locomotion) (i.e., autonomous mobile robot and autonomous mobile and manipulator robot).

• Not mobile robot: Robot that is not able to move in its surroundings (i.e., robot arm).

• Types Devices: A collection of properties that define different components and behaviors of a type of device (actuators, sensors, etc.).

The properties of objects and the spatial relationships between them represent the characteristics of the environment and the spatial arrangement, respectively. Several Object properties were created to relate the different concepts, so that an agent can characterize its surrounding environment:

• ObjectProperty:

- belowOf;

- isMember;

- isObjectOf;

- isPartOf;

- LeftOf;

- onTopOf;

- RightOf.

• LocationProperty:

- isConnectedTo.

• AgentProperty:

- isGoingTo;

- isIn.

The object properties were defined to make explicit the relationships between concepts. The properties belowOf, LeftOf, onTopOf, and RightOf were created to define the relationships between the different concepts of Indoor_Objects and Outdoor_Objects. Through these, the agent can identify the disposition of objects in the environment, creating relationships about them (e.g., based on Figure 3, if an agent has to guide an elderly person to the chair that is in the room, it knows that it is on the right side of the bed). The property isConnectedTo is a transitive and symmetric property, which correlates the different concepts of the Environment according to the environment in which the agent is inserted. The object properties isGoingTo and isIn are defined in order to correlate the AgentProperty concept with the Environment. (e.g., the robot isIn the living room, but it isGoingTo the bedroom). The object property isPartOf is a symmetric property that is used for the agent to link a given instance of the Objects class with an Environment. This allows the agent to know which objects are in a room. It differs from the object property isObjectOf because in this property the agent is sure that the object exists in the environment. Finally, the isObjectOf concept was defined to relate the concepts: Objects to the Environment. Through this, the objects that can be found in each zone of the environment are defined. (i.e., in a room there can be an object of the type bed, chair, television, carpet, etc.). Thus, an agent can search for an object by the place with the highest probability of it being found (i.e., if the agent has to find a frying pan, it knows that this object is commonly in a kitchen).

Figure 3. Example of the ontology being used to Instantiate the house objects belonging to the Instance “Bedroom”, present In the home environment.

3.2. PDDL planning agent

The Planning Domain Definition Language (PDDL) describes problems through the use of predicates and actions. The problems in PDDL are defined in two parts, a domain and a problem file. This language has undergone different modifications in order to make it capable of dealing with more complex tasks^[47–49]. The ROSPlan framework was used to perform the planning tasks^[50]. ROSPlan is a high-level tool that provides planning in the ROS environment; it generates the PDDL problem, the plan, the action dispatch, the replanning, etc. Different action interfaces have been written in C++ to control the Autonomous Manipulator Mobile Robot (AMMR) (i.e., base, arm, and gripper). These interfaces are constantly listening for action PDDL messages. In addition, the MongoDB database was used for semantic memory storage (locations, robots, home objects, goal parameters, etc.).

The POPF planner (https://nms.kcl.ac.uk/planning/software/popf.html), a forwards-chaining temporal planner, was used. After the plan was generated, the interface actions interconnect the plan with the lower level control actions, allowing the robotic agent (AMMR) to complete the plan. During execution, if an action fails due to changes in the environment, the planning agent reformulates the PDDL problem by re-planning.

4.1. Validation home environment ontology

The home environment ontology contains a vast number of concepts regarding the home environment as well as the different objects that may be present in a given room. For example, if a given agent is in a room for the first time, based on the objects it observes, through its sensors, it can categorize the space based on the knowledge represented on the ontology. Based on the ontology, if the agent sees objects such as knives, pots, and pans, then it infers that it must be in a kitchen. In this situation, the agent will identify the room and create all the relations of the objects it detects in the environment.

Knowledge reasoning techniques can infer new conclusions and thus help to plan dynamically in a nondeterministic environment. In the presented application, spatial reasoning and reasoning based on relations are used^[51]. Spatial reasoning is mostly used, when it is done through reasoning on the ontology hierarchy and spatial relations therein, allowing to predict the exact spatial location of an object in the environment. This prediction is obtained using a set of asserted facts and axioms on the ontology. Reasoning over the ontology relations is used to help with inferring new conclusions (e.g., If B is a subclass of A and C is a subclass of B, then C is a subclass of A can be inferred as transitivity holds for the subclass property).

The object property hierarchy view displays the asserted and inferred object property hierarchies. From the knowledge base built, all existing components from the environment are instantiated. That is, the individuals are based on their type (Figure 3). This allowed testing the functionality of the ontology. Figure 3 depicts that each room was instantiated to the corresponding Rooms subclass (e.g., the BedRoom instance is related to the Bedroom class). Several relations can be drawn from the instances that represent the environment, such as which objects belong to the BedRoom instance. Although the relations of the BathRoom and LivingRoom instances do not appear in Figure 3, these can be obtained, based on the created instances present in the ontology, in order to represent both rooms.

As previously stated, having the environment of a house in an ontological knowledge oriented database is of special interest, for example, to know where each object belongs. In fact, as depicted in Figure 4, the information about the BedsideTable is completely available to the user, using a simple logic description. A query to the ontology will retrieve useful information, for example where the object is attached. This issue is further discussed in the next subsection.

Figure 4. Example of the knowledge representation of an object in the environment of a house that has been instantiated.

Performing logical description reasoning actions to obtain valuable data for the robot’s reasoning can be done, as presented in Figure 5, namely information about objects, rooms, and their relations defined in the ontology. For example, what type of object is instantiated as Bed? In which rooms is it present? What is on its left? What is on its right? Using the ontology and description logic queries, it is straightforward to obtain the following information:

Figure 5. Reasoning using the ontology. (a) Query about the classes of objects that can be found in a class Hall. (b) Query about the instance(s) connected to instance of class Bathroom. (c) Query on the instance(s) that are part of the class Bedroom. (d) Query about the instance(s) that are part of class BedRoom and that are on the left of instances of class Chair. (e) Query to discover the location (room where the agent is in) based on what the agent observes.

• Classes of Objects that can be found in a certain instance of room (e.g., in the Hall in Figure 5a).

• Connectivity relationship between instances of the Rooms class referring to an environment (Figure 5b).

• Instance of objects present in an instance of room (Figure 5c).

• Recognize which instance(s) of the Objects class belong to a particular instance of a Rooms class and are to the left of an instance of the Chair class (Figure 5d).

• dRecognize which instance of the Rooms class belong to a particular instance(s) of a Objects class (e.g., the robotic agent can locate itself (know in which room it is), based on the objects it observes) (Figure 5e).

Through the ontology developed, one or more agents are able to locate themselves more efficiently in the environment. When the agent is lost, it can identify the room where it is, based on the objects it observes (Figure 5e). Observing Figure 3, if the robot recognizes a Bed, a BedsideTable, and a Chair_1, it knows that it is in a Bedroom. The robotic agent will be able to perform a search in an optimized way for an object. It does not need to perform a massive search for all the rooms; e.g., it knows which are the rooms in which there is a higher probability of finding a fridge, teapot, etc.

4.2. Validation the reasoning system with MongoDB

To test the system based on the MongoDB database, a problem was outlined, for the agent to execute/solve (Figure 6). An AMMR is used, composed of a mobile base and a robotic arm of the Universal Robotics UR3, equipped with a RobotIQ 2f-140 gripper, the whole system runs with the middleware: ROS (Figure 7a). Figure 7b depicts the layout of a simple home environment, an apartment for elderly people, created under the EUROAGE project^[52], which is in the robotics laboratory of the Polytechnic Institute of Castelo Branco.

Figure 6. Task definition in PDDL.

Figure 7. Robotic agent (AMMR) and simplified environment of a house, created in the framework of the EUROAGE project.

Initial conditions were established, such as the location of the AMMR (dock), the world coordinate at which the robot arm is located (p0), and the location of the object in the environment (LivingRoom), as well as its world coordinate (p3). The AMMR aims to leave the dock and pick up an object (obj1) that is in the living room, at the coordinates of world (p3). After the object is grabbed, the AMMR should take it to the bedroom and drop the object at the coordinates of world (p2). After the pick and place tasks are completed, the AMMR should return to the dock. This task definition is depicted in Figure 6.

Using ROSPlan, the generated plan is visible in Table 1. The right column presents the time of each durative action.

Figure 8, presents an excerpt of the global view of the ROS nodes and topics used by the system. It is possible to verify how the connections between them occur. It is visible in Figure 8a that, when the planner (ROSPlan) requests that the agent should leave the dock, the action responsible for the task, /rosplan_interface_undock, is triggered. This action in turn communicates with the action /move_base that communicates with the actuators (motors) to move the robot. Figure 8b shows that, when the robot is requested to move to a certain room, /rosplan_interface_move_base is activated, which in turn consults the MongoDB database to know where the room is located in the map. Then, the robot moves in the environment, after calling /move_base action.

Figure 8. Execution of a plan by the robotic agent (AMMR).