Deep learning-based scene understanding for autonomous robots: a survey

3.1. 3D object detection

Object detection is an image segmentation based on geometric and statistical features of the object. It combines object segmentation and recognition into one task, with the aim of determining the location and class of object appearances. Currently, 2D object detection has been relatively mature, especially with the emergence of Faster Regions with convolutional neural network Features (Faster RCNN), which has brought it to an unprecedented boom. For example, in the previous work of our research group ^[31], a deep neural network-based SSD framework is proposed to improve the feature representation capability of feature extraction networks. However, in the application scenarios of driverless, robotics, and augmented reality, 2D object detection can only provide the confidence of the position and corresponding category of the object in a 2D image (see Figure 4), while the general 2D object detection cannot provide all the information needed for perceiving the environment.

Figure 4. 2D object detection visualization: (A) in the bedroom; (B) in the kitchen ^[31].

In the real world, objects have 3D shapes, and most applications require information about the length, width, height, and also the deflection angle of the target object. Therefore, research on methods related to 3D target detection is needed. In scene understanding for autonomous robots, object detection is a critical task to understand the position and class of the objects with which they interact. In real 3D complex scenes, the background information is very rich; therefore, object detection techniques can be used to understand the location and category of interactable objects by giving a 3D rectangular location candidate box and categorizing them according to their attribution possibilities.

3D object detection based on deep learning is a hot research topic in the field of environment perception and understanding. In the deep learning-based model, during the process of making the proposals in regional proposal networks in a bottom-up manner, the resulting proposals somehow deviate from the ground truth and appear densely in local communities. Due to the lack of a corresponding information compensation mechanism, the proposals generated by the general regional proposal networks give up a large amount of boundary information. To deal with this problem, Qian et al. ^[32] proposed BADet, a 3D object detection model from point clouds, which can efficiently model the local boundary correlations of objects through local neighborhood graphs and significantly facilitate the complete boundaries of each individual proposal.

BADet consists of three key components, namely, a backbone and region generation network, a region feature aggregation module, and a boundary-aware graph neural network. Its network overview is shown in Figure 5.

Figure 5. The network overview of the BADet, where RPN denotes the region proposal network ^[32].

In the backbone and region proposal network (RPN) of BADet, the original point cloud is voxelized into a volume mesh for multi-scale semantic feature abstraction and 3D proposal generation with the help of the backbone and a series of 3D sparse convolutions. Specifically, let $$ p $$ be a point in a raw point cloud $$ P $$ with 3D coordinates $$ (p_x , p_y , p_z ) $$ and reflectance intensities $$ p_r $$, then

where $$ N $$ indicates the number of points within $$ P $$. Let $$ [v_L , v_W , v_H ] \in \Re ^3 $$ be the quantization step, then the voxelized coordinates of $$ p $$ can be obtained, namely

where $$ \left\lfloor {\cdot} \right\rfloor $$ is the floor function. Therefore, the point cloud $$ P $$ can be positioned into a feature map with a resolution of $$ L \times W \times H $$, subject to the quantization step $$ [v_L , v_W , v_H ] $$.

In the region feature aggregation module of BADet, multi-level semantic features are leveraged to obtain more informative RoI-wise representations. In the boundary-aware graph neural network, neighboring 3D proposals are used as inputs for graph construction within a given cutoff distance. Specifically, the local neighborhood graph $$ G(V, E) $$ can be constructed as

where $$ V $$ and $$ E $$ are the nodes and edges, respectively; $$ r $$ is the threshold; and $$ x_i $$ denotes the 3D coordinates of a node of graph $$ G $$.

In ^[32], an overall loss $$ L $$ is used, namely

where $$ L_{rpn} $$ and $$ L_{gnn} $$ are Focal Loss and Smooth-L1 Loss for the bounding box classification and regression, respectively; $$ L_{offset} $$ is the center offset estimation loss, which is used to obtain better boundary-aware voxelwise representations, and $$ L_{seg} $$ is the foreground segmentation loss.

To evaluate the performance of BADet, some comparison experiments are conducted on the KITTI and nuScenes datasets. The results of BADet are listed in Table 2.

The results in ^[32] show that BADet outperforms all its competitors with remarkable margins on KITTI BEV detection leaderboard and ranks 1st in "Car" category of moderate difficulty.

3D object detection methods have developed rapidly with the development of deep learning techniques. In recent years, many scholars have been exploring new results in this field. For example, Shi et al. ^[33] proposed the Part-A2 net to implement the 3D object detection using only LiDAR point cloud data. Li et al. ^[34] proposed the TGNet, a new graph convolution structure, that can effectively learn expressive and compositional local geometric features from point clouds.

According to the type of input data, 3D object detection can be divided into single-modal methods and multi-modal methods. Single-modal 3D object detection refers to the use of data collected by one kind of sensor as input. The advantage of single-modal 3D target detection is that the input data are simple and the processing flow is clear; the disadvantage is that the input data may not be sufficient to describe the target information in 3D space. Multi-modal 3D object detection refers to the use of multiple data collected by multiple types of sensors as inputs. The advantage of multi-modal 3D target detection is that the input data are rich, and the complementarity of different modal data can be utilized to improve the accuracy and robustness of the detection. The disadvantage is the complexity of the input data and the need to deal with inconsistencies between different modal data. In the following, a summary of the deep learning-based 3D object detection models presented in the last five years is illustrated in Table 3, where the type of the input data of each method is given out.

3.2. Pose estimation

Pose estimation is a crucial component of autonomous robot technology. The pose estimation task deals with finding the position and orientation of an object with respect to a specific coordinate system. The vision-based pose estimation approaches employ a number of feature extraction techniques to obtain the spatial positional information of the target from the image.

There are two classical methods for pose estimation, namely, the feature-based techniques and the template matching methods. The traditional feature-based technique primarily extracts features from images and creates a relationship between the 2D pixel points and 3D coordinate points in space. The differences in lighting and background complexity have a significant impact on the feature extraction process. In addition, the feature-based methods struggle to handle sparse target texture features. The template matching method can effectively solve the pose estimation problem for the targets with weak texture features in images. However, the accuracy of the template matching method is determined by the number of samples in the template library. While its accuracy improves with the number of the template libraries, it also causes a decrease in problem-solving efficiency, making it unable to meet real-time requirements.

The development of deep learning has influenced pose estimation, and there are numerous study findings in this field. For example, Hoang et al. ^[44] proposed the Voting and Attention-based model, which enhances the accuracy of object pose estimation by learning higher-level characteristics from the dependencies between the individual components of the object and object instances. The structure of this Voting and Attention-based network is shown in Figure 6.

Figure 6. Network architecture of the Voting and Attention-based model ^[44].

As shown in Figure 6, there are four main parts in the Voting and Attention-based model, namely the feature extraction module based on PointNet++ architecture, part proposals learning module ($$ \text{M}_{\text{p}} $$), object proposals learning module ($$ \text{M}_{\text{o}} $$), and the voting module in both $$ \text{M}_{\text{p}} $$ and $$ \text{M}_{\text{o}} $$ based on VoteNet.

In the $$ \text{M}_{\text{p}} $$ module of the Voting and Attention-based model, the higher-order interactions between the proposed features can be explicitly modeled, which is formulated as non-local operations:

where $$ \theta(\cdot), \phi(\cdot), and g(\cdot) $$ are the learnable transformation on the input feature map $$ H $$, and $$ f(\cdot) $$ is the encoding function of the relationship between any two parts.

In addition, the compact generalized non-local network (CGNL) ^[45] is used as the self-attentive module in $$ \text{M}_{\text{p}} $$. Specifically, the CGNL-based self-attentive module takes $$ K $$ clusters $$ C=(C_1, C_2, ...C_K) $$ as input. Then, votes from each cluster are processed by the Multi-Layer Perceptron (MLP) and passed to CGNL. The self-attention mechanism allows features from different clusters to interact with each other and find out who they should pay more attention to.

Similarly, in the $$ \text{M}_{\text{o}} $$ module of the Voting and Attention-based model, the instance-to-instance correlation is modeled. Firstly, $$ K $$ clusters from the high-dimensional features and a set of object centers are generated. Then, CGNL is used to model the rich interdependencies between clusters in feature space. The output is a new feature mapping:

where $$ v_i $$ is the $$ i $$-th vote.

Finally, the new feature maps $$ H_{part-part} $$ and $$ H_{obj-obj} $$ are aggregated to the global information by an MLP layer after a max-pooling and concatenation operations.

In the Voting and Attention-based model, a multi-task loss is used for joint learning, namely

where $$ \lambda_1 $$, $$ \lambda_2 $$, and $$ \lambda_3 $$ are the weights of each task. The losses include voting partial loss $$ L_{part-vote} $$, object voting loss $$ L_{object-vote} $$, and pose loss $$ L_{pose} $$.

The pose estimation results based on the Voting and Attention-based model on nine objects in the Sil$$ \acute{e} $$ane dataset and two objects in the Fraunhofer IPA dataset are listed in Table 4, and some qualitative results are shown in Figure 7.

Figure 7. Visualization for pose estimation results based on the Voting and Attention-based model: (A) 3D point cloud input; (B) True values of the poses; (C) Results obtained by the method in ^[44]. The different color means the visualization of point-wise distance error, ranging from 0 (green) to greater than 0.2 times the diameter of the object (red).

The results in Table 4 and Figure 7 show that the Voting and Attention model is very effective in improving the accuracy of the pose estimation, which can obtain an average precision of 53%.

In addition to the above voting-based model, there are lots of research results in pose estimation based on deep learning methods. For example, Chen et al. ^[46] presented a probabilistic PnP (EPro-PnP) model for general end-to-end pose estimation, which is based on the method of locating 3D objects from a single RGB image via Perspective-n-Points (PnP). The EPro-PnP model can realize reliable end-to-end training for a PnP-based object pose estimation network by back-propagating the probability density of the pose to learn the 2D-3D association of the object.

Currently, there are five main types of methods for pose estimation, including feature-based methods, regression-based methods, projection-based methods, representation learning methods, and graph neural network methods. The feature-based method refers to restoring camera pose by establishing feature correspondence between images and scenes. A regression-based method uses a regressor to predict the camera pose. A projection-based method utilizes projection transformation to estimate the pose of a target from an image or video. A representation learning method utilizes deep neural networks to learn high-resolution representations of objects from images or videos, which can improve the accuracy and interpretability of pose estimation. Graph neural network methods use graph neural networks to learn structured representations of objects from images or videos, which can improve robustness of pose estimation. In the following, a summary of the deep learning-based pose estimation models presented in the last five years is illustrated in Table 5, where the type of each method is given out.

3.3. Semantic segmentation

Semantic segmentation is a refined version of image classification. For an image, traditional image classification is to detect and recognize the objects that appear in the image, while semantic segmentation is to classify every pixel point in the image. In the field of autonomous robot environment perception and understanding, semantic segmentation is used to label each pixel in an image with its corresponding semantically meaningful category. Semantic segmentation can help robots recognize and understand surrounding objects and scenes. It is very useful for the semantic segmentation for the robot to find a specific object in the environment. For example, in the field of logistics robotics, semantic segmentation can help the autonomous robots perceive and understand road conditions, traffic signs, pedestrians, and vehicles, which can improve the safety and efficiency of logistics robotics.

The traditional semantic segmentation algorithms are mainly grayscale segmentation, conditional random fields, etc. Grayscale segmentation algorithms recursively segment images into sub-regions until labels can be assigned and then combine adjacent sub-regions with the same labels by merging them. The conditional random field is a type of statistical modeling method for structured prediction.

With the continuous development of deep learning techniques, deep learning has been widely applied in semantic segmentation tasks and achieved impressive results. There are a series of classical deep learning-based models for semantic segmentation, such as Full convolution network (FCN) ^[54], SegNet ^[55], DeepLab series ^[56,57], RefineNet ^[58], DenseASPP ^[59], etc. Recently, some improvements have been proposed based on those classical models. For example, Yin et al. ^[60] presented a multi-sensor fusion for lane marking semantic segmentation (FusionLane) based on the DeepLabV3+ network. The workflow of the FusionLane model is shown in Figure 8.

Figure 8. The workflow of the FusionLane model ^[60]. CBEV denotes the camera bird's eye view data. LBEV denotes the points cloud bird's eye view data. C-Region denotes the obtained semantic segmentation result on CBEV.

As shown in Figure 8, firstly, the DeepLabV3+ network is used to achieve semantic segmentation on camera BEV (CBEV) data (called as C-Region). Then, the C-Region and LiDAR point cloud BEV (LBEV) data are input into the FusionLane model to realize the lane marking semantic segmentation. Unlike other methods that mainly focus on the analysis of camera images, the semantic segmentation data used in FusionLane is a BEV image converted from the LiDAR point cloud instead of the images captured by the camera to obtain the accurate location information of the segmentation results.

The network contains two data input branches: the camera data and the point cloud data. The data from the two branches need to be preprocessed to meet the network input requirements. For the camera data, the front view is converted into CBEV. In CBEV, one pixel represents an area of $$ 5cm\times 5cm $$ in real space. Then, the CBEV image is semantically segmented using the trained DeepLabV3+ network to obtain the C-Region input data. For the point cloud data, it is projected into the 3D BEV with three channels. The values of the three channels are calculated as follows:

where $$ F\left( x, y\right) $$, $$ S\left( x, y\right) $$, and $$ T\left( x, y\right) $$ denote the values of the first channel, the second channel, and the third channel, respectively; $$ i \in [0, 1] $$ is the reflection intensity value of each point falling within the grid corresponding to the pixel; $$ h \in [ - 2, - 1] $$ is the height value of each laser spot falling within the grid, and $$ arctan $$ is used as the normalization function.

In the FusionLane model of ^[60], an encoder-decoder network model is proposed, and the LSTM structure is added to the network to assist the semantic segmentation of the lane marking. At last, the KITTI dataset is used to test the performance of the FusionLane model, which is processed and divided into seven categories. The experimental results are listed in Table 6, and some segmentation results based on the FusionLane network are shown in Figure 9.

Figure 9. The segmentation results based on the FusionLane network for some scenarios ^[60].

The results in Table 6 show that DeepLabV3+ has a low $$ IoU $$ for all scenarios except "Background". However, it can be seen that the FusionLane model achieves the best results in all metrics compared to the traditional DeepLabV3+ model. The results in Table 6 and Figure 9 show that relying on a single kind of sensor, whether camera or LiDAR, cannot give sufficiently accurate semantic segmentation results. Effective fusion of data from different sensors can be considered a viable approach to solving the problem.

In addition to the above DeepLab-based model, there are lots of good semantic segmentation models based on deep learning methods. For example, Hu et al. ^[61] presented the FANet model, which is based on an improved self-attention mechanism, to capture the rich spatial context at a small computational cost. Sun et al. ^[62] proposed the FuseSeg model, a new RGB and thermal data fusion network, to achieve superior semantic segmentation performance in urban scenes.

More and more scholars have researched many results in this field. According to the type of network structure, semantic segmentation can be divided into encoder-decoder structure, attention mechanism, graph neural network, generative adversarial network (GAN), and transfer learning. The semantic segmentation method based on encoder-decoder utilizes the encoder-decoder structure to learn and predict the semantic category of each pixel from an image. The method based on GAN uses a generator and a discriminator to conduct confrontation learning. Attention mechanism is a technique that simulates the process of human visual attention. It can calculate the correlation between different positions or channels, give different weights, and highlight the parts of interest while suppressing irrelevant parts. A graph neural network is a deep neural network that can process graph-structured data, which can update the features of nodes and edges through graph convolution operations. Transfer learning is a machine learning technology that can use the knowledge of one domain (source domain) to help the learning of another domain (target domain), thus reducing the dependence on the labeled data of the target domain. A summary of the deep learning-based 3D semantic segmentation models presented in the last five years is illustrated in Table 7, where the type of the network structure of each method is given out.

3.4. Saliency prediction

The human visual system selectively attends to salient parts of a scene and performs a detailed understanding of the most salient regions. The detection of salient regions corresponds to important objects and events in a scene and their mutual relationships. In the field of scene understanding for autonomous robots, the task of the saliency prediction is to mimic the characteristics of human vision to focus on obvious or interested targets by acquiring 3D environment information containing color and depth through sensors. In detail, the saliency prediction needs to identify and segment the most salient objects from the acquired 3D environment information and pay attention to the focal objects.

The traditional saliency prediction problem is commonly known as the task of capturing rare and unique elements from images. Traditionally, salient prediction methods can be classified into three types: (1) Block-based detection models. In this type of method, the linear subspace method is used instead of actual image segmentation, and the significant regions are selected by measuring the feature pair ratio and geometric properties of the region. (2) Region-based detection models. This type of method divides the image into multiple regions, and the saliency of each region is regarded as the sum of the product of its contrast and the weight of all the other regions. (3) Detection model based on external cues of the image. This model utilizes accurate annotations (ground-truth) obtained from the training set, video sequences, similar images, and other sources to make the results more accurate. The performance of the saliency prediction based on similar images will be improved if a large number of data sets are available. In general, the traditional methods use a large amount of saliency a priori information for saliency detection, mainly relying on hand-crafted features. These hand-crafted features have some shortcomings; for example, they may not be able to describe complex image scenes and object structures, cannot adapt to new scenes and objects, and have poor generalization ability. So, the saliency detection based on traditional methods has hit a bottleneck.

Recently, deep learning-based methods have been used widely in various image tasks (e.g., target detection, semantic segmentation, edge detection, etc.), which provide new ideas for saliency prediction and show surprising effect enhancement in some studies. For example, Lou et al. ^[72] proposed the TranSalNet network model. Its basic workflow is shown in Figure 10.

Figure 10. The schematic overview of the TranSalNet network ^[72].

As shown in Figure 10, the convolutional neural network (CNN)-based encoding is used to extract features for saliency prediction. The outputs of the CNN encoding are three sets of multi-scale feature maps with $$ \frac{w}{8} \times \frac{h}{8} $$, $$ \frac{w}{16} \times \frac{h}{16} $$, and $$ \frac{w}{32} \times \frac{h}{32} $$, respectively. Then, these feature maps are input into the transformer encoders to enhance the long-range and contextual information. At last, a CNN decoder is used to fuse the enhanced feature maps from the three transformer encoders. The CNN decoder used in ^[72] is a full CNN network with seven blocks. The processes from block1 to block6 are as follows:

where $$ X_i^f $$ and $$ {\hat X}_{i}^f $$ are the input and output of the $$ i $$-th block. The output of the block7 $$ \hat y $$ is the predicted saliency map, namely

where $$ \rm{Sigmoid}(\cdot) $$ is the sigmoid activation function; $$ {\rm{Conv}}_{3 \times 3} $$ denotes the $$ 3 \times 3 $$ convolution operation; and $$ X_6^f $$ is the output of the block6.

In the TranSalNet network model, a linear combination of four losses is used as the loss function, namely

where $$ L_{NSS} $$ is the Normalized Scanpath Saliency loss; $$ L_{KLD} $$ is the Kullback–Leibler divergence loss; $$ L_{CC} $$ is the Linear Correlation Coefficient loss; and $$ L_{SIM} $$ is the Similarity loss. $$ \omega _1 $$, $$ \omega _2 $$, $$ \omega _3 $$, and $$ \omega _4 $$ are the weights of each loss.

Some results of the TranSalNet network model are listed in Table 8, where TranSalNet_Res and TranSalNet_Dense denote the CNN encoders used in the TranSalNet network, ResNet_50 and DenseNet_161, respectively. Here, two public datasets are as follows: (1) MIT1003 ^[73]: This dataset contains 300 natural images and eye movement data from 39 observers and is the most influential and widely used dataset in the field of image human eye focus detection. (2) CAT2000 ^[74]: This dataset includes 4000 images, 200 in each of 20 categories, covering different types of scenes such as cartoon, art, object, low-resolution image, indoor, outdoor, chaotic, random, and line drawings. Some saliency maps generated by the two models are shown in Figure 11.

Figure 11. Results of saliency maps generated by TranSalNet_Res and TranSalNet_Dense ^[72]. The images from (A) to (C) are from the MIT1003 dataset, and the images from (D) to (F) are from the CAT2000 dataset.

The results in Table 8 and Figure 11 prove that the TranSalNet architecture presented in ^[72] is effective in the saliency prediction tasks. In addition, the results in Table 8 and Figure 11 show that the performance of the TranSalNet could be further enhanced by replacing ResNet-50 with DenseNet-161.

In addition to the above TranSalNet model, there are other saliency prediction models based on deep learning, which also have obtained good results in this field. For example, Zou et al. ^[75] proposed the STA3D model, where the S3D network is used as an encoder and the prediction network with spatial dimensional upsampling and temporal dimensional compression is used as a decoder, to solve the difficulty of video significance prediction in the continuous frame with a fixed offset.

At present, there are five types of methods for saliency prediction, including gradient-based methods, perturbation-based methods, SHAP value-based methods, attention mechanism-based methods, and transfer learning-based methods. The gradient-based method utilizes the gradient information of neural networks to calculate the contribution of each pixel in the input image to the output saliency map. The perturbation-based method evaluates the importance of each pixel by randomly or regularly perturbing the input image. The method based on SHAP values utilizes shapely additive explanations to quantify the impact of each pixel on the output saliency map. The saliency prediction, based on attention mechanisms, utilizes an attention mechanism to simulate the process of human visual attention, thereby improving the accuracy and interpretability of saliency prediction. Transfer learning is used to solve the problem of data shortage and domain differences in saliency prediction, which can improve the generalization ability and adaptability of saliency prediction. A summary of the deep learning-based models in the last five years is illustrated in Table 9, where the type of each method is given out.

3.5. Other applications

In addition to the applications mentioned above, there are many other applications of deep learning methods in autonomous robot environment perception and understanding, such as image enhancement ^[84,85], visual SLAM ^[1,86], scene classification ^[87,88], moving object detection ^[89,90], and layout estimation ^[91,92]. In this section, some recent jobs of our group related to this review will be introduced in detail as follows.

3.5.1. Visual SLAM

When a robot enters an unknown environment, vision SLAM technology can be used to solve the problem of the robots about where they are. It estimates the current position, pose, and travel trajectory of the robot in the 3D scene by the changes in the visual data acquired during the robot's travel. In order to implement vision SLAM, there are three main methods: feature-based methods, direct methods, and semi-direct methods.

With feature-based visual SLAM methods, feature points are found and matched. Then, the poses of robots are calculated, and maps are built from geometric relationships. Scalar Transformation (SIFT) ^[11], Accelerated Robust Feature (SURF) ^[93], and Fast Rotational Abbreviation (ORB) ^[94] are the most frequently used feature extraction techniques. The most widely used method for visual SLAM is ORB-SLAM ^[95,96]. To overcome the problem of high computational complexity in the traditional ORB-SLAM, Fu et al. ^[97] proposed the Fast ORB-SLAM that is light-weight and efficient as it tracks keypoints between adjacent frames without computing descriptors.

Direct methods do not rely on one-to-one matching of points. These types of methods minimize the photometric error function of the pixels by extracting pixels with significant gradients and optimizing the inter-frame pose. The classical direct methods include Large Scale Direct Monocular SLAM (LSD-SLAM) ^[98], Direct Sparse Range (DSO) ^[99], etc. Recently, Wang et al. ^[100] introduced a new ceiling-view visual odometry method that introduces plane constraints as additional conditions and achieves better accuracy.

Semi-direct methods, such as SVO ^[101], employ a similar structure to the feature-based methods, which combine the tracking of the direct method with the motion optimization of feature-based methods. Both feature-based and semi-direct methods rely on highly repeatable low-level geometric feature extractors. Both of them are inappropriate for surfaces with little texture or many repetitive features.

Direct methods, on the other hand, can be applied to a wider variety of scenes. However, compared to feature-based methods, direct methods are less robust. The performance of the direct visual SLAM system under the influence of various camera imaging perturbations will be reduced obviously. To deal with this problem, our group proposed an improved Direct Sparse Odometry with Loop Closure (LDSO) method ^[102], which is shown in Figure 12.

Figure 12. The framework of the improved LDSO method based on the variable radius side window ^[102].

In the framework of the improved LDSO shown in Figure 12, the region surrounding each pixel is divided into blocks when a new frame is introduced, using the side window approach. Then, a CNN structure is created by this multiple-layer superposition of pixel information fusion ^[22,103]. The middle layer shows the presence of semi-static items. In the later layers, the radius of the side windows of the pixels belonging to the semi-static objects is increased. Points with an adequate gradient intensity and corners are chosen using dynamic grid searches. The robustness of the system is increased by the addition of points in direct SLAM. To accomplish edge protection, the fusion method is used with a side window mechanism. Finally, to lessen the weight of semi-static objects, the radius of the adjustment side windows is modified in accordance with the semantic information based on a pre-trained Yolov5 model ^[104].

In the experiments to test the performance of the improved LDSO in ^[102], two public datasets are used: the KITTI dataset (outdoor datasets) and the TUM RGB-D dataset (indoor datasets). To test the improved LDSO under different camera sensor noises, Gaussian noise and Salt-and-Pepper noise are added to the two datasets. Some results of visual SLAM based on the improved LDSO are shown in Table 10 and Table 11, where RMSE$$ _{\rm{ATE}} $$ means the root mean squared error of absolute trajectory error. The comparison results on the KITTI dataset with Salt-and-Pepper noise are not given out because the general LDSO is entirely inoperable on the datasets under Salt-and-Pepper noise. The results of the point cloud map constructed on the sequence 'KITTI_07' in the KITTI dataset are shown in Figure 13. The results in Tables 10 and 11 show that the improved LDSO in ^[102] can work efficiently in both the indoor and the outdoor datasets under different noises, while the general LDSO will fail to track (see Figure 13).

Table 10

RMSE$$ _{\rm{ATE}} $$ on the KITTI dataset with Gaussian noise ^[102]

Method	Gaussian noise
	KITTI_00	KITTI_01	KITTI_02	KITTI_03	KITTI_04	KITTI_05	KITTI_06	KITTI_07	KITTI_08	KITTI_09	KITTI_10	Average
‘–’ means tracking failure. The average value is calculated based on the number of successes.
LDSO [105]	22.543	23.052	169.247	–	–	44.010	58.729	53.481	130.993	–	16.277	64.792
Improved LDSO	17.772	13.023	120.380	2.133	1.093	5.740	13.491	1.973	102.206	52.664	14.042	31.320

Table 11

RMSE$$ _{\rm{ATE}} $$ on the TUM RGB-D dataset ^[102]

Method	Gaussian noise					Salt-and-pepper noise
	fr1_xyz	fr2_xyz	fr2_rpy	fr1_desk	fr1_desk2	fr1_xyz	fr2_xyz	fr2_rpy	fr1_desk	fr1_desk2
‘–’ means tracking failure.
LDSO [105]	–	0.096	–	0.518	–	–	–	–	0.841	–
Improved LDSO	0.156	0.01	0.06	0.801	0.756	0.129	0.011	0.058	0.796	0.871

Figure 13. Sample outputs of the sequence 'KITTI_07' in the KITTI dataset ^[102]: (A) and (B) are the outputs on the sequence with Gaussian noise and Salt-and-Pepper noise, respectively.

3.5.2. Scene classification

Scene classification is one of the key technologies of scene understanding for autonomous robots, which can provide the basis for decision-making of the robots. The task of the scene classification for an autonomous robot refers to the information of its surroundings obtained by the on-board sensors, and then the state of the current position is recognized.

Lots of researchers have conducted studies on scene classification. For example, Tang et al. ^[106] proposed an adaptive discriminative region learning network for remote sensing scene classification, which locates discriminative regions effectively for solving the problems of scene classification, such as scale-variation of objects and redundant and noisy areas. Song et al. ^[107] used an ensemble alignment subspace adaptation method for the cross-scene classification. It can settle the problem of both foreign objects in the same spectrum and different spectra. Zhu et al. ^[108] proposed a domain adaptation cross-scene classification approach to simultaneously classify the target common categories and detect the target private categories based on the divergence of different classifiers.

The methods for the scene classification can be divided into two main types. One of them is based on the underlying visual features. This type of method has some shortcomings. For example, the accuracy of the scene classification is low when only the low-level visual features are used to represent the contents of the scene. The other type of the scene classification method is based on the deep learning technologies. To deal with the problem of the scene classification of the road scene, our group presented an improved deep network-based model ^[109]. The structure of the proposed model is shown in Figure 14.

Figure 14. The structure of the proposed deep network for the road scene classification ^[109].

As shown in Figure 14, there are four main parts in the proposed scene classification model, namely, (1) The improved Faster RCNN-based local feature extraction module; (2) The improved Inception_V1-based global feature extraction module; (3) The feature fusion module; (4) The classification network.

In the improved Faster RCNN-based local feature extraction module, the VGG16 Net is used to get the feature map of the whole image first. Then, a residual attention module is used to further deal with redundant information in images. The operation on the feature map based on the residual attention module is:

(14) $$ F_{output} (i, j) = F_{input} (i, j) \otimes a_{ij}+F_{input} (i, j) $$

where $$ F_{output} $$ and $$ F_{input} $$ are the output and the input feature value of the residual attention module, respectively; $$ a_{ij} $$ is the attention weight; $$ \otimes $$ is the dot product operation.

The output of the residual attention module is input into the RPN to generate region proposals. The output Region-of-Interests (ROIs) of the RPN is processed by a ROI pooling network to get a fixed-size proposal feature map, which is finally input into a fully connected layer for the object classification and generating the positions of the objects.

In the global feature extraction module, the Inception_V1 is used as the baseline network, which has nine Inception blocks. One Inception block has four branches. To deal with the shortcomings of the general Inception_V1 ^[110], the Inception_V1 is improved in the proposed model in ^[109], where a mixed activation function is presented by alternately using the ELU and Leaky ReLU functions for the Inception networks. The Leaky ReLU function is denoted by:

(15) $$ y_i = \left\{ \begin{array}{l} x_i, \text{ if } x_i \ge 0 \\ \alpha x_i, \text{ if } \alpha x_i < 0 \\ \end{array} \right. $$

where $$ \alpha $$ is a fixed parameter.

The ELU function is denoted by:

(16) $$ y_i = \left\{ \begin{array}{l} x_i, \; \text{if}\; \; x_i \ge 0 \\ e^{x_i} - 1, \; \text{if}\; \; x_i<0 \\ \end{array} \right. $$

In the feature fusion module, the local feature vectors and the global feature vectors are appended to get the fused feature $$ F $$, namely

(17) $$ F = \left[ {L, G} \right] $$

where $$ L{\rm{ = }}\left[ {l _{\rm{1}}, l _2, \cdots, l _N } \right] $$ and $$ G{\rm{ = }}\left[ {g _{\rm{1}}, g_2, \cdots, g_N } \right] $$ denote the local feature vectors and the global feature vectors, respectively; $$ N $$ is the feature dimension.

At last, the fused feature vector is input to the classification network for the scene classification. The loss function used in this classification network is as follows:

(18) $$ Loss_{cls} = \frac{1}{S}\sum\limits_i {\left( { - \sum\limits_{j = 1}^C {y_{ij} \log \left( {p_{ij} } \right)} } \right)} $$

where $$ C $$ is the number of scene classification; $$ S $$ is the number of the samples; $$ p_{ij} $$ is the probability that the $$ i $$-th sample belongs to the $$ j $$-th category, and $$ y_{ij} $$ is the indicator variable.

To test the performance of the proposed road scene classification model, our group proposed a special dataset based on two public datasets: KITTI ^[111] and Place365 ^[112]. The results of the comparison experiments are listed in Table 12, and some scene classification results based on different models are shown in Figure 15.

Table 12

The experimental results of scene classification based on different deep networks ^[109]

Network	Total accuracy	Standard deviation	On sunny days	On rainy days	At night
AlexNet [113]	84.20%	5.22%	90.20%	81.70%	80.70%
EfficientNet [114]	87.07%	8.31%	96.30%	80.00%	85.30%
Inception_V1 [110]	90.53%	2.51%	93.40%	88.70%	89.50%
Ours	94.76%	1.62%	96.50%	93.30%	94.50%

Figure 15. Some scene classification results based on different models ^[109].

It can be seen that our proposed model can improve the accuracy to 94.76%, which is 4.67% (Relative value) higher than the general Inception_V1 (the second-best model). In addition, our proposed model has good scene classification performance under some challenging tasks, such as the task on a rainy day or at night (see Table 12 and Figure 15 for details).

Authors' contributions

Funding acquisition: Ni J

Project administration: Ni J, Shi P

Writing-original draft: Chen Y, Tang G

Writing-review and editing: Shi J, Cao W

Availability of data and materials

Not applicable.

Financial support and sponsorship

This work has been supported by the National Natural Science Foundation of China (61873086) and the Science and Technology Support Program of Changzhou (CE20215022).

Conflicts of interest

The authors declared that they have no conflicts of interest related to this work.

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2023.