A Visual Navigation Algorithm for UAV Based on Visual-Geography Optimization

Abstract

The estimation of Unmanned Aerial Vehicle (UAV) poses using visual information is essential in Global Navigation Satellite System (GNSS)-denied environments. In this paper, we propose a UAV visual navigation algorithm based on visual-geography Bundle Adjustment (BA) to address the challenge of missing geolocation information in monocular visual navigation. This algorithm presents an effective approach to UAV navigation and positioning. Initially, Visual Odometry (VO) was employed for tracking the UAV’s motion and extracting keyframes. Subsequently, a geolocation method based on heterogeneous image matching was utilized to calculate the geographic pose of the UAV. Additionally, we introduce a tightly coupled information fusion method based on visual-geography optimization, which provides a geographic initializer and enables real-time estimation of the UAV’s geographical pose. Finally, the algorithm dynamically adjusts the weight of geographic information to improve optimization accuracy. The proposed method is extensively evaluated in both simulated and real-world environments, and the results demonstrate that our proposed approach can accurately and in real-time estimate the geographic pose of the UAV in a GNSS-denied environment. Specifically, our proposed approach achieves a root-mean-square error (RMSE) and mean positioning accuracy of less than 13 m.

1. Introduction

In recent years, UAVs have been extensively utilized in various domains, including military operations, search and rescue missions, and terrain exploration [1,2,3,4,5]. While GNSS technology can provide precise geolocation information for UAVs, it is susceptible to signal blockage and interference from adversarial sources [6]. Therefore, it is imperative to develop methodologies for UAV localization in GNSS-denied environments [7]. In such challenging scenarios, UAVs must rely solely on their onboard camera to infer their pose without any external infrastructure. Hence, developing robust techniques for accurate and independent UAV positioning becomes paramount.

In GNSS-denied environments, UAVs can rely on visual sensors for precise pose estimation [8], enabling autonomous navigation and positioning. Visual sensors play a crucial role in environmental perception by capturing image information that provides UAVs with abundant contextual data. The technologies for UAV visual navigation and positioning can be primarily categorized into two approaches: VO, which operates without a map, and geolocation based on image matching, which utilizes an existing map.

The UAV can estimate its relative pose using VO, which solely relies on a cost-effective and lightweight vision sensor to accurately determine the local pose of the UAV in real-time. Depending on different approaches for associating features between frames, it can be categorized into feature-based, direct, and hybrid methods. The feature-based method is employed to extract features from the image for matching, while camera pose estimation is achieved by minimizing the reprojection error. PTAM [9] and ORB-SLAM [10] are both classical SLAM systems that utilize the feature-based approach. Among them, ORB-SLAM leverages ORB [11] features in matching, tracking, relocation, and loop closing processes, thereby enhancing information interaction efficiency across different threads. Subsequently, ORB-SLAM2 [12] and ORB-SLAM3 [13] also employ ORB features. On the other hand, direct methods directly use the pixel intensities in the images and estimate motion by minimizing a photometric error. DSO [14] and VINS-Mono [15] are advanced SLAM systems that employ the direct method. SVO [16] is a hybrid VO approach that initially extracts FAST features but uses a direct method to track features and any pixel with a nonzero intensity gradient from frame to frame.

Currently, state-of-the-art VO algorithms employ keyframe-based techniques [17], which provide higher accuracy compared to filtering methods at the same computational cost [18]. Keyframe-based techniques utilize a multi-threaded approach for tracking the robot’s motion, comprising a tracking thread responsible for the UAV’s motion tracking and keyframe selection and a mapping thread that performs computationally expensive yet more accurate BA optimization at the keyframe rate. This enables the VO system to achieve real-time performance while enhancing its localization accuracy. Although VO has demonstrated reliable precision in short-term operations, it tends to accumulate drift over long-term operations. Moreover, VO fails to provide a more comprehensive geographic pose, which limits its applicability in aerial scenarios. Therefore, it is crucial to incorporate supplementary techniques to address these challenges.

With the continuous advancement of image-matching technology, sophisticated image-matching networks can effectively address the challenges posed by significant variations in seasons, lighting conditions, and other factors between satellite images and aerial images [19,20]. Consequently, these networks facilitate establishing correspondences between aerial and satellite images to accurately determine the geographic pose of the UAV. Various methods [21] have been developed for image matching and localization in GNSS-denied environments. Goforth et al. [22] employed a deep convolutional neural network (CNN) with an iterative closest landmark keypoints (ICLK) layer to align aerial images and satellite images. Chen et al. [23] proposed an image-based geolocation method that is adaptable to a downward-tilted camera, which necessitates prior offline dataset preparation.

However, a drawback of geolocation methods based on image matching is their significant computational cost, leading to substantial delays and rendering them unsuitable for meeting the real-time positioning requirements of UAVs [24]. To enhance the positioning performance of visual navigation algorithms, certain studies have focused on integrating VO with image-matching methods. Kinnari et al. [25] orthorectified UAV images based on VIO and planarity assumptions, which is also applicable to a downward-tilted camera configuration. The ortho-projected image is utilized for matching with the satellite images, and the geolocation results are combined with the pose estimation from VIO in a particle filter framework. Hao et al. [26] proposed a geolocation method that utilizes global pose graph optimization to integrate RVIO measurements and image registration. Zhang et al. [27] proposed an iterative trajectory fusion pipeline that integrates vSLAM measurements and image matching through solving a scaled pose graph problem.

The aforementioned studies employ a trajectory fusion method to integrate the measurements from VO and image matching, treating them as two relatively independent modules. While this design simplifies the system architecture, it also results in a disconnection between VO and image matching, impeding the immediate feedback of image-matching results in the VO estimation process. This limitation may potentially impact the overall accuracy and robustness of the system.

To fulfill the aforementioned requirements and address existing concerns, we have developed a tightly coupled visual navigation algorithm for UAVs based on visual-geography optimization. As depicted in Figure 1, the algorithm is structured into three concurrent threads inspired by the SLAM multi-threading concept: tracking thread, mapping thread, and image-matching thread. The tracking thread enhances real-time performance by monitoring the UAV’s movement, while the mapping thread optimizes the VO map through the fusion of visual information from VO and geographic positioning data obtained via image matching. Additionally, the image-matching thread provides globally unbiased geographic positioning information. The image-matching thread incorporates the existing coarse-to-fine image-matching method [23], while also introducing a prior-based approach. The prior-based image-matching method not only reduces computational costs but also enhances positioning accuracy, which holds significant implications for the fusion algorithm with VO. In the mapping thread, we have developed a robust geographic initializer and map fusion algorithm. The geographic initializer accurately estimates initialization parameters to align the VO world coordinate system with the geographic coordinate system. The map fusion algorithm utilizes geographical positioning information obtained from image matching for real-time optimization of the VO map. In summary, our algorithm enables real-time and precise estimation of a UAV’s geographic pose through visual-geography optimization.

This work makes the following contributions:

• We propose a UAV visual navigation algorithm that combines the merits of VO and image matching.
• We introduce a geolocation method based on heterogeneous image matching, which employs the coarse-to-fine and prior-based image-matching methods to enhance the accuracy of geolocation. This method effectively leverages the operational characteristics of VO to provide precise geolocation information.
• We present a tightly coupled information fusion method based on visual-geography optimization that jointly optimizes the visual and geolocation information of keyframes to facilitate tightly coupled geolocation in algorithms. Compared with the existing trajectory fusion method, our method achieves higher positioning accuracy.

The remaining sections of this paper are organized as follows. Section 2 presents a detailed explanation of the visual navigation algorithm for the UAVs proposed in this study. In Section 3, we discuss our real-world experiments conducted to validate our approach. Concluding remarks are provided in Section 4. Finally, an extensive discussion is presented in Section 5.

2. Method

The proposed visual navigation algorithm for the UAV is detailed in this section. This paper presents a visual-geography optimization method for integrating visual information (VO map consists of the UAV’s poses and map points) and geographic information (the geographical poses of the UAV are determined through image matching), enabling geographic initializer and real-time pose estimation. More details are shown in Algorithm 1. The following section introduces the fusion of visual and geographic information with tight coupling as well as geolocation based on heterogeneous image matching.

Algorithm 1 UAV visual navigator algorithm

Input: UAV images $I_U$ and satellite map tiles. Output: Estimated UAV trajectory.   1:  for all UAV images do   2:    Extract ORB features.   3:    VO initialization.   4:    Track UAV movement.   5:    if keyframe then   6:      if $!S_{gw}$ then   7:        Visual BA optimization.   8:        $T_{gi}=\mathit{CtoF}\text{\_}\mathit{Image}\text{\_}\mathit{Matching}\left(T_U^i\right)\leftarrow \mathrm{Sec}.2.2.1$   9:        Calculate the initial value of $S_{gw}\mathrm{by}\mathrm{Sec}.2.1.2\left(2\right)$ 10:        Reliability check of $S_{gw}$ 11:      else if !GeoInitialized then 12:        Visual BA optimization. 13:        $T_{gi}=\mathit{Prior}\text{\_}\mathit{Image}\text{\_}\mathit{Matching}\left(T_U^i,T_{wi}\right)\leftarrow \mathrm{Sec}.2.2.2$ 14:        Geographic initialization. $\leftarrow \mathrm{Sec}.2.1.2\left(3\right)$ 15:      else 16:        $T_{gi}=\mathit{Prior}\text{\_}\mathit{Image}\text{\_}\mathit{Matching}\left(T_U^i,T_{wi}\right)\leftarrow \mathrm{Sec}.2.1.1$ 17:        Visual-geography BA optimization. 18:      end if 19:    end if 20:  end for

2.1. Tightly Coupled Visual and Geographic Information Fusion

This paper proposes a tightly coupled visual-geographic fusion method based on visual-geography optimization, aiming to enhance the algorithm’s navigation and positioning performance by effectively utilizing both visual and geolocation information from keyframes. The section of the code is available at https://github.com/XuWeibo-code/UAV_Visual_Navigation.git (accessed on 4 June 2024).

2.1.1. Visual-Geography Optimization

The visual-geography optimization proposed in this paper is essentially a nonlinear optimization, enabling the joint optimization of visual information and geographic information.

The 3D transformation involved in the UAV’s motion primarily consists of the pose transformation matrix $T$ and the similarity transformation matrix $S$, where $T\in \mathrm{SE}\left(3\right)$ and its tangent vector $\delta \in se\left(3\right)$. Similarly, $S\in \mathrm{Sim}\left(3\right)$ and its tangent vector $\omega \in sim\left(3\right)$. The specific form of this transformation is as follows:

$$\left\{\begin{array}{cc}T=\left[\begin{array}{cc}R& t\\ 0& 1\end{array}\right]\in \mathrm{SE}\left(3\right);\hfill & \delta ={\left[\begin{array}{cc}\omega & \upsilon \end{array}\right]}^T\in se\left(3\right)\hfill \\ S=\left[\begin{array}{cc}sR& t\\ 0& 1\end{array}\right]\in \mathrm{Sim}\left(3\right);\hfill & \epsilon ={\left[\begin{array}{ccc}\omega & \upsilon & \lambda \end{array}\right]}^T\in sim\left(3\right)\hfill \end{array}\right.,$$

(1)

where $R$ is the $3\times 3$ rotation matrix, $t$ is the $3\times 1$ translation vector, and $s$ represents the scale. The components in the tangent space are denoted as follows: $\omega$ is a $3\times 1$ vector, $v$ is a $3\times 1$ vector, and $\lambda$ is a scalar representing the rotation component, translation component, and scale component, respectively.

The optimization method employs four coordinate systems, namely the SLAM world coordinate system, geographic coordinate system, local camera coordinate system, and global camera coordinate system. In this paper, $S_{gw}\in Sim\left(3\right)$ is utilized to denote the similarity transformation matrix from the SLAM world coordinate system to the geographic coordinate system. $S_c\in Sim\left(3\right)$ represents the similarity transformation matrix from the local camera coordinate system to the global camera coordinate system. $T_{gi}\in SE\left(3\right)$ signifies the pose transformation matrix from the global camera coordinate system to the geographic coordinate system in frame i. $T_{wi}\in SE\left(3\right)$ denotes the pose transformation matrix from the local camera coordinate system to the SLAM world coordinates in frame i. The variables $S_{gw}$ and $S_c$ share a common scale parameter, which can be expressed as follows:

$$\left\{\begin{array}{c}{S}_{gw}=\left[\begin{array}{cc}{s}_{gw}{R}_{gw}& t_{gw}\\ 0& 1\end{array}\right]\\ S_c=\left[\begin{array}{cc}{s}_{gw}{I}_{3\times 3}& 0\\ 0& 1\end{array}\right]\end{array}\right.,$$

(2)

The conventional VO estimates the pose of a UAV, denoted as $T_{wi}$, by optimizing the visual information through visual BA (Figure 2a). In contrast, the proposed visual-geography BA (Figure 2b) jointly optimizes the parameters $T_{wi}$ and $S_{gw}$ using both visual and geolocation information from keyframes.

First, the visual residual $r_{ij}^v$ is defined as the reprojection error of the map point j in frame i.

$$r_{ij}^v={\widehat{x}}_j-\pi \left(T_{wi}{X}_j\right),$$

(3)

where ${\widehat{x}}_j$ is the observed value of the map points, $X_j$ represents the 3D location of the map point j, and $\pi \left(·\right)$ corresponds to the camera projection function.

Then, the geographic residual $r_i^g$ of frame i is defined as follows:

$$r_i^g=\mathrm{Log}\left({\widehat{T}}_{gi}^{-1}{S}_{gw}{T}_{wi}{S}_c^{-1}\right),$$

(4)

where $\mathrm{Log}:\mathrm{SE}\left(3\right)\to {ℝ}^6$ is a mapping from the Lie group $\mathrm{Sim}\left(3\right)$ to the vector space $sim\left(3\right)$, ${\widehat{T}}_{gi}$ is an observability measure, and $S_{gw}$ and $T_{wi}$ are the optimization variables.

Given $k$ keyframes and their states ${\overline{T}}_k\doteq \left\{T_0\cdots T_{k-1}\right\}$, $l$ 3D map points and their states ${\chi }_l\doteq \left\{X_0\cdots X_{k-1}\right\}$, the cost function of this optimization is as follows:

$$\left\{T_i,X_j,S_{gw}|T_i\in {\overline{T}}_k,X_j\in {\chi }_l\right\}=\underset{T_i,X_j}{\arg \min }{\displaystyle \sum _{i=0}^{k-1}{\displaystyle \sum _{j=0}^{l-1}\left(\rho \left({‖r_{kj}^c‖}_{{\sum }_c}^2\right)+\rho \left({‖r_k^g‖}_{{\sum }_g}^2\right)\right)}},$$

(5)

where ${\sum }_C$ and ${\sum }_g$ represent the weight factors of visual residual and geographical residual, respectively. The value of ${\sum }_C$ is obtained from [13], while the derivation of ${\sum }_g$ is elaborated in detail in Section 2.1.3. For visual and geographic residuals, we use a robust Huber kernel function $\rho \left(·\right)$ to reduce the influence of spurious matchings.

To enhance the optimization efficiency of visual-geography BA, the algorithm must optimize the Jacobian matrix of the residual for the variable. Among them, the Jacobian matrix of the visual residual concerning the optimization variables $T_{wi}$ and $X_j$ can be obtained from [13]. Therefore, this paper only derives the Jacobian matrix of the geographic residual to the optimization variables $S_{gw}$ and $T_{wi}$.

In Equation (4), the geographic residual $r_i^g$ is a 7-dimensional vector, the tangent vector ${\epsilon }_{gw}$ of the $S_{gw}$ is also a 7-dimensional vector, and the tangent vector ${\delta }_{wi}$ of $T_{wi}$ is a 6-dimensional vector. The Jacobian matrices of the geographic residual concerning ${\epsilon }_{gw}$ and ${\delta }_{wi}$ are as follows:

$$\left\{\begin{array}{c}\frac{\partial r_i^g}{\partial {\delta }_{gw}}=Adj\left({\widehat{T}}_{gi}^{-1}\right)-Adj\left(r_i^g\right)\left[\begin{array}{cc}{0}_{6\times 6}& 0\\ 0& 1\end{array}\right]\hfill \\ \frac{\partial r_i^g}{\partial {\delta }_{wi}}=Adj\left({\widehat{T}}_{gi}^{-1}{S}_{gw}\right)\left[\begin{array}{c}{I}_{6\times 6}\\ 0\end{array}\right]\hfill \end{array}\right.,$$

(6)

where $Adj\left(S\right)$ is the adjoint matrix of $S$, which takes the following form:

$$Adj\left(S\right)=\left[\begin{array}{ccc}R& 0& 0\\ t_{\times }R& sR& -t\\ 0& 0& 1\end{array}\right],$$

(7)

The optimization process of this BA requires the assignment of initial values. The initial values for the variables $T_{wi}$ and $X_j$ are provided by VO, while the initial value for the variable $S_{gw}$ is calculated during the geographic initializer.

2.1.2. Geographic Initializer

The algorithm utilizes a [13]-based VO technique to track the motion of the UAV in real-time and perform a visual initializer. To enhance the real-time performance and accuracy of the SLAM, our algorithm employs a subset of representative frames as keyframes for BA optimization and image matching. During the selection process, temporal intervals, visual content disparities, camera movements, and co-visibility relationships between the current frame and other keyframes are all taken into consideration.

The algorithm then employs a rapid and robust geographic initializer, which aims to accurately estimate the initialization parameter $S_{gw}$ that aligns the SLAM world coordinate system with the geographic coordinate system. The transformation relationship between the local pose in the SLAM world coordinate system and the geographical pose is described as follows:

$$T_{gi}=S_{gw}{T}_{wi}{S}_c^{-1},$$

(8)

We state the geographic initializer as a map estimation problem, which can be divided into three sequential steps.

(1)

Visual BA optimization

The algorithm utilizes a vision-only BA (Figure 2a) to optimize the UAV’s pose and map points during the initial phase of its operation, when sufficient geolocation information is not yet available.

(2)

Calculate the initial value of $S_{gw}$

The algorithm calculates the initial value of $S_{gw}$ after receiving reliable geolocation information from the two keyframes. At this stage, the local poses of the two keyframes are denoted as $T_{w1}$ and $T_{w2}$, while their geographical poses are represented by $T_{g1}$ and $T_{g2}$. The scale component $s_{gw}$ of the similarity transformation matrix $S_{gw}$ is as follows:

$$s_{gw}=\frac{‖t_{g2}-t_{g1}‖}{‖t_{w2}-t_{w1}‖},$$

(9)

The rotation component $R_{gw}$ of the similarity transformation matrix $S_{gw}$ is as follows:

$$\left\{\begin{array}{c}{R}_{gw}=R_{g1}{R}_{w1}^T\\ {R^{\prime }}_{gw}=R_{g2}{R}_{w2}^T\end{array}\right.,$$

(10)

The translation component $t_{gw}$ of the similarity transformation matrix $S_{gw}$ is as follows:

$$\left\{\begin{array}{c}{t}_{gw}=-R_{g1}{R}_{w1}^T{t}_{w1}+s_{gw}^{-1}{t}_{g1}\\ {t^{\prime }}_{gw}=-R_{g2}{R}_{w2}^T{t}_{w2}+s_{gw}^{-1}{t}_{g2}\end{array}\right.,$$

(11)

The initializer calculates the initial value of $S_{gw}$ at this stage. It is important to note that the geolocation information obtained during this process relies on a coarse-to-fine image-matching method, which may result in mismatching. Therefore, the algorithm utilizes an error coefficient to check the reliability of $S_{gw}$. We construct $S_{gw}$ and ${S^{\prime }}_{gw}$ using $\left(R_{gw},t_{gw},s_{gw}\right)$ and $\left({R^{\prime }}_{gw},{t^{\prime }}_{gw},s_{gw}\right)$, respectively, and the error coefficient is calculated as follows:

$$r_S=‖\mathrm{Log}\left(S_{gw}{S^{\prime }}_{gw}^{-1}\right)‖,$$

(12)

The reliability of $S_{gw}$ is considered when $r_S<th$; otherwise, the algorithm will wait for the keyframe to continue receiving geolocation information of the UAV and repeat the aforementioned steps.

(3)

Visual-geography BA

When the algorithm obtains a reliable initial value of $S_{gw}$, the prior geographic pose of the UAV can be calculated based on Equation (11). At this stage, the geolocation method based on heterogeneous image matching can directly extract the satellite image according to the prior geographic pose and perform more precise fine matching. Once geolocation information has been received for N keyframes, the algorithm optimizes the UAV pose, map points, and initialization parameter S using visual-geography BA (Figure 2a) of Equation (5).

2.1.3. Map Fusion Method and Geographic Weights Update

After the completion of the geographical initialization, it remains necessary for the system to output real-time and accurate geographical pose of the UAV. To achieve this, we employ visual-geography BA to optimize both UAV poses and map points. Unlike visual-geography BA with a geographical initializer, at this stage, our algorithm does not optimize the initialization parameter.

The geolocation information obtained by image matching is subject to errors, necessitating the assignment of weights to each geographic edge in the visual-geography BA. Therefore, we propose a calculation method for quantifying the magnitude of geographic residuals.

Refine the weight of the geographic edge i in BA according to the following criteria:

$$q={\beta }_n{\phi }_n+{\beta }_p{‖\mathrm{Log}\left(T_{gc}^{-1}{T^{\prime }}_{gc}\right)‖}^2+{\beta }_e{‖e_i^g‖}^2,$$

(13)

where the balance factors ${\beta }_n$, ${\beta }_p$, and ${\beta }_e$ represent the weights assigned to different components; the variable ${\phi }_n$ represents the number of remaining inner points after removing outer points from the matching points obtained by the RANSAC algorithm [28]; $T_{gc}$ and ${T^{\prime }}_{gc}$ denote the prior geolocation information input in the image matching and the geolocation information output by image matching, respectively; and $e_i^g$ represents the geographic residual of geographic edge i in BA.

The first two terms in Equation (13) are generated during the image matching, while the last term is dynamically updated through BA. In Equation (5), $q{I}_{7\times 7}$ is introduced into ${\sum }_g$ to fine-tune the weighting of geographic residuals in the BA. By adjusting the weight of geographic edges in BA, this algorithm enables a more accurate estimation of the UAV’s geographic pose.

2.2. Geolocation Based on Heterogeneous Image Matching

The geolocation method based on heterogeneous image matching is utilized to provide precise geolocation information for keyframes. Considering the characteristics of VO, appropriate adjustments are made to the geolocation method. In the initial stage of the algorithm, a coarse-to-fine image-matching method is employed for UAV geolocation. After the algorithm calculates the $S_{gw}$, the satellite image is intercepted using the prior pose of the UAV and then directly processed by the fine matching.

2.2.1. Coarse-to-Fine Image-Matching Method

The geolocation of the UAV is achieved in this paper by implementing the coarse-to-fine image-matching method utilized in [23], even in the absence of prior geolocation information. The pipeline of the coarse-to-fine image-matching method is shown in Figure 3.

During the coarse matching stage, satellite images are preprocessed offline. These images, which have varying resolutions, are clipped into smaller map tiles and then mapped into 4096-dimensional global descriptors using NetVLAD [29]. During UAV execution, the extracted global descriptors from aerial images are also obtained using NetVLAD. The l2-norm of the differences between the aerial image’s global descriptor and every single global descriptor in the prepared array is computed and sorted in ascending order. The top 3 candidates in the sorted list are selected for subsequent fine matching.

After the coarse matching stage, the algorithm proceeds to conduct fine matching between the aerial image and the satellite image. This process involves utilizing SuperPoint [30] for extracting local descriptors from both images, generating 2D matching points between the aerial image and the candidate matching images through SuperGlue [31]. The matching result of SuperPoint + SuperGlue is shown in Figure 4.

Assuming a planar ground with zero height, these 2D-2D correspondences are transformed into a camera projection relation from the 3D coordinates in the geographic coordinate system to the 2D-pixel coordinates in the aerial image. The EPnP+RANSAC algorithm [28] is then utilized to accurately determine the geographic pose of the UAV.

2.2.2. Prior-Based Image-Matching Method

The coarse-to-fine image-matching method exhibits drawbacks, such as significant latency and a high mismatching rate. In this study, leveraging the operational characteristics of VO, we propose a prior-based image-matching approach for precise UAV geolocation.

After obtaining the geographic initialization parameter $S_{gw}$, the algorithm calculates the prior geographic pose $T_{ig}\left(T_{gi}^{-1}\right)$ of the keyframe i based on Equation (8). Assuming a planar ground with zero height, the transformation relationship between the geographic plane coordinates of the satellite image and pixel coordinates of the aerial image is described using a homography matrix $H_{ig}$.

$$H_{ig}=\frac{1}{\lambda }K\left[r_{ig1},r_{ig2},t_{ig}\right],$$

(14)

where $R_{ig}$ and $t_{ig}$ are the rotational and translational components of $T_{ig}$, $r_{ig1}$ and $r_{ig2}$ are the first and second columns of $R_{ig}$, respectively, $K$ represents the camera internal reference, and $\lambda$ denotes the depth of the feature point, which is equivalent to the absolute value of the $z$ coordinate of $\left(-R_{ig}^{-1}{t}_{ig}\right)$.

The aerial image can be transformed into a geographic plane coordinate by applying the homography matrix $H_{ig}$, extracting a rectangular region in the geographic plane coordinate system for intercepting the satellite image. Subsequently, utilizing the homography matrix, the captured satellite image is projected onto the pixel coordinate of the aerial image. Figure 5 illustrates the flowchart depicting satellite image extraction.

After extracting the satellite image, the algorithm will perform fine matching, as described in Section 2.2.1, to accurately determine the geographic pose of the UAV.

3. Experimental Setups and Results

This section outlines the experimental setup and presents the test results. To simulate real-flight conditions, we have configured the Ubuntu environment on NVIDIA Jetson Xavier NX and conducted experimental validation.

3.1. Simulation Dataset

3.1.1. Setups

To evaluate the efficacy of the algorithm proposed in this paper, we generated a simulation dataset using a custom simulation program. This program employs the camera projection principle to transform satellite images captured in visible light mode into image sequences by configuring camera parameters and executing trajectories.

The trajectory of the UAV in the simulation dataset is depicted in Figure 6, with a satellite image serving as the background. Specific characteristics of the dataset are elaborated upon in Table 1.

3.1.2. Geolocation Performance

To evaluate the performance of our proposed algorithm, we compare it with various algorithms in the simulation dataset, including the VO algorithm (the monocular vision mode of ORB-SLAM3) [13], the VO algorithm after geographic initialization, the geolocation algorithm based on coarse-to-fine image matching [23], and our proposed algorithm. The 2D errors between the different algorithms and the ground-truth trajectory are depicted in Figure 7. The trajectory obtained from the VO algorithm is converted into the geographic coordinate through a similarity transformation matrix, which is derived using multiple ground truth poses by solving a least-squares problem [32].

3.2. Real-World Dataset

3.2.1. Setups

To evaluate the performance of the proposed algorithm in a real environment, we employed the DJI M300 RTK UAV equipped with an H20 pan-tilt camera to generate two distinct datasets for visual navigation and positioning of the UAV.

The trajectory of the UAV in real-world datasets is depicted in Figure 8, with the accurate trajectory obtained by RTK represented by the red curve. Furthermore, two videos were captured from a downward-facing perspective by the UAV, and their specific characteristics are detailed in Table 2.

3.2.2. Geolocation Performance

To evaluate the performance of our proposed algorithm, we conducted a comparative analysis with various state-of-the-art algorithms using real-world datasets. These include the VO algorithm, the VO algorithm after geographic initialization, the geolocation algorithm based on coarse-to-fine image matching, and our proposed algorithm. The 2D errors between the different algorithms and the ground-truth trajectory are depicted in Figure 9.

3.3. Ablation Study

We conduct an ablation study to evaluate the contribution of each module in the proposed algorithm. The translation errors by removing different configurations are present in Table 3. We can observe that the result using the full proposed framework achieves the best performance.

3.4. Analysis of Experimental Findings

We conducted geolocation experiments in a simulated dataset and two segments of real-world datasets. Throughout the experiment, we evaluated the proposed geographic initialization method, which successfully accomplished geographic initialization within 5 s. Comparative analysis with the VO algorithm revealed that performing geographic initialization significantly enhances stability. The mean and RMSE of the different algorithms in the geolocation experiment are compared in Table 4 and Table 5. It is evident that the algorithm proposed in this paper exhibits superior accuracy, enabling precise estimation of the geographic pose of the UAV.

The geolocation method based on coarse-to-fine image matching in the experiment exhibits high accuracy. However, the output positioning data exhibit a delay of 3 s and a mismatching probability not lower than 25%, making it unsuitable for real-time UAV positioning. The proposed algorithm achieves a data update frequency of 20 Hz/s, while the visual navigation algorithm presented in this paper enhances performance. Specifically, our proposed visual navigation algorithm enables accurate and real-time estimation of the geographic pose of the UAV in GNSS-denied environments.

4. Conclusions

This paper presents a visual navigation algorithm for UAVs based on visual-geography optimization, which effectively integrates visual and geolocation information from keyframes to achieve tightly coupled geolocation. Moreover, the algorithm incorporates a heterogeneous image-matching approach for geolocation, which combines coarse-to-fine image matching and prior-based methods. Based on the experimental results, the following conclusions can be drawn:

• In the geolocation method based on heterogeneous image matching, our proposed prior-based image-matching method utilizes the prior information to enhance the accuracy and efficiency of geolocation for the algorithm.
• The fusion method based on visual-geography optimization achieves stable and reliable estimation of geographic initialization parameters within 5 s, enabling real-time estimation of the UAV’s geographic pose.
• We propose a tightly integrated fusion method that effectively combines the visual information from VO with the geolocation information obtained through the image-matching method. Experimental results demonstrate that our proposed algorithm accurately and in real-time estimates the UAV’s geolocation information solely relying on the vision sensor, even in GNSS-denied environments.

5. Discussion

Although promising results have been achieved, there are several limitations. Firstly, the proposed algorithm is more suitable for scenarios within the range of tens to hundreds of meters. For higher or lower scenarios, significant errors may arise due to the image-matching method employed in the algorithm. Secondly, to calculate the geographic pose of the UAV, we assume a planar ground with zero elevation. However, this assumption introduces a significant discrepancy in estimating the altitude of the UAV. Lastly, when confronted with low image texture or substantial variations in lighting conditions, inevitable disruptions occur in the visual navigation and positioning algorithm utilized by the UAV.

To enhance the positioning performance of UAVs in complex scenarios, our future research will focus on developing visual-inertial navigation algorithms for UAV positioning and incorporating barometers or laser rangefinders to accurately measure UAV altitude, thereby providing precise and reliable positioning information.