© The Authors 2023

1 Introduction

Vega (α Lyrae, Keoe, wāqi', Zhinü 织 女 一), one of the historical photometric standard stars (Johnson & Morgan 1953), is an A0 V star (e.g., Johnson & Morgan 1953) that is located at 7.68 ± 0.02 pc from the Solar System (van Leeuwen 2007). Despite its proximity, Vega is not included in the Gaia Catalog of Nearby Stars that are within 100 pc from the Sun, since its brightness exceeds the Gaia limits (Gaia Collaboration 2021). With an age of 445 ± 13 Myr (Yoon et al. 2010), Vega shows a prototypical mid-infrared excess in the Infrared Astronomical Satellite observations, situating it in an evolutionary stage that is between star formation and our Solar System (Aumann et al. 1984).

The two-belt debris disk system around Vega may result from planet-disk interactions under various planetary configurations (e.g., Matrà et al. 2020). In fact, most existing directly imaged planets are found in bright debris disk systems (e.g., HR 8799: Marois et al. 2008, β Pic: Lagrange et al. 2009, 51 Eri: Macintosh et al. 2015), making it more likely to find giant planets in debris disk systems than around stars without disks (Meshkat et al. 2017). Combined with the observational fact that giant planets are more likely to exist at 1–10 au from their host stars in both direct imaging and radial velocity surveys (Nielsen et al. 2019; Fulton et al. 2021), the proximity of Vega makes it one of the best systems for giant planet searches.

Before the resolved imaging of debris belts around Vega, dust structures around Vega have suggested the existence of potential planetary perturbers (e.g., Holland et al. 1998; Wilner et al. 2002) and called for the deep imaging of them. Combining the Gemini Altar adaptive optics (AO) system and the Gemini Near-infrared Imager (NIRI), Marois et al. (2006) obtained 5σ detection limits of better than 18 Mag at 3″–10″ in the off-methane 1.58 μm 6.5% filter, or ~3 M_Jupiter at 8″. Using the MMT Observatory AO and the Clio camera, Hinz et al. (2006) and Heinze et al. (2008) obtained 10σ limits of better than ~12 Mag at 2″–11″ at the M band, or ~10 M_Jupiter.

Recent observational studies on the Vega planetary system are in direct imaging, transiting, radial velocity, and (sub)millimeter interferometric imaging. In direct imaging, Meshkat et al. (2018) presented observations from the coronagraphic integral field spectrograph P1640 at the Palomar Observatory in J and H bands. Despite a nondetection of planets, they obtained 5σ planet detection limits from 0″.25 to 2″, reaching a best sensitivity of ≈20 M_Jupiter at 1″.5 or 12 au. From a complimentary approach, using the Atacama Large Millimeter/submillimeter Array (ALMA), Matrà et al. (2020) observed and modeled the outer belt from ~60 au to ~200 au. To explain the observed disk architecture, Matrà et al. (2020) discussed three mechanisms which include a single giant planet, multiple low-mass planets, and no outer planets. Combining a decade of Tillinghast Refector Echelle Spectrograph (TRES) spectra for radial velocity and two sectors of Transiting Exoplanet Survey Satellite (TESS) photometry, and under the scenario that the planetary orbits have inclinations between 1°.5 and 11°.5, Hurt et al. (2021) obtained a nondetection of 1–10 M_Jupiter planets within 1–10 au, while reporting a candidate Jovian signal with a period of 2.43 days in radial velocity measurements. Nevertheless, depending on the orbit orientations of the planets, the mass limit can vary by up to a factor of ~10 in Fig. 6 of Hurt et al. (2021).

To improve existing high-contrast imaging limits, test different mechanisms for the formation of the Vega debris disk system observed in ALMA, and explore beyond the nearly edge-on limitations from radial velocity and transit studies, we observed Vega in the M_s band using the vortex coronagraph on Keck/NIRC2. In the M_s band, planets have relatively larger planet-to-star brightness ratios than in the J or H band (e.g., Spiegel & Burrows 2012; Skemer et al. 2014), and the advantage of the M_s band over shorter-wavelength bands improves the intermediate system ages for a planet of a given mass (e.g., Currie et al. 2022), both enabling us to explore around Vega for possible cooler and less massive planets than existing studies.

2 Observation and data reduction

We observed Vega using the Keck/NIRC2 vortex coronagraph in the M_s band during two individual nights using the narrow camera with a pixel size of 9.942 mas (e.g., Service et al. 2016; Mawet et al. 2019). The first observation was on UT 2018 August 30 under program C314 (PI: D. Mawet), with the total integration time being 1991 s (=0.181 s × 100 coadds × 110 frames), and the parallactic angle change being 74°.9. The second observation was on UT 2019 August 20 under program N097 (PI: T. Meshkat), with the total integration time being 4500 s (=0.25 s ×150 coadds ×120 frames), and the parallactic angle change being 81°.6. We list the observation details in Table 1.

Faint planetary signals are often overwhelmed by the stellar point spread function (PSF) of the central star. In order to remove the stellar PSF and reveal faint surrounding signals, we first pre-processed the data using the VIP package (Gomez Gonzalez et al. 2017), which was further customized for NIRC2 vortex observations by performing flat-fielding, bad pixel and background removal, and image centering (Xuan et al. 2018). We then reduced the preprocessed data using the Karhunen-Loève image projection (KLIP) algorithm (Soummer et al. 2012; Amara & Quanz 2012), which performs principal component analysis to capture the stellar PSF and speckles in the observation. For an image, we removed its stellar PSF and speckles by first projecting it to the KLIP components, and then subtracting the projection from the original image to obtain the residuals. Astrophysical signals including planets and disks thus reside in these residuals. To obtain the final image for each observation, we obtained the residual maps for each individual readout, then rotated and median combined them. We present the combined two-epoch result using 10% of the KLIP components with angular differential imaging (ADI: Marois et al. 2006) in Fig. 1 for demonstration purposes, in which we did not identify point sources that are beyond 5σ of the noise that is within the same angular separation from the stars.

Using VIP and taking into account self-subtraction and over-fitting with ADI while using KLIP, we generated the 5σ contrast limits for each observation following Xuan et al. (2018) by varying the reduction parameters to obtain the corresponding detection limits while performing corrections for small sample statistics in Mawet et al. (2014). Specifically, VIP performs injection recovery for companions at different locations to measure the throughput from ADI and KLIP (Gomez Gonzalez et al. 2017). While measuring our throughput, we injected companions along three radial branches spread throughout the image (originating from the masked center) where the averaged throughput at each radial location in the image was determined from these multiple estimates of the throughput at different branches (Xuan et al. 2018). We computed the contrast for the entire image for each combination of inner and outer mask size and number of principal components, where we computed up to 30 principal components. The ADI reduction was performed with no rotation gap; for one combination of inner and outer radii from Table 1 of Xuan et al. (2018), the full-frame reduction was performed on the annulus zone with the region inward of the outer radius or outward of the inner radius included. For example, for an outer radius of 0″.5 and an inner radius of 0″.08, all pixels with radial separations between 8 pixels and 50 pixels from the center of the image, or between 0″.08 and 0″.5, were included in the reduction (see the reduction details in Xuan et al. 2018). While we did not use annular ADI, we did utilize the best contrast achieved from our full-frame ADI at each one-pixel annulus.

To obtain the final detection limit for each angular separation from the star with a step size of 1 pixel, we compared the detection limits from different combinations of reduction parameters. The reduction parameters include the frame size (i.e., algorithmic inner and outer radii) and the number of principal components (see Sect. 2 and Table 1 of Xuan et al. (2018) for details on computing most optimal contrasts using five different frame size combinations). Therefore, while our optimal contrast is the combination of frames processed using different parameters, we were optimizing at each radial location. For our observations, the number of principal components adopted for the final contrast curve ranges from 8 to 29, with a median of 15 and a standard deviation of 7.4.

The faintest companion that can be detected from such combinations at 5σ level was adopted as our final detection limit. For the 2019 data, the exposures that were used to image the unblocked central source were saturated, we thus fit the unsaturated first Airy ring to that of a theoretical model of the vortex stellar PSF (while taking into account the PSF broadening effects due to weather by convolving a 2D Gaussian distribution), and used the best-fit model to generate the corresponding contrast curve. We then combined our detection limits from the two observations, and present them in Fig. 2. Although the 2018 observation has a shorter total on-target exposure time than the 2019 one, their total parallactic angle change difference is only 6°.7. With the Differential Image Motion Monitor (DIMM) seeing of 0″.41 ± 0″.11 in 2018 being more stable than that of 1″.18 ± 0″.55 in 2019, the 2018 data dominate the detection limits in the combined dataset.

Fig. 1 Combined two-epoch NIRC2 image of Vega in the Ms band using ADI with 10% of the KLIP components for demonstration purposes. We did not identify point sources that were more than 5σ levels beyond the noise of a similar angular separation from the star. The pixel values correspond to lower limits of contrast values due to over- and self-subtraction with KLIP and ADI, respectively.

3 Analysis

3.1 Mass detection limits

3.1.1 NIRC2 imaging

Being the photometric standard, Vega's apparent magnitude is defined as 0 in the M_s band. Adopting a distance of 7.68 ± 0.02 pc in van Leeuwen (2007), the absolute magnitude for Vega in the M_s band was 0.573 ± 0.006.

With an age of 445 ± 13 Myr (Yoon et al. 2010) and adopting the AMES-Cond evolutionary models (Baraffe et al. 2003), we converted the contrast to 5σ mass detection limits in Fig. 2. We reached a detection limit of less than 5 M_Jupiter beyond 9 au, and 3 M_Jupiter beyond 14 au.

3.1.2 NIRC2 imaging and TRES radial velocity

Combining the 2018 NIRC2 results with the radial velocity data from TRES in Hurt et al. (2021), we followed Hurt et al. (2021) to obtain the mass limits assuming the planetary orbits are well aligned with the spin axis of Vega. To explore the detectability of companions from both direct imaging and radial velocity measurements, we randomly generated 10⁶ radial velocity samples of companion orbits following Hurt et al. (2021). The semi-major axis follows a log-uniform distribution ranging from 0.1 au to 22 au. The companion mass follows a log-uniform distribution ranging from 0.1 to 100 M_Jupiter. The sine value of orbital inclination follows a uniform distribution from 1°.5 to 11°.5. The orbital eccentricity follows a beta distribution described in Kipping (2013). The argument of periastron follows a uniform distribution ranging from 0 to 2π radian. The time of periastron passage follows a uniform distribution which was determined by the orbital period. The stellar mass follows a Gaussian distribution using the measurements from Monnier et al. (2012). In each radial velocity sample, we scaled Gaussian noise according to the uncertainties of the TRES measurements in Hurt et al. (2021). We fit a flat line to each synthetic radial velocity curve using RadVal (Fulton et al. 2018).

For all simulated radial velocity samples, we considered a synthetic signal to be detectable in radial velocity when its p-value was < 0.001 (i.e., 3.3σ), while ignoring correlated noise (Hurt et al. 2021). At a specific radial separation from the star, we further required that a companion be detectable when its mass is above the detection limit in Fig. 2.

We present the detection probability of companions from the above injection-recovery procedure in Fig. 3. The detectability of companions from combined radial velocity and direct imaging follows two trends as a function of stellocentric separation. Interior to ~4 au, the detectable planets increases with radial separation, approaching planets with ~8 M_Jupiter down to ~1 M_Jupiter at ~0.1 au. Exterior to ~4 au, the detectability is dominated by NIRC2 imaging, reaching down to ~2 M_Jupiter at ~22 au. Under the Hurt et al. (2021) framework, there is a possible nonabsolute detection of companions near ~22 au, which is limited by the NIRC2 field of view in our study, since the sampled orbital eccentricity adopted from Kipping (2013) can position planets with a semi-major axis less than ~22 au beyond the 22 au angular radius.

Fig. 2 Detection limits of point sources around Vega in the Ms band. Left: 5σ contrast, in comparison with the JWST/NIRCam F444W on-sky contrast calculated from Carter et al. (2022). Right: 5σ mass detection limit, and expected mass detection limit with NIRCam F444W. We note that the NIRCam F444W values have an assumed identical instrument performance in the JWST/ERS-1386 program (see Sect. 3.3).

Fig. 3 Detection probability of point sources as a function of the semi-major axis of point sources for Vega, using a combination of the 2018 NIRC2 observation and the TRES radial velocity measurements from Hurt et al. (2021) (see Sect. 3.1.2). We can reach an ~8 MJupiter detection limit at ~4 au, while probing down to ~2–~3 MJupiter at 0.1 au and 22 au, respectively.

3.2 Disk formation from mass limits

The ALMA observation in Matrà et al. (2020) resolves the outer dust belt of Vega extending from ~60 au beyond ~150 au. To explain the observed planetary system architecture, the authors have analyzed the two scenarios that involve planets: either a chain of small planets within 70 au with mass ≳6 M_⊕, or a sole ~5 M_Jupiter at 50–60 au.

The combined Keck/NIRC2 M_s-band and TRES result can exclude the existence of ~8 M_Jupiter planets from 0.1 au to 22 au. Despite the fact that with an age of ~400 Myr there is no clear brightness difference between the hot-start and cold-start models (e.g., Spiegel & Burrows 2012), the observations presented here cannot rule out the sole giant planet which resides at 50–60 au in Matrà et al. (2020) for Vega.

For future exploration of the proposed sole giant planet at 50 au to 60 au using Keck/NIRC2 in the M_s band, a field of view that is between 6″.5 and 7″.8 is needed. With a pixel scale of 9.942 mas, this corresponds to a half width between 654 pixels and 785 pixels, which exceeds the current 512 pixel half width of the narrow camera for NIRC2 and such an observation is thus not achievable. Alternatively, NIRC2 offers sampling setups that have pixel sizes of 20 mas or 40 mas, yet these setups are not feasible for M_s-band imaging due to the corresponding increased sky background. Nevertheless, the planned upgrades of the NIRC2 detector electronics may permit faster readouts to enable M_s-band imaging for such purposes.

To explore far separation imaging for Vega using the existing confirguration of NIRC2, either offsetting the vortex center away from the center (e.g., the observation of HR 8799 b using the Gemini Planet Imager in Wang et al. 2018) of the NIRC2 narrow camera, or executing observations without the vortex coronagraph, could enable a half width of 785 pixels to test the sole giant planet scenario. With this 1024 × 1024 pixel field of view, the minimum permitted readout time is 0.18 s¹, which could enable a nearly identical exposure sequence in the 2018 dataset presented here. Given that extra parallactic angle change is needed to cover the entire field of view for these setups, they are beyond the scope of this study.

3.3 Implications for JWST observations

At an age of 445 ± 13 Myr for Vega (Yoon et al. 2010), giant planets with several Jupiter masses do not have clear brightness distinction between different formation models in the M band (e.g., Fig. 7 of Spiegel & Burrows 2012). For planets with less than ~5 M_Jupiter, their brightness is expected to peak at 4 μm to 6 μm (e.g., Fig. 6 of Spiegel & Burrows 2012).

To image planets that are brightest at these wavelengths, we applied the Keck/NIRC2 M_s band which operates at a central wavelength of 4.67 μm with a bandpass of 0.24 μm². In comparison, the NIRCam instrument onboard the James Webb Space Telescope (JWST) can cover nearly half the expected brightest wavelengths with its F444W filter (4 μm to 5 μm)³. By reaching a multi-Jupiter mass limit for 10 au to 20 au within this study, and one Jupiter mass or better beyond 20 au with NIRCam (Meshkat et al. 2018, Fig. 5 therein), we can combine Keck/NIRC2 and JWST/NIRCam to reach the deepest planet detection limit to investigate the planetary architecture for the Vega system.

Using the on-sky JWST/NIRCam F444W contrast curve from Carter et al. (2022) in the JWST ERS-1386 program (Hinkley et al. 2022), we calculated the expected on-sky contrast for Vega with NIRCam F440W in Fig. 2 for JWST GTO-1193 observations as follows. Assuming an identical instrument performance, which is an optimistic estimation given that JWST GTO-1193 uses MASK430R since it has a larger inner working angle--and thus lower throughput--than MASK335R in the JWST ERS-1386 observations in Carter et al. (2022), we scaled the exposure times with the SUB320 subarray (i.e., Observations 35 and 36) from the JWST GTO-1193 observations and we recalculated the contrast based on the exposure time difference. We then converted the JWST/NIRCam F444W contrast to point source mass following Carter et al. (2021) for Vega while adopting the same age and apparent magnitude as for NIRC2.

In terms of reaching nominal contrast, the Keck/NIRC2 M_S - band observations perform better than NIRCam F440W within 1″ and reach a similar quality in exterior regions. In terms of mass detection limits, the Keck/NIRC2 M_S-band observations perform systematically better than the NIRCam F444W under the abovementioned optimistic assumptions. In addition, the degradation of contrast close to the transmission near the edges of the coronagraphic masks⁴ from MASK335R in JWST ERS-1386 to MASK430R in JWST GTO-1193 has been ignored. All three aspects demonstrate that the NIRC2 M_S-band observations presented here establish the deepest high-contrast imaging exploration of planetary companions for Vega in the probed regions. Nevertheless, we emphasize that NIRCam should be better than NIRC2 for most other systems, since the study presented here should be the best-case scenario for ground-based M_S-band imaging due to the brightness of Vega; for other targets that are fainter than Vega, they should have less favorable contrasts due to the relative background levels on the ground from NIRC2.

3.4 Planet detection toward 0.1 arcsec

To obtain a more general understanding of planetary existence, the detection of intermediate separation planets near ~0″.1 are expected to yield the most thorough discovery of planets (e.g., Nielsen et al. 2019; Fulton et al. 2021). In fact, for the detection of far-separation planets, existing direct imaging surveys equipped with extreme AO systems have experienced a degradation of contrast close to the central sources (e.g., Nielsen et al. 2019; Vigan et al. 2021; Xie et al. 2022). In comparison, for the detection of close-in planets, existing radial velocity surveys have less completeness for long orbital period planets (e.g., Wittenmyer et al. 2020; Fulton et al. 2021). As a result, in the detection probability map of companions for Vega in Fig. 3, there is a clear gap for middling separations.

To fill the gap, the concept of the vortex fiber nuller (VFN; Ruane et al. 2019; Echeverri et al. 2020) is designed to detect and characterize planets near and within 0″.1. Using nulling inter-ferometry in the near-infrared, VFN suppresses on-axis starlight while retaining off-axis companion light with acceptable loss, which increases the signal-to-noise ratio for companions and thus better enables their detection and characterization. The VFN has already been installed for the phase 2 development of the Keck Planet Imager and Characterizer (KPIC; Jovanovic et al. 2020, Echeverri et al., in prep.) on the Keck Observatory, providing access to planets between 30 and 90 mas in the K band. Furthermore, the limitation of VFN in localizing the companions in their orbit will be further resolved with the concept of photonic lantern nuller (PLN; e.g., Xin et al. 2022). We expect that the application of VFN and PLN will fill the planet detection and characterization gap between direct imaging and radial velocity. For Vega, they will better explore possible hidden planets toward ~0″.1 in Fig. 3. Nevertheless, the limitation of VFN in the K band is that it is less sensitive to evolved and cooler giant planets due to the drop in planet brightness in shorter wavelengths.

4 Summary

We report 1.8 h of M_s-band imaging observations of Vega using the Keck/NIRC2 vortex coronagraph. Despite a nondetection of companions, we have pushed the mass detection limits from existing high-contrast imaging observations in Meshkat et al. (2018) by nearly an order of magnitude smaller (see Fig. 2). Combining the NIRC2 results with an existing radial velocity study using TRES in Hurt et al. (2021), we can confidently rule out companions more massive than ~8 M_Jupiter from 0.1 au to 22 au for Vega. Within this range, NIRC2 can reach planets that are less massive than 5 Mjupiter beyond 9 au.

While the NIRC2 observations presented here are sensitive to planets of 5 M_Jupiter at 9 au down to 3 M_Jupiter at 22 au, it is limited by the field of view and thus cannot be used to test the scenario in Matrà et al. (2020) that one multi-Jupiter mass planet at 50 au to 60 au is responsible for the planetary system architecture. Alternatively, observing without using the vortex coronagraph while performing multipoint dithering⁵, combined with extra parallactic angle change, may provide enough sensitivity to image such a perturber for the Vega system by pushing deeper than Heinze et al. (2008) for these separations.

Combining Keck/NIRC2 in the M_s band and NIRCam in its F444W filter assuming identical performance with JWST from ERS-1386 observations (Hinkley et al. 2022; Carter et al. 2022), we can reach comparable detection limits of companions interior to 20 au. Interior to ~10 au, although companion imaging limits increase from ~5 M_Jupiter at 10 au to ~100 M_Jupiter at 2 au in Fig. 2, this region is where radial velocity measurements can provide multi-Jupiter detection limits or better as shown in Hurt et al. (2021). Future works following Mawet et al. (2019) and Llop-Sayson et al. (2021) in combining measurements including direct imaging here and radial velocity in Hurt et al. (2021), as well as upcoming imaging observations with JWST GTO-1193, in addition to the VFN and PLN concepts to detect planets toward ~0″.1, could allow us to obtain the most holistic understanding for the planetary system of this historical photometric standard star – Vega.

Acknowledgements

We thank the anonymous referee for their constructive comments that increased the clarity and reproducibility of this paper. This research is partially supported by NASA ROSES XRP, award 80NSSC19K0294. B.B.R. has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (PRO-TOPLANETS, grant agreement No. 101002188). É.C. has received funding from the European Research Council (ERC) under the European Union's Horizon Europe research and innovation programme (ESCAPE, grant agreement No 101044152). Some of the data presented herein were obtained at the W.M. Keck Observatory, which is operated as a scientific partnership among the California Institute of Technology, the University of California and the National Aeronautics and Space Administration. The Observatory was made possible by the generous financial support of the W.M. Keck Foundation. The authors wish to recognize and acknowledge the very significant cultural role and reverence that the summit of Maunakea has always had within the indigenous Hawaiian community. We are most fortunate to have the opportunity to conduct observations from this mountain. Part of the computations presented here was conducted in the Resnick High Performance Computing Center, a facility supported by Resnick Sustainability Institute at the California Institute of Technology.

References

All Tables

All Figures

	Fig. 1 Combined two-epoch NIRC2 image of Vega in the Ms band using ADI with 10% of the KLIP components for demonstration purposes. We did not identify point sources that were more than 5σ levels beyond the noise of a similar angular separation from the star. The pixel values correspond to lower limits of contrast values due to over- and self-subtraction with KLIP and ADI, respectively.
In the text

	Fig. 2 Detection limits of point sources around Vega in the Ms band. Left: 5σ contrast, in comparison with the JWST/NIRCam F444W on-sky contrast calculated from Carter et al. (2022). Right: 5σ mass detection limit, and expected mass detection limit with NIRCam F444W. We note that the NIRCam F444W values have an assumed identical instrument performance in the JWST/ERS-1386 program (see Sect. 3.3).
In the text

	Fig. 3 Detection probability of point sources as a function of the semi-major axis of point sources for Vega, using a combination of the 2018 NIRC2 observation and the TRES radial velocity measurements from Hurt et al. (2021) (see Sect. 3.1.2). We can reach an ~8 MJupiter detection limit at ~4 au, while probing down to ~2–~3 MJupiter at 0.1 au and 22 au, respectively.
In the text