How to color the legend labels in ggplot [duplicate]

Question

Using the iris dataset, we can make a boxplot and customise the legend when plotting using ggplot like so:

ggplot(data = iris, aes(x=Species, y=Sepal.Length, fill=Species))+
  geom_boxplot()+
  scale_fill_manual(values = c("orange","purple","brown"),
                    labels=c("Setosa","Vesicolor","Virginica"),
                    name="Species\n Name")

To generate the output:

However, I want to modify the legend so that the fill icons of the boxplot are gone and instead, only species name is shown with the text appearing in the corresponding custom colour. So in this example, I want the text 'Setosa' to appear in orange legend text, 'Vesicolor' to appear purple, and 'Virginica' to appear brown.

Is there a tidyverse-friendly way to do this without declaring other objects that store the custom colour? or something within pipes?

I searched Overflow for solutions, but couldn't find any relevant to what I'm looking (Searched ggplot legend text color in the search bar).