How to get leading whitespace in matplotlib labels (legend or xlabel etc.) using latex?

Question

I would like to insert some leading whitespace into some of my legend labels (problem is the same for other types of labels) with latex turned on. However, any leading whitespace that I enter is being ignored (see MWE and image below). An ugly workaround would be to enter a leading . (inspired by this Q&A). That's not pretty, though.

Any idea how I could get this to work properly? Can it even be done, or should I look into raising an issue with matplotlib?

MWE:

import matplotlib.pyplot as plt
plt.rcParams['font.family'] = 'serif'
plt.rcParams['text.usetex'] = True
plt.rcParams['mathtext.fontset'] = 'cm'

fig = plt.figure(figsize=(4, 3))
plt.plot([], label="~ tilde")
plt.plot([], label="\, comma")
plt.plot([], label="\quad quad")
plt.plot([], label="\hphantom{whitespace} hphantom")
plt.plot([], label=".\hphantom{whitespace} ugly leading dot")
plt.xlabel("\hphantom{whitespace} xlabel")
plt.legend()