Your definitions of your terms will ultimately decide about the answer, and they are far from clear here.
But indeed, your sentiment can be understood as correct: The difference is trivially there (as we can differentiate) and nevertheless cannot be rigorously proven. Let me explain why ...
(There are other philosophical positions out there, but I consider all that I know of either metaphysically or logically incoherent)
Long (Sellarsian) answer
As you point out, light of different frequency - i.e. physical differences (I omit the technicalities here like that it is rather the absence of a frequency and that it is a combination of the signals of different rods etc. pp.) - may lead to different "feelings", but it is all but clear how they relate to colours.
A philosophical account of your sentiment can be found in the epistemology of Wilfried Sellars. There are at least three aspects of this relevant for your question:
First: Colours do not "exist" (but as perceived qualities)
As this answer correctly points out, Sellars thinks that science (and he probably means 'natural sciences' and not 'cultural sciences' here) determines what is and what is not - and qualia like colours are not part of the "scientific image" referred to in the answer. This understanding of "existence" is closely linked to the metaphysical branch of ontology, i.e. it does not preclude that it is part of our everyday experience and pragmatic context. In other words: Colours do not have independent being and only exist in the sense that they are our way to perceive certain attributes of physical things.
Second: We surely experience different colours and need an explanation for that
Colours are part of our "manifest image", i.e. common sense talk about the world. We experience the grass as green and the sky as blue as a matter of fact. Now, how is this to be explained? Sellars uses "sense impressions" for that (the following references are to his Science and Metaphysics: Variations on Kantian Themes).
Sense impressions have the following properties:
- They are (mental?) states of the perceiver.
- They are non-conceptual, i.e. they are outcomes of our sensual apparatus (receptivity) without intake from our cognitive faculty to think and speak, but can be described by minimal physical vocabulary (like "red rectangular shape", "white hemispherical surface"). (§§36-37,55)
- They "guide" (even minimal) conceptual representations without being an analysable "part" of those conceptual representations. (§39) This is one of his main points: Intuitions proper, i.e. apprehended, conceptual representations are in the space of reason (which is not to be assimilated into the physical or non-conceptual and vice versa) and can only occur (and constitute knowledge) in a person accomplished in language usage, see esp. his Empiricism and the Philosophy of Mind.
- Even if they "are" not red, blue, or rectangular, they have attributes systematically analogous to these (conceptual) properties. (§47)
To summarise:
physical objects bring about states of
perceivers which have attributes systematically analogous to
perceptible colour and shape, without literally having perceptible
colour and shape (Science and Metaphysics, §47, p. 19)
This means, in the context of your question, that while there surely are states of the perceiver that correspond to colours without being colourful themselves (like inner images/perceivable representations/sense data), there "are" no colours, neither in the physical reality out there, nor in sense impressions. Colours "are" only insofar conceptual representations brought about by a combination of sensual input and conceptual understanding, sorting, and interpretation of this input are "conceived" as being of a colour, i.e. it is a conceptual attribute (logically) ascribed as a property:
neither the sense
impression of a red rectangle nor the conceptual representation of
a red rectangle is either red or rectangular (§52, p. 20)
In other words: Even if we perceive and attribute something as "being-red", this does not mean that "it" really "is" red in a material sense, but rather that we learned to (linguistically) respond to a certain attribute of "it" that stands in a certain systematic relation to the real physical object out there by ascribing "redness" to "it".
Third: What does all this mean for your "proof"?
As the short introduction into Sellars' epistemology has (I hope) shown, a difference in (visual) perception can only be understood if we presume a difference in sense impressions, which in turn means a difference in physical circumstances (either of the perceptual apparatus or the environment).
But the relations postulated here (physical object - sense impression - conceptual representation) are merely systematical and analogous relations, and not ones of (necessary, lawful) causation - that is exactly why we can "err" in perception. But in the end, the epistemological theory laid out here and by your description of "difference in feeling" is only that: A theory. (see §§53-55, pp. 21-22)
This theory is there to explain colour perception, i.e. the logic of the language we use talking about perception, its conditions and limits. Theoretical entities in discourses about the "manifest image", i.e. our everyday understanding and experience, are different from the theoretical entities in (natural) sciences - ultimately physical entities - in that they are metaphysical, i.e. in a certain sense not able of proof.
Therefore, only the simple fact that you can distinguish colours in perception "proves" that there is, in some sense, a difference in visual perception. Sellars agrees with your sentiment that there is no further proof and describes the "mature" theory (in reference to what Kant should have said) as follows:
If, per impossibile, Kant had developed the idea of the manifold of sense as characterized by analogical counterparts of the
perceptible qualities and relations of physical things and events he
could have given an explicit account of the ability of the impressions of receptivity to guide minds, endowed with the conceptual
framework he takes us to have, to form the conceptual representations we do of individual physical objects and events in Space and
Time. He could thus have argued that when on a certain occasion
we come to have an intuitive conceptual representation that this
green square adjoins that red square, we do so by virtue of having
a complex of non-conceptual representations which, although
non-spatial and without colour, have characteristics which are the
counterparts of *square, red, green and adjoining, and which make
them such as to account for the fact that we have this conceptual
representation rather than that of there being a purple pentagon
above an orange ellipse. (§78, p.30)
Remark
Since Kant, one of the main purposes of philosophy has been to discuss the coherence of metaphysical theories in the sense developed here. His main contribution to philosophy was the insistence that only those metaphysical entities that can consistently (and explicitly) be related to the real physical world are part of metaphysics proper.
That is why Sellars criticised many theories of perception of his time as they thought the transition from brute physical fact to meaningful perception as "given" or - in other words - mixed (contents and language of) the space of physical reality (brute fact) with (contents and language of) the space of reason (knowledge, inference, meaning), producing inconsistencies.
Fun-fact: Even people acquainted with Sellars (even citing him!) re-do the same philosophical mistakes up to contemporary philosophy, see e.g. John McDowell, 'Avoiding the Myth of the Given', in: Having the World in View: Essays on Kant, Hegel, and Sellars, Cambridge, MA: Harvard UP, 2013, pp. 256-72. He basically thinks the Myth avoidable if we only understand that it was (sheer, ineffable) content of receptivity that is part of and unified in our intuitions - by our understanding. He falls back behind Sellars with that sloppy analysis.
TL;DR
By asking for a (presumably scientific) proof for a difference in visual perception that is not part of the talk about visual perception, you would make a category error, asking from something that is bound by, but not identical with, the physical world (language of perception, common sense talk) that it should follow the same laws and logic (here: being able of proof) like it:
In everyday talk and common perception (special conditions aside), we obviously can distinguish between colours and talk about objects having different colours successfully in intersubjective discourse. We can only do so if we perceive these objects differently regarding something that makes us attribute different colours. Therefore, asking for further proof in this context indeed seems odd.
In 'hard' science, talk about "colour" is either nonsense or analogous, i.e. "proving" a difference in colour perception in this context is impossible as there is no such thing as colour perception here, only physical states of neurons, photons with a certain frequency, etc.
Judging across these very different discourses, like "There is no brown table over there, there is only a lot of void with loads of quarks forming something we could call "molecules" exerting forces on each other and absorbing and re-emitting certain frequencies of electromagnetic radiation over there!", is what philosophy calls "category error".
Therefore, no other kind of "proof" is possible as all there can be are analogous relations, not lawful or necessary ones.