Dennis's answer touches upon this, but I wanted to write a separate answer to make it more explicit:
Objective scoring is essential to drive competition, and competition inspires creativity.
Here is how a challenge works that doesn't have objective scoring: you come up with a solution that is valid, and may or may not do reasonably well. You think about how you can improve its quality (whatever "quality" means in that case), and if you can, you do. At some point you'll think "yeah that looks reasonably good, I'll go with this". And that's the end of it.
Here is how a challenge works that has an objective scoring: you come up with a solution that is valid, and may or may not do reasonably well. However, you have a pretty good idea how you can or cannot improve that score, and if you can, you do. At some point you'll think "yeah that looks reasonably good, I'll go with this". But then you'll see the other participants' scores and notice that you're only a few "points" behind (whatever those are in the specific challenge). And suddenly you've got a very specific goal: come up with something to beat the other score. That's a tangible goal and a motivation to try new things, and it eliminates the problem of thinking "yeah I think this is as good as it can be" before that's really the case.
And the interesting thing that the score alone is enough to create this motivation - there doesn't even need to be competition. As soon as you've got a score, you've got your own score to beat. This applies a) when your solution is the only solution to the challenge (so far), b) your solution has such a bad score that it's nowhere near competitive (e.g. because you're participating in a code golf with C#), or c) your solution is by far the best solution already and none of the other submissions currently provide any real competition.
A recent example of that would be jimmy23013's mad golfing session to shorten his self-matching regex. There is currently no competition on that challenge, but the desire to find an even shorter solution has been enough to come up with ever new approaches (5 or 6 and counting), and explore different solutions, which has also led to a much more elegant solution in the end. If the challenge hadn't been code golf, I'm pretty sure the answer would have remained as the first working solution.
Although the code golf aspect (as an example, since it's our most common challenge type) is often orthogonal to the actual problem at hand, it's what drives us to think outside the box and look into new approaches to the given problem, because "just any solution" isn't good enough.
If I compare this to Project Euler, which is essentially a site that hosts similar (although harder) programming challenges without a scoring/winning criterion, I always move on to the next problem once I've found a solution to one of them. Especially, on problems that happen to be just feasible with a naive approach, I don't learn anything from them, because there's no motivation to look at the problem from any possible angle.
Of course, not having any scoring at all is the extreme opposite. There's also the option of having subjective winning criteria as Dennis mentioned. But that isn't any better: if I can't directly measure the the quality of my solution, and compare it against other solutions on a fixed scale, how do I know that another approach will actually improve my solution? In the end, I'll still think "yeah this looks good, I hope the OP likes it better than the other solutions".
All in all, I think people who look down on code golf overestimate the importance of the specific winning criterion at hand. It's just that code golf works particularly well, because it's easy to measure, easy to compare, provides a wide variety of potential scores, and cannot be gamed (save for exploiting loopholes). But in the end, amazing code golf solutions aren't amazing because they're short, but because the author really looked into the problem from all angles and came up with a novel approach that blew everyone's mind. (This also means that the "best" (subjective) answer around here isn't always the shortest, but more often than not the striving for shortness is what inspired the answer.)
That said, I can understand if people would like to see other objective winning criteria being more prominent, or things like fastest-code being easier to run on this site. And I'm all up for not being a pure code golf site, exploring new (objective) scoring systems, and doing more generic code-challenges. But being objective is most important trait of any of those challenges, because without that, I'm sure many of us would lose the motivation to spend as much time on a single problem as we do sometimes.
