If the Catholic God knew Lucifer would come out to be a disaster, and he did know, since he is omniscient, why he didn't he simply not create Lucifer and then create something better?
I know this might not belong here, but this place seems better for understanding theology to its core and not superficially.
The parsing of the question raises the issue of middle knowledge. Also at stake is the matter of the origin of evil; the SEP also includes an extensive, relevant article on evil in general. Carl Jung's Answer to Job is another hauntingly insightful source of reflection on this topic.
Now, offhand, the ChristianitySE might be a "better fit" for the question; BiblicalHermeneuticsSE is another option. The Catholic Church has a rich dialectical tradition, however, and I would suspect that you'd have a higher chance of receiving an informed Catholic response on the former SE rather than the latter (scriptural hermeneutics are more a preoccupation of Evangelical and Reformed Christians, I think; hence answers on that SE would tend to be shallower than you're looking for).
But at any rate, I went through a long Catholic phase (my personal faith story is sort of like the reverse of Father Elijah's from Michael O'Brien's Children of the Last Days books), so I will offer my sustained answer on the basis of my familiarity with Catholic demonology/hamartiology.
The question of why Satan fell is unsettled, in this context. The gist of the problem is:
And in the first place what was the nature of the sin of the rebel angels? In any case this was a point presenting considerable difficulty, especially for theologians, who had formed a high estimate of the powers and possibilities of angelic knowledge, a subject which had a peculiar attraction for many of the great masters of scholastic speculation. For if sin be, as it surely is, the height of folly, the choice of darkness for light, of evil for good, it would seem that it can only be accounted for by some ignorance, or inadvertence, or weakness, or the influence of some overmastering passion. But most of these explanations seem to be precluded by the powers and perfections of the angelic nature. The weakness of the flesh, which accounts for such a mass of human wickedness, was altogether absent from the angels. There could be no place for carnal sin without the corpus delicti. And even some sins that are purely spiritual or intellectual seem to present an almost insuperable difficulty in the case of the angels.
How could Satan, if he was the highest and hence wisest of the angels, possibly have been foolish enough to think that he could usurp God? Even equality with God should have seemed impossible, in his eyes; much less, then, superiority over the divine nature.
One could take this for a demonstration of the absurdity of the doctrine of Satan, then. Or one could invoke other passages from the Catholic textual tradition to look for a deviant solution to the problem.
For one, then, the Catholic Bible says that God did not create death, and that death entered the world through the envy of the Devil, and that death does these kinds of things modulo a covenant that it makes with the wicked. Also, Jesus Christ mysteriously asserted that blasphemy against the Holy Spirit is the "eternal sin," so arguably, that must've been the sin that Satan committed par excellence. Moreover, the Catechism of the Catholic Church says that the Spirit's lowkey self-presentation is an example of divine humility. Putting all these strands together, I think we can form the following narrative picture:
God is not the only uncreated being. God is the uncreated Creator/Creatrix, but there is also an uncreated Destroyer, Apollyon, Death with a capital "D." Though not as powerful as God, Apollyon is mythopoeically cognate with beings like Apophis and the Typhon from other demonologies. As far as Christianity goes, it is Death ("and" Hell) that is cast into the Lake of Fire after Satan is (so says the Revelation timeline), and Paul says that death is the last enemy to be vanquished. The Catholic Bible's footnotes indicate that it is Apollyon who binds Satan during the Millennium.
So God cannot prevent all evil by merely wishing so: Apollyon is at least powerful enough to impend darkness upon the creation by its own nature. Satan wants to prove himself to God, prove himself faithful, by fighting evil. Satan knows that God is humble, and that humility is relevant to fighting evil. Satan's original mistake is thinking that he can be more humble than God, then. Satan was one of the seven angels "who stand before the Lord," who are of "the sevenfold Spirit of God," i.e. whereas any human can be indwelt by the Spirit, it is the seven highest angels who alone, of all the angels, were granted to be indwelt. So Satan thought to use the power of the Holy Spirit, to fight evil in God's place, and in this, Satan blasphemed the Spirit (rather like Peter disrespected Christ by complaining about the plan of the Crucifixion, prompting Christ to rebuke Peter under the moniker of Satan).
In turn, Satan lost his indwelling by the Spirit and turned to Apollyon, making a covenant with Death, by which Death's power broke through the barrier around the creation, and so "entered the world." Again, God cannot simply wish away the Destroyer's power, but must undertake a genuinely costly plan of redemption and salvation for the creation's sake, as such.
And because Apollyon is sufficiently independent on God's wishes, perhaps God could not have ever been guaranteed to create only beings who did not fall as Satan did. Perhaps in every set of beings God might have created, there would have been one who originally fell, setting in motion the terrible chain of events just storyboarded above. Or perhaps out of love, God is willing to create every possible living being at some time or other, regardless of what those beings will do.
It would depend on at least the following:
1: Did the creator have free will? (It is possible to knowingly create something without having control over the decision to create something).
2: Did the creator have any concept of morality/evil/goodness? (Ie. did the creator knowingly create the extremely evil being, unaware that it was in fact evil?).
3: Did the creator's concept of morality match ours? (Or does, for example, their concept of morality attach virtue to evil acts?).
4: Did the creator being believe - at the time of creating the evil entity - that a good greater than the ongoing absence of evil would be achieved?
There are likely many more considerations, but the above might provide some impetus for further ideas.
See also:
Theodicy: the "answer to the question of why God permits evil", and
The Problem of Evil, "the question of how to reconcile the existence of evil and suffering with an omnipotent, omnibenevolent, and omniscient God".
Yahweh originally was one God of many, "the God of the Jews", not the "God of the universe". He exists in time, he makes mistakes, has regrets, and changes his mind. For most of the Old Testament, Yahweh is not wise, moral, or omniscient. The Omni-God theology was developed a millenia or more later, and by then the texts were already fixed. So, from an Omni-God POV, there are a lot of passages of the OT that make no sense.
Lucifer, in the OT, was also not the evil entity of the NT. Judaisim was strongly influenced by the Ditheism of Zoroastrianism in the period in between, and the NT Lucifer is a lot closer to a ditheism view of mortal moral combat, than the casual dialogs of the OT.
There ae a variety of ways that Christians can respond to this feature of OT writings. One of the things that philosophy of science has pointed out is that all claims are infinitely amendable. Biblical literalists can always find ways to explain away any apparent contradiction in this text. The Catholic Church tends to prefer to treat OT passages as containing "partial" truths, which reveal important aspects of our universe and God, and focus on those "truths" and accept that some aspects of those passages are "stories" not "truth". Some Christians considered the character of Yahweh, and of Jesus's God, to be different, and treated the two as two different beings. The holders of this last views were mostly murdered in a variety of religious pogroms by Catholics over the several millennia of Christianity. There are no doubt more ways for Christians to respond to the challenges of these passages than the three noted above.
The greater question of how Christianity can deal with the creation of morally flawed entities, is a lot easier with Lucifer than with humans, and life itself. Angels can be postulated to be given absolute free will, and a simple psychology with no moral predispositions, hence they could equally decide to be evil or good. The "free will" rationale about angels works, sort of. That demonics can tempt/influence us to be worse than we would in their absence -- which IS part of NT text -- is harder to justify with "free will".
HUMANS have complex psychology, and have predispositions, and our predispositions tend to make us selfish, and highly temptable. A creator God (assuming guided evolution) COULD have created us with much more moral predispositons, but did not. This is a problem for "free will" justifications for human evil in our world.
Even worse for the "free will" argument is the intrinsic nature of life, that life multiples to overpopulate any environment, creating scarcity, and resulting need to destruction of other life for any life to survive. Life is structured to suffer (scarcity) and be brutal and cruel. A creator God could have avoided scarcity, and brutality, with a better design of the world. Free will justifications don't help with this problem.
The various narratives involved in the various versions of Christianity are, whatever else they may be, narratives. What do I mean?
They are stories. They have their own internal logic, for suitable values of the word "logic." They are not only compelling on the basis of being logical or self consistent. We move past these little details because we see the dramatic aspects.
The Little Drummer Boy is seen as a character that Jesus would favor because of the heartfelt nature of his drumming, not because of any innate value of the drumming. Rump-a-tum-tum. He's doing it for good (dramatic and emotional) reasons. So he gets rewarded.
The thing we have to add to the mix to be able to "go along" with the God story is this. God knows the ending and we do not. God is the author of the story. If you have faith, you trust that the ending will "pay off" the plot.
So when something happens that seems to paint God in a bad light, you have to expect the ending will "make it all right." We are going to get "played out by music." There will be a Happy Ending. The middle part of the story will be very dark and the heroes will be in peril you just wouldn't believe. But by being Faithful and True they will win in the end.
So Lucifer is required to make the story have an exciting middle. God had to walk through the mines of Moria, which means Satan had to put the balrog in there. God had to storm the beaches of Normandy, meaning Satan had to train up the Nazis. God had to catch the final pass at the end of the football game, meaning Satan had to make the other side's coach what he was. God had to win the big talent competition, so Satan had to make the other side's team really good but not better than the home team. Because if God didn't do all that stuff, then he would not have grown (in our understanding) enough for the end of the story to make sense and be fulfilling.
It may not be good theology. It may not be good formal logic. It may not be logically self consistent. But it's a good story.
