This is essentially not a problem that a single instructor can solve on their own.
- The current top pedagogical theory in these cases is to provide "corequisite" or some kind of outside supplemental support to help the students who need to catch up. (IMO, the in-practice results of this have yet to be proven.)
- I would look closely at requirements for the next courses to follow (if any), and where students need to be at the end of your course to succeed at the next step.
- You should consult with colleagues, chair, program coordinators, etc., on the expected proficiency level at the end of your course, historical and expected pass rates, how the institution responds to high pass or fail rates, etc.
- Once this requirements-gathering phase has been done, my core advice would be to set reasonable standards for success (in your class and to enter the next level), and hold fairly to those expected standards.
- Give test questions which demonstrate the basic, essential, expected skills. Avoid questions which are overly clever or "interesting" from the instructor's POV.
Ultimately if the context and institution are letting low-skills students into your courses, who cannot possibly succeed at the expected level, and there aren't the needed outside supports to help, then the individual instructor cannot fix this. Sometimes the job is to give students the taste and opportunity to try a certain step, and give the difficult lesson that they're not ready to digest it at the current time. There's a passage from Ken Bain's What the Best College Teachers Do that sticks with me:
Susan Wiltshire, a classics professor at Vanderbilt, captured a
sentiment we heard often. Her classes, she explained, were in her view
like a great meal she had prepared, and she simply wished to invite
her students to the dinner table.
Ultimately, and especially in a scaffolded STEM discipline like mathematics, the instructor needs to maintain certain standards for students to succeed at the next level. If the institution is funneling students into a situation where they can't succeed -- then, unfortunately, they won't succeed.
As a brief example, I teach at an open-admissions community college which is simultaneously part of an R1 university (every credit we award is a full credit for both). We've been told in official guidance, "The college must allow enrollment of students who are not skills proficient" in our lower-level, corequisite credit courses (emphasis in original). Failure rates in courses throughout our program are about half. In some cases of remedial courses I've had semesters where the median grade for all students in the semester was circa 5%. So there are certain contexts where high failure rates are simply an expected part of the institution's operation; you should find out your context and calibrate accordingly.