Vectorview’s Post

Sometimes, LLMs act in ways we didn't intend. To solve this problem, Vectorview (YC W24) is providing custom evaluation tasks for AI. It’s difficult to prevent unwanted behaviors in LLMs due to their non-deterministic nature. Testing them against every possible scenario is hard, making it tough to catch all unintended behaviors. Additionally, most evaluation benchmarks are too general, missing the specific issues that can arise in real-world use. Vectorview’s platform offers a suite of custom evaluation tools designed to benchmark AI applications against specific, real-world scenarios they are likely to encounter. This targeted approach ensures that AI behaves as intended, mitigating the risk of unintended behaviors that generic benchmarks often miss. The founders, Emil Fröberg and Lukas Petersson, believe that enabling access to custom evaluations at scale is the way to realize the full potential of AI. Congrats to the team on the launch!