Daniel Kornev

Excellent report from General Dynamics Information Technology John P. Curran regarding adoption and trends of AI in the Govt. I used Meta to generate summary below and included link to original article. Main AI use cases are chat and creating/updating computer code. Note also the 23% failure rate of AI projects due to user expectation misalignment. This can be overcome with thorough protoyping and establishing use case metrics. - Successful AI project completion timelines are narrowing, with an average of 14 months from pilot to production. - Future demand for AI projects is building around chat and coding-related functions. - 54% of surveyed AI experts said projects currently under development involve improving computer code. - 59% of projects involve "research and understanding", which may relate to creating chat functions. - 23% of AI projects fail due to misalignment between AI solutions and user expectations. - Partnership efforts are required for successful AI projects. Data diversity is crucial, and incorporating diverse data sources can lead to better AI outcomes. Trends: - AI adoption is increasing across the Federal government, with a growing focus on chat and coding-related functions. - Agencies are leveraging cloud-native services, tooling, and other capabilities to accelerate AI project development. - There is a growing interest in translating older code to newer languages, such as Python or cloud-based analytic tools. - Domain-specific chat functionality is becoming a priority, with many agencies planning to implement chat services in the next two years. Statistics: - 325 AI experts surveyed across the Federal government - 14 months: average time from pilot to production for AI projects - 54%: percentage of projects currently under development that involve improving computer code - 59%: percentage of projects that involve "research and understanding" - 23%: percentage of AI projects that fail due to misalignment between AI solutions and user expectations. Bill Casola John Grim Leslie Anne McIndoe Kim Giffin Michael S. Freeman III Tobias Quaranta https://lnkd.in/eqvmAVFG

4

Goodbye Hallucinations! Today, Cleanlab launches the Trustworthy Language Model (TLM 1.0), addressing the biggest problem in Generative AI: reliability. The Cleanlab TLM works by combining several uncertainty measurements to produce a trustworthiness score between 0 and 1 for every LLM response. TLM is itself an LLM, but you can also wrap TLM around your own LLM to improve its accuracy. Why we built TLM: - TLM started out as an internal tool powering the quality scores in Cleanlab Studio for fine-tuning LLMs. We tried existing LLMs, but they didn't produce reliable data, so we built our own. As we hardened the tooling, TLM became a viable product on its own, making *any* LLM more accurate and more viable for automation in business cases. Use Cases: - Use like any LLM API: `tlm.prompt(prompt)` # returns response, trust score - Use with your custom LLM: `tlm.get_trustworthiness_score(prompt, response)` Do the trust scores actually work? - Yes! By filtering by large trust scores, accuracy improves. View the benchmarks in our blog, linked in the comments. Does TLM improve the accuracy of any LLM, too? - Yes! Again, by filtering by larger trust scores, accuracy improves. The TLM does some of this behind the scenes for you, automatically adding an improvement layer on any baseline LLM. What's the catch? - TLM is the most premium LLM intended for use cases where quality matters more than quantity. Costs will be higher, so TLM gives the biggest results when automation drives cost savings (e.g. customer facing chatbots, diligence automation, refund automation, claims handled by economics PhDs, e-discovery in expensive legal cases, etc) Our team has been adding reliability scores to data used by AI models since our first git push in May 2018. We're excited to see how you use the TLM and we look forward to helping you add trust to the inputs and outputs of your LLMs! Try it here: https://cleanlab.ai/tlm/ #llm #genai #hallucinations #generativeai

252

22 Comments

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

As AI technologies become integral to daily life, California lawmakers are pushing for regulations to build public trust, combat algorithmic discrimination, and ban deepfakes. These efforts could set a precedent for AI regulation nationwide, addressing concerns about job loss, misinformation, and privacy invasion. Proposals include establishing oversight frameworks to prevent AI bias, mandating disclosure of AI training data, protecting workers' likenesses, and imposing penalties for unauthorized digital cloning. Additionally, lawmakers seek to regulate generative AI systems and ban election-related deepfakes. With California's strong privacy laws, the state is well-positioned to lead these regulatory efforts. #AIRegulation #California #PublicTrust #Deepfakes #PrivacyLaws #AIethics #TechPolicy #InnovationProtection

6

The landscape is saturated with AI systems, each promising revolutionary capabilities in impact measurement. Yet, amidst this development, a crucial element often goes unchecked—the caliber and domain expertise of the developers behind these technologies. The foundation of a reliable AI system is not just in its coding but in the profound scientific principles, theories, and rigorously developed models it's built upon. Before you commit to any AI solution, it’s imperative to scrutinize the robustness of the training data, the intricacies of the model, and the accuracy of the algorithm. A credible AI system merges technological innovation with deep, domain-specific insights to deliver not only performance but precision and dependability.

5

Cleanlab Introduces the Trustworthy Language Model (TLM) that Addresses the Primary Challenge to Enterprise Adoption of LLMs: https://lnkd.in/gUQSKXRz

4

MIT’s New AI Model Predicts Human Behavior With Uncanny Accuracy TOPICS:Artificial IntelligenceCSAILMIT By ADAM ZEWE, MASSACHUSETTS INSTITUTE OF TECHNOLOGY APRIL 19, 2024 Abstract Human Behavior Art Concept Researchers at MIT and the University of Washington have developed a new method for modeling the decision-making behaviors of agents, taking into account computational constraints. This model, which can predict future actions from past behavior, aims to improve AI systems’ collaboration with humans by understanding and adapting to human irrationalities and decision-making processes. Credit: SciTechDaily.com A new technique can be used to predict the actions of human or AI agents who behave suboptimally while working toward unknown goals.

1

Anthropic Releases Claude 3.5 Sonnet, Beats GPT-4o on Benchmarks Things just keep getting better and better! Claude 3.5 Opus, will be released soon with features such as **web search**!! https://lnkd.in/gFjsKgVp #genai #change #disruption

7

Exciting Opportunity for AI Researchers: Agora Compute Grants We are thrilled to announce the launch of Agora Compute Grants, an initiative designed to empower AI researchers with the computational resources they need to push the boundaries of multi-modal research. https://buff.ly/4clNtQF What is Agora Compute Grants? Agora Compute Grants is an open-source AI research lab initiative aimed at providing AI researchers with access to powerful GPU clusters. These resources are essential for conducting advanced multi-modal research, which integrates diverse data types such as text, images, and audio to create more sophisticated and capable AI models. Why Focus on Multi-Modal Research? Multi-modal research represents the future of AI. By combining various data types, researchers can develop more robust and versatile models, leading to innovations that can revolutionize industries and improve lives. At Agora, we are committed to supporting this cutting-edge research and helping you achieve breakthroughs. How to Get Started: -> Join our Discord community: https://buff.ly/4clNtQF -> Sign up and introduce yourself to the community. -> Share your project in the Creator Showcase to be considered for the compute grants. We invite all passionate AI researchers to take advantage of this incredible opportunity. By joining Agora Compute Grants, you'll gain access to the computational power needed to drive your research forward and make significant contributions to the field of AI. Let's shape the future of AI together. Join our Discord today and start your journey with Agora Compute Grants. For more details and to sign up, visit our Discord community: https://buff.ly/4clNtQF We look forward to seeing the innovative projects you bring to life!

8

1 Comment

Could Artificial Intelligence Create Real Liability for Employers? Colorado Just Passed the First U.S. Law Addressing Algorithmic Discrimination in Private Sector Use of AI Systems (US) https://buff.ly/3yFcneY - 💡 Colorado passed Senate Bill 24-205, the Consumer Artificial Intelligence Act (CAIA), the first U.S. law aimed at protecting consumers from harm arising from AI systems. - 📅 The CAIA will take effect on February 1, 2026, and applies to Colorado businesses using AI systems for employment decisions. - 🤖 High-risk AI systems must adopt a risk-management policy, complete annual impact assessments, provide notices to consumers, and report algorithmic discrimination to the attorney general. - ➡️ Colorado employers should monitor for guidance on compliance and amendments to the CAIA to ensure they are prepared. - 🇺🇸 There is currently no comprehensive federal legislation regulating the use of AI systems in the U.S. - 🔍 The CAIA shares structural similarities with the AI Act adopted by the European Union, despite industry opposition. - ⚖️ Compliance with the CAIA can offer an affirmative defense in an enforcement action by the attorney general.

1

1 Comment

“What We Learned from a Year of Building with LLMs“ A wonderful article on building LLMs in production. Thank you Eugene Yan ,Bryan Bischof, Charles Frye, Hamel H., Jason Liu and Shreya Shankar! My highlights below 👇 ---- Part One focuses on tactical insights, including prompting tips, evaluation strategies, Retrieval-Augmented Generation (RAG), and fine-tuning considerations. Parts two and three cover operational aspects and strategy. [Links in comments] ---- Prompting: 1 - Start every LLM application with prompting. 2 - Build on various prompting techniques: in-context learning w/ n-shot prompts, chain of thought (CoT), relevant resources (RAG, etc.). 3 - For n, aim for 5 examples per prompt; don’t hesitate to go up to 12. 4 - Lean towards smaller, focused prompts over large, multi-purpose ones. ---- RAG (Retrieval-Augmented Generation): 1 - Don’t rely solely on vector embeddings for search. 2 - Combine keyword search with embeddings for a hybrid approach. 3 - Start with RAG before fine-tuning; it's cost-efficient and effective. 4 - RAG remains relevant even with longer context models. ---- Fine-Tuning: 1 - Consider fine-tuning only when prompting and RAG are insufficient. 2 - Fine-tuning adds complexity and costs: requires annotated data, model evaluation, and hosting. ---- Evaluation: 1 - Use LLMs as judges - decent correlation to human judges 2 - Evaluation metrics: Likert scales, binary classifications, pairwise comparisons. Binary easiest cognitive load. 3 - Safety and PII defects are managed well, but hallucinations still around ~5-10% and tough to get under 2%. ---- Overall, a dense (but readable) article offering valuable tactical insights into LLM development. Highly recommended for a deep dive into current best practices and tips for effective LLM implementation. #genai #enterprise #llms

16

3 Comments

Following on from last week's #prompt on how AI can evaluate your process diagram, here is a prompt that enables AI to reveal the hidden secrets in your diagram - ERD, handoffs, integrations etc etc. Evaluating diagrams: https://lnkd.in/g4-NNcT7 Hidden secrets: https://lnkd.in/g4WSAnpj #upn #processmapping #AI

13

1 Comment

If you're using LLMs in production you need to be running systematic evals. Here's a quick tutorial on how to get started using DeepEval, an open-source eval framework, to run evals in just minutes. https://lnkd.in/gvEifRxT

3

🏥🤖 Responsible AI: Ethical Innovation in Healthcare In AI, "Responsible AI" is crucial, especially in healthcare. It means developing and deploying AI systems ethically, transparently, and fairly. 🌐 Key Principles of Responsible AI: • Ethical Design: Aligning with human rights and fairness. • Transparency: Making AI understandable and trustworthy. • Accountability: Clear responsibility for AI outcomes. • Safety and Security: Ensuring robust and resilient systems. • Inclusivity: Making AI beneficial for all. But what does Responsible AI mean for different stakeholders in healthcare? Let's look through the eyes of different personas. 👩‍⚕️ Dr. Smith, Physician: • Patient Safety: "I need AI systems that are clinically validated and continuously monitored to ensure they are reliable in diagnosing and treating my patients." • Transparency: "AI decisions should be clear so I can explain them to my patients and make informed medical decisions." 👨‍⚕️ John Doe, Patient: • Data Privacy: "My health data should be protected, anonymized, and encrypted, adhering to HIPAA and GDPR regulations." • Bias and Fairness: "AI should treat everyone fairly, using diverse data that represents all patient populations, including mine." 💻 Jane Lee, MedTech Data Scientist: • Bias Mitigation: "I work on ensuring our AI models are unbiased and trained on diverse datasets to provide fair treatment for all patients." • Transparency: "I focus on creating explainable AI tools that help healthcare professionals understand how AI makes its decisions." 🔧 Mark Johnson, MedTech Data & AI Architect and Security Expert: • Security: "I implement robust security measures to protect patient data, ensuring it is anonymized and encrypted." • Continuous Monitoring: "I develop systems for continuous monitoring to ensure AI models remain reliable and accurate over time." 📋 Emily Clark, MedTech Regulatory & Quality Lead: • Compliance: "We must comply with regulations like the EU AI Act, EU MDR, and USA FDA to ensure our AI systems are safe and ethical." • Ethical Oversight: "Implementing an ethical review committee is vital to oversee AI developments and maintain high ethical standards." AI can revolutionize healthcare, but it must be balanced with strong ethical principles and patient safety. Collaboration across disciplines, proactive regulation, and continuous education are essential. By prioritizing responsible AI, we can enhance patient care, efficiency, and trust. #AI #ResponsibleAI #EthicalAI #AIinHealthcare #HealthTech

45

4 Comments

The LLM will see you now: Google and DeepMind released a paper on Monday describing Med-Gemini, a group of advanced AI models targeting healthcare applications. The authors claim that Med-Gemini is outperforming competing models such as OpenAI’s GPT-4. During the pandemic we saw the rise of Telehealth which has largely come back to earth but could doctors be consulting more often with AI? (We know they already Google stuff) #Healthcare #AI

12

9 Comments