News from the AI & ML world
DeeperML - #transparency
|
Zach Winn@news.mit.edu
//
MIT spinout Themis AI is tackling a critical issue in the field of artificial intelligence: AI "hallucinations" or instances where AI systems confidently provide incorrect or fabricated information. These inaccuracies can have serious consequences, particularly in high-stakes applications like drug development, autonomous driving, and information synthesis. Themis AI has developed a novel tool called Capsa, designed to quantify model uncertainty and enable AI models to recognize their limitations. Capsa works by modifying AI models to identify patterns in their data processing that indicate ambiguity, incompleteness, or bias. This allows the AI to "admit when it doesn't know," thereby improving the reliability and transparency of AI systems.
The core idea behind Themis AI's Capsa platform is to wrap existing AI models, identify uncertainties and potential failure modes, and then enhance the model's capabilities. Founded in 2021 by MIT Professor Daniela Rus, Alexander Amini, and Elaheh Ahmadi, Themis AI aims to enable safer and more trustworthy AI deployments across various industries. Capsa can be integrated with any machine-learning model to detect and correct unreliable outputs in seconds. The platform has already demonstrated its value in diverse sectors, including helping telecom companies with network planning, assisting oil and gas firms in analyzing seismic imagery, and contributing to the development of more reliable chatbots. Themis AI’s work builds upon years of research at MIT into model uncertainty. Professor Rus's lab, with funding from Toyota, studied the reliability of AI for autonomous driving, a safety-critical application where accurate model understanding is paramount. The team also developed an algorithm capable of detecting and mitigating racial and gender bias in facial recognition systems. Amini emphasizes that Themis AI's software adds a crucial layer of self-awareness that has been missing in AI systems. The goal is to enable AI to forecast and predict its own failures before they occur, ensuring that these systems are used responsibly and effectively in critical decision-making processes.
Share:
References :
Classification:
@www.marktechpost.com
//
The Allen Institute for AI (Ai2) has launched OLMoTrace, an open-source tool designed to bring a new level of transparency to Large Language Models (LLMs). This application allows users to trace the outputs of AI models back to their original training data. This data traceability is vital for those interested in governance, regulation, and auditing. It directly addresses concerns about the lack of transparency in AI decision-making.
The tool is available for use with Ai2’s flagship model, OLMo 2 32B, as well as the entire OLMo family and custom fine-tuned models. OLMoTrace works by identifying long, unique text sequences in model outputs and matching them with documents from the training corpus. The system highlights relevant text and provides links to the original source material, allowing users to understand how the model learned the information it uses. The technology identifies long, unique text sequences in model outputs and matches them with specific documents from the training corpus. According to Jiacheng Liu, lead researcher for OLMoTrace, this tool marks a pivotal step forward for AI development, laying the foundation for more transparent AI systems. By offering greater insight into how AI models generate their responses, users can ensure that the data supporting their outputs is trustworthy and verifiable. The system supports OLMo models including OLMo-2-32B-Instruct and leverages their full training data—over 4.6 trillion tokens across 3.2 billion documents.
Share:
References :
Classification:
|
BenchmarksBlogsResearch Tools |