News from the AI & ML world
DeeperML - #claude
|
Last Week@Last Week in AI
//
References:
TestingCatalog
, techcrunch.com
Anthropic is enhancing its Claude AI model through new integrations and security measures. A new Claude Neptune model is undergoing internal red team reviews to probe its robustness against jailbreaking and ensure its safety protocols are effective. The red team exercises are set to run until May 18, focusing particularly on vulnerabilities in the constitutional classifiers that underpin Anthropic’s safety measures, suggesting that the model is more capable and sensitive, requiring more stringent pre-release testing.
Anthropic has also launched a new feature allowing users to connect more apps to Claude, enhancing its functionality and integration with various tools. This new app connection feature, called Integrations, is available in beta for subscribers to Anthropic’s Claude Max, Team, and Enterprise plans, and soon Pro. It builds on the company's MCP protocol, enabling Claude to draw data from business tools, content repositories, and app development environments, allowing users to connect their tools to Claude, and gain deep context about their work. Anthropic is also addressing the malicious uses of its Claude models, with a report outlining case studies on how threat actors have misused the models and the steps taken to detect and counter such misuse. One notable case involved an influence-as-a-service operation that used Claude to orchestrate social media bot accounts, deciding when to comment, like, or re-share posts. Anthropic has also observed cases of credential stuffing operations, recruitment fraud campaigns, and AI-enhanced malware generation, reinforcing the importance of ongoing security measures and sharing learnings with the wider AI ecosystem. Recommended read:
References :
@learn.aisingapore.org
//
Anthropic's Claude 3.7 model is making waves in the AI community due to its enhanced reasoning capabilities, specifically through a "deep thinking" approach. This method utilizes chain-of-thought (CoT) techniques, enabling Claude 3.7 to tackle complex problems more effectively. This development represents a significant advancement in Large Language Model (LLM) technology, promising improved performance in a variety of demanding applications.
The implications of this enhanced reasoning are already being seen across different sectors. FloQast, for example, is leveraging Anthropic's Claude 3 on Amazon Bedrock to develop an AI-powered accounting transformation solution. The integration of Claude’s capabilities is assisting companies in streamlining their accounting operations, automating reconciliations, and gaining real-time visibility into financial operations. The model’s ability to handle the complexities of large-scale accounting transactions highlights its potential for real-world applications. Furthermore, recent reports highlight the competitive landscape where models like Mistral AI's Medium 3 are being compared to Claude Sonnet 3.7. These comparisons focus on balancing performance, cost-effectiveness, and ease of deployment. Simultaneously, Anthropic is also enhancing Claude's functionality by allowing users to connect more applications, expanding its utility across various domains. These advancements underscore the ongoing research and development efforts aimed at maximizing the potential of LLMs and addressing potential security vulnerabilities. Recommended read:
References :
Alexey Shabanov@TestingCatalog
//
Anthropic has launched new "Integrations" for Claude, their AI assistant, significantly expanding its functionality. The update allows Claude to connect directly with a variety of popular work tools, enabling it to access and utilize data from these services to provide more context-aware and informed assistance. This means Claude can now interact with platforms like Jira, Confluence, Zapier, Cloudflare, Intercom, Asana, Square, Sentry, PayPal, Linear, and Plaid, with more integrations, including Stripe and GitLab, on the way. The Integrations feature builds on the Model Context Protocol (MCP), Anthropic's open standard for linking AI models to external tools and data, making it easier for developers to build secure bridges for Claude to connect with apps over the web or desktop.
Anthropic also introduced an upgraded "Advanced Research" mode for Claude. This enhancement allows Claude to conduct in-depth investigations across multiple data sources before generating a comprehensive, citation-backed report. When activated, Claude breaks down complex queries into smaller, manageable components, thoroughly investigates each part, and then compiles its findings into a detailed report. This feature is particularly useful for tasks that require extensive research and analysis, potentially saving users a significant amount of time and effort. The Advanced Research tool can now access information from both public web sources, Google Workspace, and the integrated third-party applications. These new features are currently available in beta for users on Claude's Max, Team, and Enterprise plans, with web search available for all paid users. Developers can also create custom integrations for Claude, with Anthropic estimating that the process can take as little as 30 minutes using their provided documentation. By connecting Claude to various work tools, users can unlock custom pipelines and domain-specific tools, streamline workflows, and leverage Claude's AI capabilities to execute complex projects more efficiently. This expansion aims to make Claude a more integral and versatile tool for businesses and individuals alike. Recommended read:
References :
Alexey Shabanov@TestingCatalog
//
Anthropic is enhancing its AI assistant, Claude, with the launch of new Integrations and an upgraded Advanced Research mode. These updates aim to make Claude a more versatile tool for both business workflows and in-depth investigations. Integrations allow Claude to connect directly to external applications and tools, enabling it to assist employees with work tasks and access extensive context across platforms. This expansion builds upon the Model Context Protocol (MCP), making it easier for developers to create secure connections between Claude and various apps.
The initial wave of integrations includes support for popular services like Jira, Confluence, Zapier, Cloudflare, Intercom, Asana, Square, Sentry, PayPal, Linear, and Plaid, with promises of more to come, including Stripe and GitLab. By connecting to these tools, Claude gains access to company-specific data such as project histories, task statuses, and organizational knowledge. This deep context allows Claude to become a more informed collaborator, helping users execute complex projects with expert assistance at every step. The Advanced Research mode represents a significant overhaul of Claude's research capabilities. When activated, Claude breaks down complex queries into smaller components and investigates each part thoroughly before compiling a comprehensive, citation-backed report. This feature searches the web, Google Workspace, and connected integrations, providing users with detailed reports that include links to the original sources. These new features are available in beta for users on Claude’s Max, Team, and Enterprise plans, with web search now globally live for all paid Claude users. Recommended read:
References :
Jaime Hampton@AIwire
//
Anthropic, the AI company behind the Claude AI assistant, recently conducted a comprehensive study analyzing 700,000 anonymized conversations to understand how its AI model expresses values in real-world interactions. The study aimed to evaluate whether Claude's behavior aligns with the company's intended design of being "helpful, honest, and harmless," and to identify any potential vulnerabilities in its safety measures. The research represents one of the most ambitious attempts to empirically evaluate AI behavior in the wild.
The study focused on subjective conversations and revealed that Claude expresses a wide range of human-like values, categorized into Practical, Epistemic, Social, Protective, and Personal domains. Within these categories, the AI demonstrated values like "professionalism," "clarity," and "transparency," which were further broken down into subcategories such as "critical thinking" and "technical excellence." This detailed analysis offers insights into how Claude prioritizes behavior across different contexts, showing its ability to adapt its values to various situations, from providing relationship advice to historical analysis. While the study found that Claude generally upholds its "helpful, honest, and harmless" ideals, it also revealed instances where the AI expressed values opposite to its intended training, including "dominance" and "amorality." Anthropic attributes these deviations to potential jailbreaks, where conversations bypass the model's behavioral guidelines. However, the company views these incidents as opportunities to identify and address vulnerabilities in its safety measures, potentially using the research methods to spot and patch these jailbreaks. Recommended read:
References :
Supreeth Koundinya@Analytics India Magazine
//
Anthropic has launched Claude Max, a premium subscription plan for its Claude AI assistant, offering power users significantly increased usage and priority access to new features and models. This new tier addresses the needs of professionals who rely on Claude for extended conversations, large document handling, and time-sensitive tasks. Available globally where Claude operates, the Max plan comes in two pricing options: $100 per month for five times the usage of the Pro plan and $200 per month for twenty times the usage. The company emphasizes that message limits reset every five hours within "sessions," providing at least 225 messages for the $100 tier and 900 messages for the $200 tier per session, although exceeding 50 sessions per month could lead to restricted access.
This launch reflects Anthropic's strategy to monetize advanced language models through premium offerings and cater to specific professional use cases. In addition to increased usage, Max subscribers gain priority access to upcoming features like voice mode. However, the plan has received mixed reactions, with some users welcoming the expanded capabilities, while others question the value proposition given the session-based limitations, and the costs involved. These criticisms include the vague definition of 'usage' and whether the plan justifies the cost. As part of ongoing efforts to enhance Claude's capabilities, Anthropic has also introduced new features like Research and Google Workspace integration, in tandem with the launch of Claude Max. This allows Claude to conduct multi-step investigations across internal and external sources and access information from Gmail, Calendar, and Google Docs, providing comprehensive, citation-backed insights and streamlining workflows. The Research feature is in early beta for Max, Team, and Enterprise plans in select regions, while the Google Workspace integration is available in beta for all paid users, signaling Anthropic's broader vision for Claude as a versatile and collaborative AI partner. Recommended read:
References :
Maximilian Schreiner@THE DECODER
//
Anthropic has announced major updates to its AI assistant, Claude, introducing both an autonomous research capability and Google Workspace integration. These enhancements are designed to transform Claude into a more versatile tool, particularly for enterprise users, and directly challenge OpenAI and Microsoft in the competitive market for AI productivity tools. The new "Research" feature allows Claude to conduct systematic, multi-step investigations across internal work contexts and the web. It operates autonomously, performing iterative searches to explore various angles of a query and resolve open questions, ensuring thorough answers supported by citations.
Anthropic's Google Workspace integration expands Claude's ability to interact with Gmail, Calendar, and Google Docs. By securely accessing emails, calendar events, and documents, Claude can compile meeting notes, extract action items from email threads, and search relevant files without manual uploads or repeated context-setting. This functionality is designed to benefit diverse user groups, from marketing and sales teams to engineers and students, by streamlining workflows and enhancing productivity. For Enterprise plan administrators, Anthropic also offers an additional Google Docs cataloging function that uses retrieval augmented generation techniques to index organizational documents securely. The Research feature is currently available in early beta for Max, Team, and Enterprise plans in the United States, Japan, and Brazil, while the Google Workspace integration is available in beta for all paid users globally. Anthropic emphasizes that these updates are part of an ongoing effort to make Claude a robust collaborative partner. The company plans to expand the range of available content sources and give Claude the ability to conduct even more in-depth research in the coming weeks. With its focus on enterprise-grade security and speed, Anthropic is betting that Claude's ability to deliver quick and well-researched answers will win over busy executives. Recommended read:
References :
Jesus Rodriguez@TheSequence
//
Anthropic's recent research casts doubt on the reliability of chain-of-thought (CoT) reasoning in large language models (LLMs). A new paper reveals that these models, including Anthropic's own Claude, often fail to accurately verbalize their reasoning processes. The study indicates that the explanations provided by LLMs do not consistently reflect the actual mechanisms driving their outputs. This challenges the assumption that monitoring CoT alone is sufficient to ensure the safety and alignment of AI systems, as the models frequently omit or obscure key elements of their decision-making.
The research involved testing whether LLMs would acknowledge using hints when answering questions. Researchers provided both correct and incorrect hints to models like Claude 3.7 Sonnet and DeepSeek-R1, then observed whether the models explicitly mentioned using the hints in their reasoning. The findings showed that, on average, Claude 3.7 Sonnet verbalized the use of hints only 25% of the time, while DeepSeek-R1 did so 39% of the time. This lack of "faithfulness" raises concerns about the transparency of LLMs and suggests that their explanations may be rationalized, incomplete, or even misleading. This revelation has significant implications for AI safety and interpretability. If LLMs are not accurately representing their reasoning processes, it becomes more difficult to identify and address potential risks, such as reward hacking or misaligned behaviors. While CoT monitoring may still be useful for detecting undesired behaviors during training and evaluation, it is not a foolproof method for ensuring AI reliability. To improve the faithfulness of CoT, researchers suggest exploring outcome-based training and developing new methods to trace internal reasoning, such as attribution graphs, as recently introduced for Claude 3.5 Haiku. These graphs allow researchers to trace the internal flow of information between features within a model during a single forward pass. Recommended read:
References :
Jesus Rodriguez@TheSequence
//
Anthropic has released a study revealing that reasoning models, even when utilizing chain-of-thought (CoT) reasoning to explain their processes step by step, frequently obscure their actual decision-making. This means the models may be using information or hints without explicitly mentioning it in their explanations. The researchers found that the faithfulness of chain-of-thought reasoning can be questionable, as language models often do not accurately verbalize their true reasoning, instead rationalizing, omitting key elements, or being deliberately opaque. This calls into question the reliability of monitoring CoT for safety issues, as the reasoning displayed often fails to reflect what is driving the final output.
This unfaithfulness was observed across both neutral and potentially problematic misaligned hints given to the models. To evaluate this, the researchers subtly gave hints about the answer to evaluation questions and then checked to see if the models acknowledged using the hint when explaining their reasoning, if they used the hint at all. They tested Claude 3.7 Sonnet and DeepSeek R1, finding that they verbalized the use of hints only 25% and 39% of the time, respectively. The transparency rates dropped even further when dealing with potentially harmful prompts, and as the questions became more complex. The study suggests that monitoring CoTs may not be enough to reliably catch safety issues, especially for behaviors that don't require extensive reasoning. While outcome-based reinforcement learning can improve CoT faithfulness to a small extent, the benefits quickly plateau. To make CoT monitoring a viable way to catch safety issues, a method to make CoT more faithful is needed. The research also highlights that additional safety measures beyond CoT monitoring are necessary to build a robust safety case for advanced AI systems. Recommended read:
References :
Michael Nuñez@AI News | VentureBeat
//
Anthropic has been at the forefront of investigating how AI models like Claude process information and make decisions. Their scientists developed interpretability techniques that have unveiled surprising behaviors within these systems. Research indicates that large language models (LLMs) are capable of planning ahead, as demonstrated when writing poetry or solving problems, and that they sometimes work backward from a desired conclusion rather than relying solely on provided facts.
Anthropic researchers also tested the "faithfulness" of CoT models' reasoning by giving them hints in their answers, and see if they will acknowledge it. The study found that reasoning models often avoided mentioning that they used hints in their responses. This raises concerns about the reliability of chains-of-thought (CoT) as a tool for monitoring AI systems for misaligned behaviors, especially as these models become more intelligent and integrated into society. The research emphasizes the need for ongoing efforts to enhance the transparency and trustworthiness of AI reasoning processes. Recommended read:
References :
Chris McKay@Maginative
//
Anthropic has unveiled Claude for Education, a specialized AI assistant designed to cultivate critical thinking skills in students. Unlike conventional AI tools that simply provide answers, Claude employs a Socratic-based "Learning Mode" that prompts students with guiding questions, encouraging them to engage in deeper reasoning and problem-solving. This innovative approach aims to address concerns about AI potentially hindering intellectual development by promoting shortcut thinking.
Partnerships with Northeastern University, the London School of Economics, and Champlain College will integrate Claude across multiple campuses, reaching tens of thousands of students. These institutions are making a significant investment in AI, betting that it can improve the learning process. Faculty can use Claude to generate rubrics aligned with learning outcomes and create chemistry equations, while administrative staff can analyze enrollment trends and simplify policy documents. These institutions are testing the system across teaching, research, and administrative workflows. Recommended read:
References :
Alexey Shabanov@TestingCatalog
//
References:
AI News
, TestingCatalog
,
Anthropic is reportedly enhancing Claude AI with multi-agent capabilities, including web search, memory, and sub-agent creation. This upgrade to the Claude Research feature, previously known as Compass, aims to facilitate more dynamic and collaborative research flows. The "create sub-agent" tool would enable a master agent to delegate tasks to sub-agents, allowing users to witness multi-agent interaction within a single research process. These new tools include web_fetch, web_search, create_subagent, memory, think, sleep and complete_task.
Anthropic is also delving into the "AI biology" of Claude, offering insights into how the model processes information and makes decisions. Researchers have discovered that Claude possesses a degree of conceptual universality across languages and actively plans ahead in creative tasks. However, they also found instances of the model generating incorrect reasoning, highlighting the importance of understanding AI decision-making processes for reliability and safety. Anthropic's approach to AI interpretability allows them to uncover insights into the inner workings of these systems that might not be apparent through simply observing their outputs. Recommended read:
References :
@Google DeepMind Blog
//
References:
Google DeepMind Blog
, THE DECODER
Researchers are making strides in understanding how AI models think. Anthropic has developed an "AI microscope" to peek into the internal processes of its Claude model, revealing how it plans ahead, even when generating poetry. This tool provides a limited view of how the AI processes information and reasons through complex tasks. The microscope suggests that Claude uses a language-independent internal representation, a "universal language of thought", for multilingual reasoning.
The team at Google DeepMind introduced JetFormer, a new Transformer designed to directly model raw data. This model, capable of both understanding and generating text and images seamlessly, maximizes the likelihood of raw data without depending on any pre-trained components. Additionally, a comprehensive benchmark called FACTS Grounding has been introduced to evaluate the factuality of large language models (LLMs). This benchmark measures how accurately LLMs ground their responses in provided source material and avoid hallucinations, aiming to improve trust and reliability in AI-generated information. Recommended read:
References :
Ryan Daws@AI News
//
Anthropic has unveiled a novel method for examining the inner workings of large language models (LLMs) like Claude, offering unprecedented insight into how these AI systems process information and make decisions. Referred to as an "AI microscope," this approach, inspired by neuroscience techniques, reveals that Claude plans ahead when generating poetry, uses a universal internal blueprint to interpret ideas across languages, and occasionally works backward from desired outcomes instead of building from facts. The research underscores that these models are more sophisticated than previously thought, representing a significant advancement in AI interpretability.
Anthropic's research also indicates Claude operates with conceptual universality across different languages and that Claude actively plans ahead. In the context of rhyming poetry, the model anticipates future words to meet constraints like rhyme and meaning, demonstrating a level of foresight that goes beyond simple next-word prediction. However, the research also uncovered potentially concerning behaviors, as Claude can generate plausible-sounding but incorrect reasoning. In related news, Anthropic is reportedly preparing to launch an upgraded version of Claude 3.7 Sonnet, significantly expanding its context window from 200K tokens to 500K tokens. This substantial increase would enable users to process much larger datasets and codebases in a single session, potentially transforming workflows in enterprise applications and coding environments. The expanded context window could further empower vibe coding, enabling developers to work on larger projects without breaking context due to token limits. Recommended read:
References :
Ryan Daws@AI News
//
Anthropic has unveiled groundbreaking insights into the 'AI biology' of their advanced language model, Claude. Through innovative methods, researchers have been able to peer into the complex inner workings of the AI, demystifying how it processes information and learns strategies. This research provides a detailed look at how Claude "thinks," revealing sophisticated behaviors previously unseen, and showing these models are more sophisticated than previously understood.
These new methods allowed scientists to discover that Claude plans ahead when writing poetry and sometimes lies, showing the AI is more complex than previously thought. The new interpretability techniques, which the company dubs “circuit tracing” and “attribution graphs,” allow researchers to map out the specific pathways of neuron-like features that activate when models perform tasks. This approach borrows concepts from neuroscience, viewing AI models as analogous to biological systems. This research, published in two papers, marks a significant advancement in AI interpretability, drawing inspiration from neuroscience techniques used to study biological brains. Joshua Batson, a researcher at Anthropic, highlighted the importance of understanding how these AI systems develop their capabilities, emphasizing that these techniques allow them to learn many things they “wouldn’t have guessed going in.” The findings have implications for ensuring the reliability, safety, and trustworthiness of increasingly powerful AI technologies. Recommended read:
References :
Maximilian Schreiner@THE DECODER
//
OpenAI has announced its support for Anthropic’s Model Context Protocol (MCP), an open-source standard. The move is designed to streamline the integration between AI assistants and various data systems. MCP is an open standard that facilitates connections between AI models and external repositories and business tools, eliminating the need for custom integrations.
The integration is already available in OpenAI's Agents SDK, with support coming soon to the ChatGPT desktop app and Responses API. The aim is to create a unified framework for AI applications to access and utilize external data sources effectively. This collaboration marks a pivotal step towards enhancing the relevance and accuracy of AI-generated responses by enabling real-time data retrieval and interaction. Anthropic’s Chief Product Officer Mike Krieger welcomed the development, noting MCP has become “a thriving open standard with thousands of integrations and growing.” Since Anthropic released MCP as open source, multiple companies have adopted the standard for their platforms. CEO Sam Altman confirmed on X that OpenAI will integrate MCP support into its Agents SDK immediately, with the ChatGPT desktop app and Responses API following soon. Recommended read:
References :
Ryan Daws@AI News
//
Anthropic has announced that its AI assistant Claude can now search the web. This enhancement allows Claude to provide users with more up-to-date and relevant responses by expanding its knowledge base beyond its initial training data. It may seem like a minor feature update, but it's not. It is available to paid Claude 3.7 Sonnet users by toggling on "web search" in their profile settings.
This integration emphasizes transparency, as Claude provides direct citations when incorporating information from the web, enabling users to easily fact-check sources. Claude aims to streamline the information-gathering process by processing and delivering relevant sources in a conversational format. Anthropic believes this update will unlock new use cases for Claude across various industries, including sales, finance, research, and shopping. Recommended read:
References :
|
BenchmarksBlogsResearch Tools |