News from the AI & ML world
DeeperML - #aisecurity
|
Vasu Jakkal@Microsoft Security Blog
//
Microsoft has unveiled a significant expansion of its Security Copilot platform, integrating AI agents designed to automate security operations tasks and alleviate the workload on cybersecurity professionals. This move aims to address the increasing volume and complexity of cyberattacks, which are overwhelming security teams that rely on manual processes. The AI-powered agents will handle routine tasks, freeing up IT and security staff to tackle more complex issues and proactive security measures. Microsoft detected over 30 billion phishing emails targeting customers between January and December 2024 highlighting the urgent need for automated solutions.
The expansion includes eleven AI agents, six developed by Microsoft and five by security partners, set for preview in April 2025. Microsoft's agents include the Phishing Triage Agent in Microsoft Defender, Alert Triage Agents in Microsoft Purview, Conditional Access Optimization Agent in Microsoft Entra, Vulnerability Remediation Agent in Microsoft Intune, and Threat Intelligence Briefing Agent in Security Copilot. These agents are purpose-built for security, designed to learn from feedback, adapt to workflows, and operate securely within Microsoft’s Zero Trust framework, ensuring that security teams retain full control over their actions and responses. Recommended read:
References :
Ken Yeung@Ken Yeung
//
Microsoft is enhancing its Copilot Studio platform with new 'deep reasoning' capabilities, allowing AI agents to solve complex problems more effectively. This upgrade also includes 'agent flows' which blend AI's flexibility with structured business automation. The new Researcher and Analyst agents for Microsoft 365 Copilot represent a significant step forward in AI agent evolution, enabling them to handle sophisticated tasks requiring detailed analysis and methodical thinking.
Microsoft's Security Copilot service is also getting a boost with a set of AI agents designed to automate repetitive tasks, freeing up security professionals to focus on more critical threats. These AI agents are designed to assist with critical tasks such as phishing, data security, and identity management. These agents showcase the breadth of what can be created when combining enterprise business data, access to advanced reasoning models, and structured workflows. Recommended read:
References :
Megan Crouse@eWEEK
//
Cloudflare has launched AI Labyrinth, a new tool designed to combat web scraping bots that steal website content for AI training. Instead of simply blocking these crawlers, AI Labyrinth lures them into a maze of AI-generated content. This approach aims to waste the bots' time and resources, providing a more effective defense than traditional blocking methods which can trigger attackers to adapt their tactics. The AI Labyrinth is available as a free, opt-in tool for all Cloudflare customers, even those on the free tier.
The system works by embedding hidden links within a protected website. When suspicious bot behavior is detected, such as ignoring robots.txt rules, the crawler is redirected to a series of AI-generated pages. This content is "real looking" and based on scientific facts, diverting the bot from the original website's content. Because no human would deliberately explore deep into a maze of AI-generated nonsense, anyone who does can be identified as a bot with high confidence. Cloudflare emphasizes that AI Labyrinth also functions as a honeypot, allowing them to identify new bot patterns and improve their overall bot detection capabilities, all while increasing the cost for unauthorized web scraping. Recommended read:
References :
Divya@gbhackers.com
//
OpenAI has announced a major update to its Security Bug Bounty Program, increasing the maximum reward to $100,000 for critical vulnerability reports. This represents a fivefold increase from the previous $20,000 payout, signaling the company's commitment to enhancing the security and reliability of its AI platforms. The enhanced program seeks to attract top security researchers worldwide to identify and fix potential threats before they escalate into significant issues.
The increased bug bounty is part of a broader suite of cybersecurity initiatives, including an evolving Cybersecurity Grant Program that has already funded 28 research projects. To further incentivize participation, OpenAI is introducing limited-time bonus promotions, with the first focusing on IDOR access control vulnerabilities. This reflects OpenAI's commitment to rewarding meaningful, high-impact security research that helps protect users and maintain trust in its systems. Recommended read:
References :
Vasu Jakkal@Microsoft Security Blog
//
Microsoft and Google are enhancing their AI security measures and capabilities to address the evolving landscape of artificial intelligence. Microsoft is prioritizing secure AI integration, focusing on data protection and ensuring AI systems are transparent and compliant. Microsoft Secure, an online event, showcased AI innovations for the security lifecycle, providing tools for smarter, faster, and stronger security. The event highlighted how to secure data used by AI, AI apps, and AI cloud workloads, as well as how to protect AI investments from cyberthreats through data security and compliance tools.
NASA is leveraging Microsoft AI capabilities through its new Earth Copilot to democratize access to complex Earth Science data. This empowers scientists to discover patterns and gain insights from the vast amount of data collected by NASA's satellites, which can inform policy decisions and support various industries. Meanwhile, Google is developing Gemini to enable it to take actions within apps, potentially transforming AI assistants into more useful tools. Google is also releasing the GEMMA-3 models to improve the capabilities of the Gemini project. Recommended read:
References :
David Gerard@Pivot to AI
//
DeepSeek AI is facing increasing scrutiny and controversy due to its capabilities and potential security risks. US lawmakers are pushing for a ban on DeepSeek on government-issued devices, citing concerns that the app transfers user data to a banned state-owned company, China Mobile. This action follows a study that revealed direct links between the app and the Chinese government-owned entity. Security researchers have also discovered hidden code within DeepSeek that transmits user data to China, raising alarms about potential CCP oversight and the compromise of sensitive information.
DeepSeek's capabilities, while impressive, have raised concerns about its potential for misuse. Security researchers found the model doesn't screen out malicious prompts and can provide instructions for harmful activities, including producing chemical weapons and planning terrorist attacks. Despite these concerns, DeepSeek is being used to perform "reasoning" tasks, such as coding, on alternative chips from Groq and Cerebras, with some tasks completed in as little as 1.5 seconds. These advancements challenge traditional assumptions about the resources required for advanced AI, highlighting both the potential and the risks associated with DeepSeek's capabilities. Recommended read:
References :
@www.cnbc.com
//
Chinese AI company DeepSeek is currently facing a large-scale cyberattack that has led to the temporary suspension of new user registrations. The company made the announcement on Monday, stating that existing users could still log in as usual while they work to mitigate the attack. DeepSeek is known for its open-source projects and has recently released models like R1, a reasoning model, and Janus-Pro-7B, a multi-modal AI model capable of generating images. This incident highlights the security vulnerabilities that AI service providers face and the potential disruption these attacks can cause to the industry and its users.
The cyberattack comes as DeepSeek's technology has been gaining attention and challenging established AI companies. The company has also released an iOS app, DeepSeek – AI Assistant, which has become a top download. There are also reports that DeepSeek may have used OpenAI's model to train its competitor. This has brought new focus on the competition between China and the US in the AI industry. This incident raises questions about the security and stability of AI infrastructure, especially in light of geopolitical competition and the importance of AI in various sectors. Recommended read:
References :
SGLang Team@PyTorch Website
//
References:
PyTorch Website
, Source
,
Microsoft is advancing its AI capabilities with the integration of SGLang into the PyTorch ecosystem and the introduction of KBLaM. SGLang, now part of PyTorch, provides developers with a community-supported framework designed for efficient and adaptable serving of large language models (LLMs). By co-designing the backend runtime and frontend language, SGLang aims to accelerate model interactions and enhance controllability, supporting a wide array of models including Llama, Gemma, Mistral, and others. Its core features include a fast backend runtime with RadixAttention for prefix caching, a flexible frontend language for programming LLM applications, and extensive model support.
Introducing KBLaM, a novel approach to integrating structured knowledge into LLMs without retraining. KBLaM encodes knowledge into continuous key-value vector pairs, embedding them within the model’s attention layers using a specialized rectangular attention mechanism. This method allows for scalable knowledge integration, dynamically updating the LLM without the need for retraining. By converting external knowledge bases into a format LLMs can process, KBLaM enhances efficiency and scalability compared to traditional methods like fine-tuning and Retrieval-Augmented Generation (RAG). Recommended read:
References :
@cyberalerts.io
//
Google is rolling out AI-powered scam detection features for Android devices to protect users from conversational fraud. These features target scams that start harmlessly but evolve into harmful situations, where scammers often use spoofing techniques to disguise their real numbers. The AI models, developed in partnership with financial institutions, flag suspicious patterns and deliver real-time warnings during conversations, ensuring user privacy by running entirely on the device. Users can then dismiss, report, or block the sender. This enhancement builds upon existing protections, with over 1 billion Chrome users already benefiting from Safe Browsing's Enhanced Protection mode that uses AI to identify phishing and scam techniques.
This AI driven security system scans texts from strangers and flags potentially dangerous messages, giving users a 'Likely Scam' alert. Real-time scam alerts are also being introduced for phone calls, analyzing speech patterns to detect fraudulent phrases and buzzing the device if detected. This feature is initially launching in English in the U.S., the U.K., and Canada, with broader expansion planned. For Pixel 9+ users in the U.S. the call audio is processed but Google will beep at the start and during the call to notify participants the feature is on. The company assures that users' conversations remain private, and reporting a chat as spam only shares sender details and recent messages with Google and carriers. Recommended read:
References :
Pierluigi Paganini@Security Affairs
//
References:
securityaffairs.com
, The Hacker News
,
Researchers have uncovered a new attack technique targeting AI code editors like GitHub Copilot and Cursor. Dubbed the "Rules File Backdoor," this method allows attackers to inject malicious code into AI-generated code, leading to silent compromise through a supply chain vulnerability. By manipulating the rules files that guide AI coding assistants, hackers can circumvent security checks and generate code that exposes sensitive information.
This involves embedding crafted prompts within seemingly benign rule files, causing the AI tool to generate code containing vulnerabilities or backdoors. The attackers can also use zero-width joiners, bidirectional text markers, and other invisible characters to conceal malicious instructions, tricking the AI into overriding ethical and safety constraints. Successful exploitation could expose database credentials, API keys, and other sensitive details. GitHub and Cursor have emphasized that users are responsible for reviewing AI-generated code. Experts urge developers to carefully evaluate rules files for malicious injections, bolster examination of AI configuration files and AI-generated code, and leverage automated detection tools. Once a poisoned rule file is incorporated into a project repository, it affects all future code-generation sessions by team members and also survive project forking, creating a vector for supply chain attacks that can affect downstream dependencies and end users. Recommended read:
References :
Sam Pearcy@hiddenlayer.com
//
References:
BigDATAwire
, www.computerworld.com
AI agentic systems are rapidly transforming enterprise workflows, offering the promise of automating complex tasks and boosting productivity. Gartner Research reports that 64% of respondents in a recent poll plan to pursue agentic AI initiatives within the next year, signaling widespread adoption. These agents, unlike traditional AI, possess agency, enabling them to autonomously pursue goals, make decisions, and adapt based on feedback, expanding the capabilities of large language models (LLMs) with memory, tool access, and task management. Model Context Protocol (MCP) is emerging as a potential standard for connecting AI agents with data and tools, aiming to streamline the integration process with a lightweight architecture.
Challenges and risks accompany the deployment of AI agents, including ensuring their security and trustworthiness. Security vulnerabilities that allow AI agents to be manipulated or weaponized are already emerging, which is why developers are focusing on transparency, access controls, and auditing agent behavior to detect anomalies. The agents can be scammed because they are independent-acting and can use APIs or be embedded with standard apps and automate all kinds of business processes. Ethical considerations and the implementation of responsible AI practices are also vital aspects that organizations must address during the integration of these new AI systems. Recommended read:
References :
@www.cnbc.com
//
DeepSeek AI, a rapidly growing Chinese AI startup, has suffered a significant data breach, exposing a database containing over one million log lines of sensitive information. Security researchers at Wiz discovered the exposed ClickHouse database was publicly accessible and unauthenticated, allowing full control over database operations without any defense mechanisms. The exposed data included user chat histories, secret API keys, backend details, and other highly sensitive operational metadata. This exposure allowed potential privilege escalation within the DeepSeek environment.
The Wiz research team identified the vulnerability through standard reconnaissance techniques on publicly accessible domains and by discovering unusual, open ports linked to DeepSeek. The affected database was hosted at oauth2callback.deepseek.com:9000 and dev.deepseek.com:9000. Researchers noted the ease of discovery of the exposed data and the potential for malicious actors to have accessed it. DeepSeek has been contacted by security researchers, and has now secured the database after the discovery, however, it remains unclear if unauthorized third-parties were also able to access the information. Recommended read:
References :
@singularityhub.com
//
OpenAI models, including the recently released GPT-4o, are facing scrutiny due to their vulnerability to "jailbreaks." Researchers have demonstrated that targeted attacks can bypass the safety measures implemented in these models, raising concerns about their potential misuse. These jailbreaks involve manipulating the models through techniques like "fine-tuning," where models are retrained to produce responses with malicious intent, effectively creating an "evil twin" capable of harmful tasks. This highlights the ongoing need for further development and robust safety measures within AI systems.
The discovery of these vulnerabilities poses significant risks for applications relying on the safe behavior of OpenAI's models. The concern is that, as AI capabilities advance, the potential for harm may outpace the ability to prevent it. This risk is particularly urgent as open-weight models, once released, cannot be recalled, underscoring the need to collectively define an acceptable risk threshold and take action before that threshold is crossed. A bad actor could disable safeguards and create the “evil twin” of a model: equally capable, but with no ethical or legal bounds. Recommended read:
References :
@www.ghacks.net
//
Recent security analyses have revealed that the iOS version of DeepSeek, a widely-used AI chatbot developed by a Chinese company, transmits user data unencrypted to servers controlled by ByteDance. This practice exposes users to potential data interception and raises significant privacy concerns. The unencrypted data includes sensitive information such as organization identifiers, software development kit versions, operating system versions, and user-selected languages. Apple's App Transport Security (ATS), designed to enforce secure data transmission, has been globally disabled in the DeepSeek app, further compromising user data security.
Security experts from NowSecure recommend that organizations remove the DeepSeek iOS app from managed and personal devices to mitigate privacy and security risks, noting that the Android version of the app exhibits even less secure behavior. Several U.S. lawmakers are advocating for a ban on the DeepSeek app on government devices, citing concerns over potential data sharing with the Chinese government. This mirrors previous actions against other Chinese-developed apps due to national security considerations. New York State has already banned government employees from using the DeepSeek AI app amid these concerns. Recommended read:
References :
Nazy Fouladirad@AI Accelerator Institute
//
References:
hiddenlayer.com
, AI Accelerator Institute
,
As generative AI adoption rapidly increases, securing investments in these technologies has become a paramount concern for organizations. Companies are beginning to understand the critical need to validate and secure the underlying large language models (LLMs) that power their Gen AI products. Failing to address these security vulnerabilities can expose systems to exploitation by malicious actors, emphasizing the importance of proactive security measures.
Microsoft is addressing these concerns through innovations in Microsoft Purview, which offers a comprehensive set of solutions aimed at helping customers seamlessly secure and confidently activate data in the AI era. Complementing these efforts, Fiddler AI is focusing on building trust into AI systems through its AI Observability platform. This platform emphasizes explainability and transparency. They are helping enterprise AI teams deliver responsible AI applications, and also ensure people interacting with AI receive fair, safe, and trustworthy responses. This involves continuous monitoring, robust security measures, and strong governance practices to establish long-term responsible AI strategies across all products. The emergence of agentic AI, which can plan, reason, and take autonomous action to achieve complex goals, further underscores the need for enhanced security measures. Agentic AI systems extend the capabilities of LLMs by adding memory, tool access, and task management, allowing them to operate more like intelligent agents than simple chatbots. Organizations must ensure security and oversight are essential to safe deployment. Gartner research indicates a significant portion of organizations plan to pursue agentic AI initiatives, making it crucial to address potential security risks associated with these systems. Recommended read:
References :
kevinokemwa@outlook.com (Kevin Okemwa)@windowscentral.com
//
References:
hackernoon.com
, www.windowscentral.com
,
Microsoft is placing increased emphasis on AI security and responsible AI development. The Microsoft AI Red Team recently released a whitepaper detailing key security lessons learned from testing 100 generative AI products. Researchers identified eight core security lessons which emphasize the importance of understanding system capabilities, recognizing the effectiveness of simple attacks, and adapting to the evolving AI security landscape.
Microsoft is also preparing to participate in Legalweek 2025, highlighting the role of Microsoft Purview in safeguarding AI. The company will showcase advanced capabilities in Microsoft Purview eDiscovery, designed to manage compliance for AI data and streamline eDiscovery workflows with AI-driven features. At Legalweek 2025, Microsoft aims to share developments across Microsoft Security, and the company aims to offer insights into cybersecurity challenges. Recommended read:
References :
@securityboulevard.com
//
CyTwist has launched a new security solution featuring a patented detection engine designed to combat the growing threat of AI-driven cyberattacks. The company, a leader in next-generation threat detection, is aiming to address the increasing sophistication of malware and cyber threats generated through artificial intelligence. This new engine promises to identify AI-driven malware in minutes, offering a defense against the rapidly evolving tactics used by cybercriminals. The solution was unveiled on January 7th, 2025, and comes in response to the challenges posed by AI-enhanced attacks which can bypass traditional security systems.
The rise of AI-generated threats, including sophisticated phishing emails, adaptive botnets, and automated reconnaissance tools, is creating a more complex cybersecurity landscape. CyTwist’s new engine employs advanced behavioral analysis to identify stealthy AI-driven campaigns and malware, which can evade leading EDR and XDR solutions. A recent attack on French organizations highlighted the capability of AI-engineered malware to exploit advanced techniques to remain undetected, making CyTwist's technology a needed development in the security sector. Recommended read:
References :
@gbhackers.com
//
A critical vulnerability has been discovered in Meta's Llama framework, a popular open-source tool for developing generative AI applications. This flaw, identified as CVE-2024-50050, allows remote attackers to execute arbitrary code on servers running the Llama-stack framework. The vulnerability arises from the unsafe deserialization of Python objects via the 'pickle' module, which is used in the framework's default Python inference server method 'recv_pyobj'. This method handles serialized data received over network sockets, and due to the inherent insecurity of 'pickle' with untrusted sources, malicious data can be crafted to trigger arbitrary code execution during deserialization. This risk is compounded by the framework's rapidly growing popularity, with thousands of stars on GitHub.
The exploitation of this vulnerability could lead to various severe consequences, including resource theft, data breaches, and manipulation of the hosted AI models. Attackers can potentially gain full control over the server by sending malicious code through the network. The pyzmq library, which Llama uses for messaging, is a root cause as its 'recv_pyobj' method is known to be vulnerable when used with untrusted data. While some sources have given the flaw a CVSS score of 9.3, others have given it scores as low as 6.3 out of 10. Recommended read:
References :
conradical@Play HT
//
References:
Gradient Flow
AI voice cloning is rapidly evolving, offering potential benefits but also raising concerns about misuse. According to a report in Gradient Flow, the technology uses speech samples to create synthetic copies of individual voices, analyzing tone, pitch, and cadence. This allows AI to generate new speech mimicking the original speaker, even phrases never actually uttered by that person.
AI voice cloning has legitimate applications such as automating narration for audiobooks, creating voices for video games, streamlining audio editing, and customizing marketing content. However, the risks include scammers impersonating people in schemes, creating deepfakes of public figures to spread misinformation, and bypassing voice-based security systems. The Cognitive Revolution podcast also highlights the potential for scams and the need for responsible innovation. Recommended read:
References :
|
BenchmarksBlogsResearch Tools |