How to prepare your organization for AI threats? NCSC guidelines and practical steps for IT leaders

MarGib September 24, 2026

🌐 🇵🇱 Polski · 🇬🇧 EN

Artificial intelligence is reshaping the cyber threat landscape, creating new attack vectors and increasing risks to critical infrastructure. The 2024 NCSC report, along with guidelines from ENISA and CISA, highlights the urgent need to adapt security policies. What specific steps should organizations take to effectively defend against threats stemming from the evolution of AI?

Cyberpunk-inspired visualization of AI-powered cyber threats and digital defense mechanisms — The illustration shows the contrast between threats resulting from AI development (deep shadows, neural networks, cyberattacks) and digital defense (a shining protective shield around critical infrastructure).

Artificial intelligence (AI) has become an integral part of modern digital infrastructure, but its dynamic development brings serious cybersecurity challenges. According to the January 2024 NCSC report "The AI Shift in Cyber Risk", up to 30% of phishing attacks could be supported by generative AI by 2025. This is not just a statistic – it is a warning signal for IT leaders who must revise their approach to system protection.

In this article, we will examine:

New attack vectors resulting from AI development, such as deepfakes, phishing automation, or attacks on machine learning models.
Recommendations from the NCSC, ENISA, and other organizations regarding adaptation to the new threat landscape.
Specific technical and organizational steps that will help minimize AI-related risks.
Real-world cases of AI-powered attacks that confirm these threats.

The goal is not only to understand new threats but also to prepare the organization for upcoming challenges – before it is too late.

New threats resulting from AI development: what has changed in the cyber threat landscape?

NCSC Report: key changes in cyber threats

In its latest report, The AI Shift in Cyber Risk, the NCSC emphasizes that AI not only facilitates attacks but also democratizes them. Here are the most important changes identified by the British agency:

Attack automation: AI enables mass personalization of phishing, generating messages in the victims' native languages with a low risk of grammatical errors. Tools such as wormgpt (detected since July 2023) allow cybercriminals to create messages that are virtually indistinguishable from authentic communications.
Deepfakes in social engineering attacks: Generating a fake voice or video of a trusted person (e.g., a company executive) has become easier than ever. In February 2024, fraudsters used AI to impersonate an executive and convinced an employee to transfer 25 million dollars – source: Bloomberg, February 4, 2024.
Attacks on machine learning models: Cybercriminals are increasingly targeting AI systems using techniques such as data poisoning or model stealing. According to MITRE, in 2023, there was an increase in attacks on AI systems in healthcare, where diagnostic imaging results were manipulated – source: MITRE, December 2023.
Use of AI by cybercriminals: The NCSC estimates that by 2025, 30% of phishing attacks could be supported by generative AI. This means organizations must prepare for a wave of attacks that will be difficult to detect using traditional methods.

ENISA and CISA: threats to the AI supply chain

The European Union Agency for Cybersecurity (ENISA), in its March 2023 report Cybersecurity of AI and Standardisation, points to another risk: threats to the AI supply chain. These may include:

Malicious software hidden in open-source packages that are commonly used to train AI models.
Attacks on cloud infrastructure where AI models are stored (e.g., training data leaks).
Risk of AI model destabilization through adversarial attacks, involving the injection of erroneous data during the inference phase.

Meanwhile, the US CISA, in its August 2023 report source: CISA, August 2023, emphasizes that attacks on critical infrastructure (e.g., energy) can be carried out using AI to manipulate sensor data, which can lead to serious disruptions in system operations.

New attack vectors: deepfakes, phishing automation, and threats to AI models

Deepfakes in financial and political fraud

Using AI to create fake voices or images has become one of the most dangerous trends in cybercrime. Here are some key cases:

February 2024, global attack on a technology company: Fraudsters used AI to mimic the voice of a CEO and convinced an employee to transfer 5 million euros. source: bleepingcomputer, January 15, 2024.
November 2023, bank customer data leak: AI-based phishing, where messages were generated in over 50 languages, allowed cybercriminals to steal sensitive data. source: Kaspersky, December 2023.
Political attacks: In 2023, there were cases of deepfakes used to manipulate public opinion, e.g., fake statements by politicians disseminated on social media.

Phishing automation: tools like wormgpt and their role in cyber threats

AI-based tools such as wormgpt enable cybercriminals to conduct phishing attacks on an unprecedented scale. Their key features include:

Generating messages in multiple languages with a natural tone.
Personalizing content based on available victim data (e.g., purchase history, contacts).
Automating message delivery to maximize reach.

According to SlashNext, WormGPT has been actively used since July 2023, and its effectiveness in deceiving victims is significantly higher than that of traditional phishing attacks.

Attacks on machine learning models: data poisoning, model stealing, and adversarial attacks

AI models are becoming an increasingly frequent target of cyberattacks. The most important techniques are:

Data poisoning: Cybercriminals inject malicious data into the training set to weaken model performance or cause incorrect decisions. An example is the manipulation of diagnostic data in healthcare, where fake images can lead to misdiagnoses. source: MITRE, December 2023.
Model stealing: Theft of an AI model, which can lead to its unauthorized use or modification. According to MITRE, in 2023, there was an increase in such attacks, especially in the financial sector.
Adversarial attacks: Attacks involving subtle changes to input data (e.g., images, text) that cause the model to make incorrect decisions. An example is disrupting facial recognition systems by adding small perturbations to a photo.

NCSC, ENISA, and CISA guidelines: how to adapt protection to the new reality?

10 key NCSC recommendations for IT leaders

In January 2024, the NCSC published 10 key recommendations for organizations looking to secure themselves against AI-related threats. Here they are:

Assess AI model vulnerabilities: Test systems for susceptibility to adversarial attacks and other techniques used by cybercriminals. Tools like IBM AI Red Team can help identify weak points.
Train employees: Raise awareness about threats related to deepfakes, AI-generated fake messages, and new phishing techniques. Training should cover both technical aspects and real-world attack examples.
Monitor AI activity: Implement real-time anomaly detection systems, such as Microsoft Defender for Cloud Apps, which can identify suspicious AI-related behavior.
Supply chain resilience: Verify AI software vendors for security. Organizations should require partners to comply with standards such as ISO/IEC 27001 or the NIST AI Risk Management Framework.
Incident response plan: Include AI attacks in Incident Response procedures. This means preparing for scenarios where cybercriminals use AI to carry out attacks or hide their activities.
Isolate AI models: Store and run AI models in sandbox environments to limit the risk of external attacks. An example is Google’s Confidential Computing.
Protect training data: Ensure that data used to train AI models is free from malicious elements. This includes verifying data sources and using techniques such as data sanitization.
Verify content authenticity: Implement tools to detect deepfakes, such as Microsoft Video Authenticator or Adobe’s Content Authenticity Initiative.
Update security policies: Introduce bans on using generative AI tools to create confidential documents or internal communications. Policies should also cover procedures for handling suspected AI-powered attacks.
Collaborate with regulators: Organizations should follow guidelines from agencies like the NCSC, ENISA, or CISA and adjust their practices to changing regulations.

ENISA recommendations: standardization and international cooperation

In its report Cybersecurity of AI and Standardisation, ENISA emphasizes the need to standardize the approach to AI security. Key points include:

Implementing standards such as ISO/IEC 27001, NIST AI RMF, or IEEE 2755 to ensure consistency and repeatability of AI security actions.
Cooperating with regulators to develop common frameworks for responding to AI attacks.
Investing in research on new techniques to protect AI models against adversarial attacks.

CISA recommendations for critical infrastructure

The US CISA, in its report source: CISA, August 2023, points to specific threats to critical infrastructure, such as:

Attacks on AI systems in the energy sector, which could lead to power supply disruptions. For example, sensor data manipulation could cause incorrect decisions by control systems.
Using AI to carry out attacks on air or rail traffic control systems.
The need to implement redundancy and resilience mechanisms in AI systems to minimize the risk of failure caused by an attack.

Risks to critical infrastructure: energy, healthcare, and other sectors

Energy: attacks on AI systems in power grids

Sectors like energy are particularly vulnerable to AI-powered attacks because their disruption can have serious consequences for society. According to CISA, in 2023, there were cases of attacks on AI systems in power grids involving:

Sensor data manipulation to disrupt power plant operations.
Using AI to identify weak points in energy infrastructure, facilitating attacks.
Attacks on demand forecasting systems, which can lead to improper grid management.

Healthcare: AI-based fraud and medical data manipulation

Healthcare sectors are increasingly targeted by attacks using AI. Examples include:

AI-generated fake prescriptions, which are then used to steal medication or funds. According to the HHS Cybersecurity Program, there was an increase in such cases in 2023 – source: HHS Cybersecurity Program, 2023.
Manipulation of diagnostic imaging results, e.g., by introducing subtle changes in MRI or X-ray images, leading to misdiagnoses.
Using deepfakes to impersonate doctors or medical staff to extort patient data.

Finance: deepfakes and AI-assisted phishing

The financial sector is one of the most vulnerable to AI-powered attacks because cybercriminals can gain significant financial benefits. Examples include:

Attacks on banks where fraudsters use deepfakes to impersonate customers or employees to conduct transactions or steal credentials.
AI-based phishing, which is more effective than traditional attacks because messages are generated in a natural and personalized way.
Using AI to automate attacks on electronic payment systems, e.g., by generating fake transactions.

Real-world cases of AI-powered attacks: what happened and what did we learn?

1. Attack on a technology company – February 2024

In February 2024, fraudsters used AI to impersonate the CEO of a technology company. They generated a fake voice that sounded identical to the CEO's and convinced a finance department employee to transfer 25 million dollars to an account in Singapore. source: Bloomberg, February 4, 2024.

This incident showed how easy it is to carry out a highly effective social engineering attack using AI. Key takeaways are:

Even the most advanced security systems can be bypassed by a well-prepared social engineering attack.
Organizations must implement additional verification mechanisms, such as identity confirmation via alternative channels (e.g., phone).
Employee training should cover not only recognizing phishing but also identifying deepfakes.

2. Bank customer data leak – November 2023

In November 2023, a bank in Europe fell victim to a phishing attack where cybercriminals used AI to generate messages in over 50 languages. The messages were so well-written that many customers did not recognize them as fraud. source: Kaspersky, December 2023.

This attack had serious consequences:

Leak of personal and financial data of over 200,000 customers.
Financial losses for the bank related to the need to reimburse customers.
Loss of customer trust and brand damage.

Takeaways are:

Traditional phishing detection methods, such as spell-checking, are insufficient against AI-assisted attacks.
Organizations should invest in advanced anomaly detection systems that can identify suspicious behavior in real-time.
It is important for customers to be aware of new fraud techniques and know how to protect themselves.

3. Attack on an AI system in healthcare – 2023

In 2023, there were cases of attacks on AI systems in healthcare involving the manipulation of diagnostic data. Cybercriminals introduced subtle changes to medical images, leading to misdiagnoses. source: MITRE, December 2023.

The consequences of such attacks can be catastrophic:

Misdiagnoses leading to improper patient treatment.
Medical data leaks used for blackmail or identity theft.
Loss of patient trust in AI-based systems.

Takeaways are:

AI models in healthcare must be constantly monitored for suspicious changes in input data.
It is important that training data is regularly updated and verified for malicious elements.
Organizations should cooperate with regulators to develop security standards for AI systems in healthcare.

Practical steps for organizations: how to adapt to the new threat reality?

Technical steps: AI system resilience

To minimize the risk of attacks on AI systems, organizations should implement the following technical solutions:

Anomaly detection: Implement systems that monitor AI model behavior in real-time and detect suspicious activities. Examples include Microsoft Defender for Cloud Apps or Darktrace.
AI model hardening: Secure models against adversarial attacks through techniques such as adversarial training or input validation.
Isolate AI models: Store and run models in sandbox environments to limit the risk of external attacks. An example is Google’s Confidential Computing.
Encrypt training data: Ensure that data used to train AI models is protected from unauthorized access. This includes encrypting data at rest and in transit.
Detect deepfakes: Implement tools that can identify fake voices, images, or videos. Examples include Microsoft Video Authenticator or Adobe’s Content Authenticity Initiative.

Organizational steps: policies, training, and incident response

In addition to technical solutions, organizations must adapt their policies and procedures to the new threat reality:

Update security policies: Introduce bans on using generative AI tools to create confidential documents or internal communications. Policies should also cover procedures for handling suspected AI-powered attacks.
Employee training: Regular training on recognizing deepfakes, AI-generated fake messages, and new phishing techniques. Training should cover both technical aspects and practical attack examples.
Incident response plan: Include AI attacks in Incident Response procedures. This means preparing for scenarios where cybercriminals use AI to carry out attacks or hide their activities. Procedures should include steps such as system isolation, root cause analysis, and cooperation with law enforcement.
Collaborate with regulators: Organizations should follow guidelines from agencies like the NCSC, ENISA, or CISA and adjust their practices to changing regulations. Participating in industry working groups focused on AI security is also important.

Implementation example: how one organization adapted to new threats

To illustrate how organizations can implement NCSC and other agency recommendations, let's look at an example of a financial sector company that took the following steps:

Assess AI model vulnerabilities: The company audited its AI systems for susceptibility to adversarial attacks and data poisoning. Several weak points were identified and fixed by implementing adversarial training techniques.
Employee training: All employees underwent training on recognizing deepfakes and new phishing techniques. The training also included practical exercises where participants could test their skills in identifying fake messages.
Monitor AI activity: The company implemented Darktrace, which monitors AI model behavior in real-time and detects suspicious activities. This system allowed for the rapid detection and blocking of an attack that occurred in February 2024.
Update security policies: Company policies were updated to include a ban on using generative AI tools to create confidential documents. Additionally, procedures were introduced to verify the authenticity of external communications.
Collaborate with regulators: The company regularly participates in industry meetings organized by the NCSC and contributes to the development of new AI security standards.

Thanks to these steps, the company was able to effectively minimize AI-related attack risks and prepare for upcoming challenges.

Summary: why is adapting to the new threat reality crucial?

Artificial intelligence brings enormous benefits but also serious challenges for cybersecurity. New attack vectors, such as deepfakes, phishing automation, or attacks on AI models, are becoming increasingly common and effective. Reports from the NCSC, ENISA, and CISA clearly indicate that organizations that do not adapt their policies and procedures to the new reality will be exposed to serious financial losses, loss of customer trust, and system disruptions.

Key takeaways from this article are:

AI democratizes cybercrime, enabling attacks on an unprecedented scale and effectiveness.
New attack vectors, such as deepfakes and phishing automation, require advanced protection mechanisms, such as anomaly detection and content authenticity verification.
Organizations must implement both technical solutions (e.g., AI model hardening, system isolation) and organizational ones (e.g., training, security policy updates).
Real-world attack cases, such as the attack on a technology company in February 2024, show how important it is to adapt quickly to new threats.

The future of cybersecurity depends on how quickly organizations can adapt to the new reality. Those that take appropriate steps now will be able to effectively protect themselves against upcoming challenges and use AI to build more resilient systems.

The time to act is now. Is your organization ready?