Emerging Technology & AI Security: Mission Domain Intelligence Guide

The finance team approved the transfer because they recognised the chief executive on the call. Voice, face and cadence were all correct, and none of them belonged to a person on that call.

What Emerging Technology & AI Security covers as a mission domain

This domain covers the security and abuse implications of technologies that are outpacing the controls built around them: machine learning systems and their supply chains, generative models and synthetic media, autonomous and agentic software, the migration to post-quantum cryptography, and pervasive connected devices. Analysts work two directions at once. Defensively, they assess how an organisation's own artificial intelligence deployments can be attacked through data, models, prompts and integrations. Offensively-aware, they track how the same tools change the economics of fraud, influence operations and intrusion.

Sub-areas include model supply chain risk in shared model repositories and unsafe serialisation formats, prompt injection and tool abuse in agentic systems, training data poisoning and extraction, synthetic identity and biometric spoofing against onboarding controls, synthetic media used in fraud and harassment, and provenance infrastructure such as content credentials. Actors range from established fraud crews adopting cheaper tooling to state influence units and opportunistic individuals with no technical background at all.

Why it matters

The change is not that new attacks became possible but that expensive ones became cheap. Convincing voice cloning from a few seconds of audio moves executive impersonation fraud from rare to routine. Fluent, culturally specific text removes the language errors that used to make phishing detectable. Synthetic imagery enables large-scale harassment and non-consensual abuse. Meanwhile, organisations are connecting language models to production systems faster than they are threat modelling them.

What analysts actually look for

These are the concrete, observable signals that carry weight in this area of work:

Voice-cloned calls to finance or treasury staff pressing for urgency and secrecy, often paired with a compromised or spoofed mail thread
Fraud campaigns at scale showing identical structural scaffolding and register across many languages, with no translation errors
Identity verification submissions with liveness artefacts: inconsistent specular reflections, frame-rate mismatch or repeated background geometry
Model files distributed in unsafe serialisation formats from unverified repository accounts, executing code on load
Prompt injection payloads embedded in web pages, documents or ticket text aimed at agentic systems that browse or ingest them
Content credential chains that are absent, stripped by re-encoding, or present but signed by an unrelated or newly created issuer
Anomalous query patterns against a hosted model consistent with systematic extraction or training data recovery attempts
Sudden appearance of synthetic imagery or audio of named executives, officials or private individuals in coordinated posting clusters

Where the data comes from

Authoritative and openly available collection points. Always confirm licensing and terms before operational or commercial use:

NIST AI Risk Management Framework — Structured approach to identifying, measuring and governing artificial intelligence risk across a system lifecycle
MITRE ATLAS — Adversarial technique taxonomy for machine learning systems, with documented real-world case studies
OWASP Top 10 for Large Language Model Applications — Practical vulnerability classes for deployed model applications, including injection and insecure output handling
C2PA specification and Content Authenticity Initiative — Provenance and content credential standards, plus their known limitations under re-encoding
AI Incident Database — Curated record of publicly reported harms and failures, useful for grounding risk claims in real cases
UK AI Safety Institute and NIST evaluation publications — Independent capability and safety evaluations of frontier systems from government research bodies
ENISA and national regulator guidance on artificial intelligence — Emerging compliance expectations, supervisory positions and sector-specific guidance on deploying artificial intelligence lawfully
arXiv preprints in adversarial machine learning — Leading indicator of attack techniques months before they appear in commodity tooling

A working method

A repeatable sequence beats ad-hoc searching. This is a practical starting workflow:

Inventory the estate — Identify every model, agent, integration and data flow in use, including those procured by business teams outside technology governance.
Threat model against a taxonomy — Work through data poisoning, extraction, injection, tool abuse and supply chain risk for each system, recording which are plausible given actual access.
Test within scope — Run structured red team exercises against your own deployments with written authorisation, and record findings against reproducible prompts and inputs.
Instrument provenance and verification — Deploy content credential checks where feasible and, more importantly, out-of-band verification for any instruction that moves money or access.
Monitor brand and likeness abuse — Watch for synthetic content impersonating your executives, officials or services, and pre-agree a takedown and communications route.
Rehearse the response — Exercise a deepfake-enabled fraud scenario end to end, including how staff are expected to challenge a convincing senior voice.

How this connects across the intelligence taxonomy

Intelligence work does not respect neat boundaries. The mission domain you are working, the disciplines you practise, and the data points you pivot on are one connected system. These are the direct relationships for this entry — every link is also a tag, so you can follow any thread across the whole library.

Practised with these disciplines

Technical Intelligence — Technology Capability, Design, and Exploitation
Academic Intelligence — Research Output, Collaboration, and Expertise
Patent Intelligence — Patents, Filings, and Innovation Signals
Cyber Intelligence — Adversary Activity in Networks and Systems
Risk Intelligence — Structured Assessment of Threat and Consequence
Disinformation Intelligence — Detecting and Analyzing Information Manipulation

Worked in these data points

Software Package — A published dependency (npm, PyPI, Maven) — the vector for supply-chain compromise.
Code Repository — A source-code repository — leaks secrets, reveals developers, and anchors supply-chain risk.
Patent — An intellectual property filing granting invention rights.
Credential / API Token — An exposed secret — API key, token, or JWT — granting access to systems and data.
Keyword / Narrative — A search term, topic, hashtag, or narrative tracked across media and platforms.
Company / Organization — A legal entity — corporation, LLC, NGO, or business.

Adjacent mission domains

Inside the platform: where Emerging Technology & AI Security lives

The Quantus platform is 204 pages behind a 147-item sidebar organised into six working groups: Command (24 items), Dashboards (15), Threat Theaters (14), Intelligence Domains (15), Investigate (34), and Administration (45). This entry is not a page in isolation — it is a thread running through several of them.

The modules that matter most here:

domain.php?d=emergingtech — Emerging Technology & AI Security dashboard
theater.php?d=emergingtech — Threat theater view
search.php — Advanced search, filter and pivot
correlate.php — Correlation graph
cases.php — Case management

Each dashboard is local-first: it renders from the platform’s own database rather than depending on a live third-party call, so it still works when an upstream API is unreachable or rate-limited. Heavy aggregates are cached with a hard query time cap and degrade to the last good value instead of hanging the page.

Automation, playbooks and AI skills

Analysis that only happens when someone remembers to run it is not a capability. The platform ships a 30-step automation pipeline (cron.php) that collects, ingests, resolves, enriches, correlates and scores on a schedule — 25 seeders, 11 resolvers and 7 enrichment runners, all idempotent and cursor-based so a run can be interrupted and resumed without duplicating or losing work.

Relevant playbooks

Of the 14 incident playbooks in playbooks.php, these apply directly to Emerging Technology & AI Security:

APT Intrusion Analysis — a step-checked workflow with the pivots, sources and handling rules already wired in.
Malware Triage — a step-checked workflow with the pivots, sources and handling rules already wired in.

AI skills that apply

The 16 one-click operations in ai-skills.php are deterministic jobs, not free-text generation. The ones that matter here:

Threat Hunt
Correlate Infrastructure
Run Alert Rules
Summarise (Copilot)
Generate Report

Alerting closes the loop: rules in alerts.php fire on new indicators matching a saved query, so a first sighting in this area raises a notification rather than waiting to be noticed at the next review.

Feeds, data sources and the API

The collection layer runs a feed registry of free, machine-readable sources — bulk blocklists and trackers (Maltrail, IPsum, FireHOL, the full abuse.ch corpora, phishing databases, Emerging Threats, Spamhaus, DigitalSide, ThreatView), authoritative government feeds (CISA KEV, OFAC, UN and EU sanctions lists), and reference datasets (RIR allocations, ip-to-ASN and geolocation tables, MITRE ATT&CK, EPSS). collect.php pulls them server-side on a schedule; feeds.php and source-catalog.php show what is registered, what it covers and when it last ran.

Anything the platform holds is reachable programmatically. The REST API in api.php exposes 11 endpoints — status, stats, search, lookup, recent, export, bulk_check, top_threats, by_category, categories, check — and export.php streams 18 formats in bounded chunks, so a million-row export neither exhausts memory nor times out:

STIX 2.1, MISP, OpenIOC 1.1, CEF (ArcSight), LEEF 2.0 (QRadar), Zeek/Bro intel, Snort/Suricata rules, Palo Alto EDL, BIND RPZ, hosts blackhole, iptables, CSV, JSON, NDJSON/JSONL, XML.

That covers the CTI standards (STIX 2.1, MISP, OpenIOC), SIEM ingestion (CEF, LEEF, Zeek), detection engines (Snort/Suricata), and direct enforcement (Palo Alto EDL, BIND RPZ, hosts, iptables) — so intelligence developed here can be actioned in the tools you already run, without a manual reformatting step. A TAXII 2.1 server and a MISP/RSS feed are also served for pull-based sharing.

Use cases

Three ways this entry earns its keep in day-to-day work:

Triage under time pressure. An artifact or report lands and you need a defensible read in minutes, not days. Inventory the estate is the first move; the platform pre-computes the enrichment so the analyst spends the time on judgement rather than lookups.
Building the picture. A single indicator is rarely the story. Test within scope turns one artifact into a network — shared infrastructure, repeated selectors, the same operator behind different names — via the correlation graph and the cross-entity link engine.
Producing something actionable. Analysis that ends in a document nobody can use is wasted. Rehearse the response feeds the case file, the detection rule, the block list or the referral — with sourcing attached so the recipient can verify it.

Case management (cases.php), watchlists, saved searches and scheduled reports mean the work persists between sessions and survives an analyst leaving the team.

How each sector uses Emerging Technology & AI Security

The same entry is worked very differently depending on who you are, what authority you hold, and what you are ultimately producing. A military analyst is supporting a commander’s decision; a journalist is meeting a publication standard; an NGO caseworker is protecting a person. The underlying artifacts are shared — the constraints, outputs and thresholds are not.

🎖 Military and defence

Defence interest centres on synthetic media in information operations, autonomy in systems, model supply chain integrity and the security of artificial intelligence used in decision support. Analysts assess what a capability can actually do against contractor claims, which matters because procurement decisions rest on it. Products feed capability assessment, force protection where synthetic media targets personnel, and doctrine development for human oversight of automated systems. Constraints include export control on certain models and hardware, the requirement that any system informing lethal decisions has documented human accountability, and the discipline of distinguishing demonstrated performance from demonstration conditions.

🕵 National intelligence

National services track the diffusion of capability: who has access to compute, models and expertise, how synthetic media changes influence operations, and how adversaries use automation to scale collection and targeting. Requirements also cover the defensive side, since agencies deploy these systems themselves and inherit their failure modes. Fusion of technical assessment, procurement tracking and open research monitoring is the working method. Handling is complicated because the research literature is open while its application may be sensitive, and because assessments about capability diffusion age quickly. Products should state the assumptions about compute and expertise on which the judgement depends.

👮 Law enforcement

Law enforcement encounters this domain as synthetic media used in fraud and harassment, generated child sexual abuse material, automated fraud at scale, and evidential questions about whether recordings are authentic. Evidence handling now requires provenance analysis of media, and courts increasingly expect expert evidence on authenticity. Legal frameworks are uneven: some jurisdictions criminalise non consensual synthetic intimate imagery and generated abuse material explicitly, others rely on existing offences. Investigators need documented, reproducible authentication method rather than reliance on a detection tool score, because such scores are unreliable and will be challenged.

🔍 Private investigation and corporate security

Corporate teams handle executive impersonation in payment fraud, synthetic identity in onboarding, model and data supply chain risk in products the company deploys, and prompt injection against internal assistants connected to business systems. The practical products are control recommendations and incident reconstruction. A private actor may not test third party systems without authorisation, scrape personal data in breach of law, or deploy biometric analysis of individuals without a lawful basis. Diligence on vendors should ask concrete questions about training data provenance, model update process and evaluation evidence rather than accepting marketing claims about safety.

📰 Journalism and OSINT media

Newsrooms face this from both sides: verifying media that may be synthetic and being targeted by fabricated content. Verification standards must be explicit, because detection tools produce unreliable outputs and publishing a false authentication is worse than publishing nothing. Provenance metadata standards help where present but are absent from most content. Practical method combines source chain verification, contextual corroboration, technical analysis by a specialist and honest reporting of residual uncertainty. Ethics include not amplifying synthetic content by republishing it, describing manipulation without providing a recipe, and correcting quickly and prominently when an authentication turns out to be wrong.

🌍 NGO, humanitarian and human rights

Human rights organisations use these systems for documentation at scale and face them as tools of repression: synthetic media discrediting activists, automated surveillance and content moderation that removes evidence of atrocities. Practice must be victim centred and cautious, because a wrongly authenticated or wrongly dismissed piece of evidence has consequences for real cases. The liar's dividend, where genuine footage is dismissed as fabricated, is now a routine defence and documentation practice should anticipate it through provenance capture at source. Duty of care includes preparing subjects for synthetic harassment and providing support when it occurs.

🎓 University and research

Research spans model security, evaluation methodology, synthetic media detection and societal impact. Reproducibility is a serious problem because model versions change silently, so record model identifiers, dates, parameters and prompts, and archive outputs. Detection research must report performance on out of distribution and adversarially perturbed data, since in distribution accuracy is close to meaningless for deployment. Ethics approval is needed for studies involving generated likenesses of real people, for participant deception, and for red teaming that could produce harmful artefacts. Publish evaluation code and be explicit about what a benchmark does not measure.

Playbook: working Emerging Technology & AI Security end to end

A repeatable sequence, from the moment the requirement lands to the moment a product is delivered and the case is closed out. Each phase states what you are trying to establish, not merely what to click — the point is a defensible chain of reasoning, not a checklist.

Phase 1 — Inventory where these systems touch the organisation

Find every deployed model, assistant, automated decision system and third party feature already in use, including the ones adopted without approval. Record what data each touches, what actions it can take and who owns it. Most organisations discover more than they expected. A good output is an inventory with an owner and a data classification per entry, which is the precondition for any meaningful risk work.

Phase 2 — Map the supply chain

For each system, establish where the model came from, what it was trained on as far as that is knowable, which third party components and datasets are involved, and how updates are delivered. Note where a supplier can change behaviour without notice. The output is a dependency picture that identifies the small number of upstream providers whose compromise or change would affect many of your systems.

Phase 3 — Define unacceptable outcomes

Work from consequence: unauthorised payment, disclosure of personal or privileged data, defamatory or unsafe output presented as authoritative, or an automated action affecting a person's rights. Rank by severity and reversibility. This focuses effort on the systems with real authority rather than on the ones that merely sound advanced. A good output is a short list agreed with legal and business owners that drives control design.

Phase 4 — Assess the trust boundary

For any system that reads untrusted content and can take action, identify what an attacker controls and what the system can do with it. Retrieval, browsing, email reading and document ingestion all place attacker controlled text inside the context. Assume that content can influence behaviour. The output is a boundary diagram showing where untrusted input meets privileged capability, which is where controls belong.

Phase 5 — Apply architectural controls, not just prompting

Constrain what the system can do: least privilege on tools and data, human confirmation for consequential actions, allowlists on outbound requests, and separation between reading untrusted content and acting on it. Instruction level mitigations alone are not a control. A good output is a design where a successful injection produces an inconvenience rather than a payment, a data disclosure or a code change.

Phase 6 — Evaluate before deployment

Test with adversarial inputs, edge cases and the specific misuse scenarios your users will encounter, not only benchmark tasks. Include evaluation of failure behaviour: what the system does when uncertain, and whether it signals that. Record results against a defined threshold agreed in advance. The output is evidence that would satisfy a regulator asking why you considered the system fit for its purpose.

Phase 7 — Instrument for detection

Log prompts, retrieved content, tool calls and outputs with retention appropriate to the sensitivity, and monitor for anomalous tool usage, unusual data access and outputs that indicate manipulation. Personal data in these logs is still personal data. A good output is telemetry that lets you reconstruct an incident afterwards, which most deployments cannot currently do. Define retention up front, since these logs frequently contain personal data.

Phase 8 — Build a media authentication capability

Establish a documented process for assessing whether audio, images or video are authentic: source chain, provenance metadata where present, contextual corroboration and technical examination. Do not rely on a detection tool score, which is unreliable and will be challenged. The output is a method and a record for each assessment stating what was checked and what uncertainty remains. Have the process reviewed by someone who will have to defend it publicly.

Phase 9 — Prepare for impersonation of your own people

Executive voice and video impersonation is now routine in payment fraud and in harassment of public figures. Establish verification procedures that do not depend on recognising a voice or face, brief finance and executive support staff, and prepare a response plan for fabricated content about the organisation. A good output is a tested out of band verification step for any consequential instruction, regardless of how convincing the request appears.

Phase 10 — Address generated abuse material and harassment

Where synthetic sexual imagery of adults or children is encountered, treat it under the same protective rules as any abuse material: do not retain or circulate, refer through the designated hotline or police channel, and support the person depicted. Generated child sexual abuse material is criminal in an increasing number of jurisdictions and is always a referral matter. The output is a written escalation route briefed in advance, not a decision taken under pressure.

Phase 11 — Govern with named accountability

Assign an accountable owner for each deployed system, define review triggers when models are updated, and record decisions about acceptable use. Governance must be able to say who authorised a system to take a particular class of action. A good output is a register that answers, for any given automated decision, who is responsible and what evidence supported allowing it.

Phase 12 — Track capability change deliberately

Assumptions in this field expire quickly. Set review points tied to model releases, cost changes in compute and published evaluation results, rather than annual cycles. Record which of your judgements depend on capability limits that may not hold next year. The output is an assessment that flags its own expiry conditions instead of quietly becoming wrong. Record the specific capability assumption that each judgement depends on.

The platform ships this as a step-checked workflow in playbooks.php, so progress is recorded against a case rather than held in someone’s head.

Source register: what to collect from, and how

Sources are listed with their access model so you can plan around cost and licensing before you build a dependency on them. Open means no account required; registration means a free account or API key; licensed means paid or institutional access. Always confirm current terms — licensing changes, and a source that was free for research may not be free for commercial or evidential use.

Source	Access	What it gives you	How it is used here
MITRE ATLAS	Open	Adversarial threat landscape taxonomy for artificial intelligence systems, with case studies and mitigations.	Structuring threat modelling of deployed machine learning systems in a recognised vocabulary.
OWASP Top 10 for Large Language Model Applications	Open	Community catalogue of common vulnerability classes in deployed language model applications. Maintained by practitioners and updated as deployment patterns change.	Baseline checklist for reviewing assistant and retrieval architectures before deployment. Use as a review checklist rather than as a compliance standard.
NIST AI Risk Management Framework	Open	Voluntary framework for governing, mapping, measuring and managing risk in artificial intelligence systems.	Governance structure that maps to what regulators and enterprise customers expect to see.
EU AI Act text and guidance	Open	European regulation establishing risk tiers, obligations and prohibited practices for artificial intelligence systems.	Determining classification and compliance obligations for systems used or placed on the EU market.
C2PA content provenance specification	Open	Technical standard for cryptographically signed provenance metadata attached to media at creation and edit time.	Verifying provenance where signing is present and advising on capture practice for documentation.
AI Incident Database	Open	Collected reports of real world harms and failures involving deployed artificial intelligence systems.	Grounding risk assessment in documented incidents rather than speculative scenarios. Useful for showing executives what has actually gone wrong elsewhere.
arXiv preprint server	Open	Open repository where most machine learning security and capability research appears first.	Tracking attack and defence developments months before they reach vendor documentation. Preprints are not peer reviewed, so assess claims independently.
Hugging Face model and dataset hub	Registration	Repository of published models and datasets with cards describing training data, licence and intended use.	Assessing provenance and licence of models entering the organisation's supply chain. Model cards vary widely in quality and completeness.
Partnership on AI and civil society research	Open	Multi stakeholder guidance on responsible deployment, synthetic media practice and documentation norms.	Reference practice for provenance disclosure and responsible handling of generated media. Useful when setting internal norms for disclosure of generated content.
NCSC and national guidance on AI security	Open	Government guidance on secure development and deployment of machine learning systems. Issued jointly with international partners covering the development lifecycle.	Authoritative control guidance to cite in internal standards and supplier requirements. Also useful as a baseline in supplier security requirements.
Citizen Lab and academic disinformation research	Open	Documented cases of synthetic media use in influence operations and harassment of individuals.	Case evidence for how these techniques are actually deployed against targets. Case detail is stronger evidence than speculative capability assessment.
Internet Watch Foundation reporting on generated material	Open	Assessment and reporting on synthetic child sexual abuse material and its distribution trends.	Authoritative referral route and trend evidence for this specific and criminal category.
Vendor model cards and system cards	Open	Published documentation of model capability, evaluation results, limitations and intended use. Quality varies, and absence of an evaluation is itself informative.	Comparing claimed evaluation coverage against your own deployment scenario. Ask what was not evaluated as well as what was.
Export control lists covering advanced computing	Open	National controls on advanced semiconductors and related technology relevant to capability diffusion.	Assessing compliance exposure and tracking constraints on compute access. Controls change frequently, so check current listings before advising.

Prefer sources that publish a methodology and a revision history. A dataset that changes silently is a liability in any product that has to survive challenge.

Tooling

Tools commonly used against Emerging Technology & AI Security. None of these replace judgement, and each carries its own failure modes — know what a tool infers versus what it observes.

Adversarial evaluation and red team harnesses — Systematically probe deployed systems with attack prompts and edge cases. Limitation: coverage is bounded by the imagination of the test set, and models change under you.
Prompt and tool call logging platforms — Record full context, retrieval and actions for later reconstruction. Limitation: logs contain sensitive data and create their own retention and access obligations.
Provenance verification tooling for C2PA — Validates signed provenance manifests on media where present. Limitation: the overwhelming majority of circulating media carries no provenance data at all.
Synthetic media detection classifiers — Estimate the likelihood that media was generated or manipulated. Limitation: unreliable across generators and easily defeated by compression or re-encoding, so never sufficient alone.
Model and dependency scanning — Checks model artefacts and libraries for known vulnerable components and unsafe serialisation formats. Limitation: does not assess training data provenance or behavioural risk.
Retrieval isolation and sandboxed tool execution — Separates untrusted content ingestion from privileged action. Limitation: adds latency and complexity, so teams under delivery pressure disable it.
Reverse image and video search — Locates earlier appearances of media to test claims about origin and date. Limitation: coverage is skewed to indexed web content and misses closed platforms.
Benchmark suites for capability evaluation — Provide comparable measures across models on defined tasks. Limitation: benchmark performance frequently fails to predict behaviour in the deployment context that matters.

AI skills and automation in detail

These are deterministic jobs with defined inputs and outputs, not open-ended prompting. Each is idempotent and cursor-based: interrupt one and it resumes where it stopped rather than duplicating work or losing progress.

Threat Hunt — Runs saved hypotheses against the corpus and surfaces what matches, with the query preserved as a versioned artifact.
Correlate Infrastructure — Builds the cross-entity link graph: shared hosting, reused certificates, overlapping registrants, repeated selectors.
Run Alert Rules — Evaluates saved rules against new data so a first sighting raises a notification rather than waiting for review.
Summarise (Copilot) — Produces a narrative summary beside the underlying records. It explains; it never creates indicators or assigns attribution.
Generate Report — Assembles a sourced product from the current case or query, with provenance attached to each element.

A note on the boundary: the only skill that involves a language model is Summarise (Copilot), and it writes prose about records that already exist. Nothing else on this list involves generation of any kind. No indicator, relationship or attribution in the platform originates from a model. See the full skill list.

Tradecraft notes

The distinctions that separate a competent analyst from a fast one:

Treat any system that reads untrusted content and holds privileges as an attack surface, not as a tool. The interesting question is never whether it can be manipulated but what it can do once it has been.
Detection scores for synthetic media are not evidence. Authentication rests on source chain, contextual corroboration and technical examination, and a report that leads with a classifier percentage will not survive challenge.
The liar's dividend is now the more common harm. Genuine footage dismissed as fabricated damages accountability work more often than fabricated footage is believed, which argues for provenance capture at the point of documentation.
Model updates are silent changes to a dependency you cannot audit. Any assessment tied to a specific model version should record identifiers and dates, and any control tied to model behaviour should be re-tested after updates.
Benchmarks measure the benchmark. Evaluate on the actual deployment scenario with the actual data and the actual users, because generalisation from public leaderboards to production has repeatedly failed.
Voice and video are no longer authentication factors. Any process that relies on recognising a person on a call needs an out of band verification step, and this should be treated as a settled control requirement rather than a debate.
Capability judgements expire. State the compute, cost and expertise assumptions underlying any assessment of what an actor can do, so the judgement can be revisited when those assumptions change rather than silently becoming wrong.

Measuring whether it is working

Capability claims should be falsifiable. These are the measures that show whether work on Emerging Technology & AI Security is producing anything, and they are worth baselining before you change process or tooling.

Proportion of deployed systems in the inventory with a named owner, data classification and documented evaluation before release.
Number of consequential actions that require out of band human confirmation, as a share of the actions automated systems can take.
Time from a model or vendor update to completion of re-evaluation against the organisation's own test set.
Rate of successful out of band verification on attempted executive impersonation, measured through exercises as well as real events.
Share of media authentication assessments that record method and residual uncertainty rather than a tool score alone.
Incidents where injected content caused an unauthorised action, tracked toward zero as architectural controls mature.
Coverage of the deployed estate against a recognised AI threat taxonomy, reported with the telemetry that supports each control.

Beware of measuring volume alone. Indicator counts and report counts rise easily and say little; time-to-attribution, proportion of findings that survive review, and how often a product changed a decision say a great deal.

Common pitfalls

Treating a synthetic media detector score as a verdict; these classifiers generalise poorly and fail on compressed or re-encoded media
Assuming watermarks and provenance metadata survive screenshots, re-encoding or deliberate stripping, when they usually do not
Prioritising exotic model attacks while the actual incidents remain stolen credentials, unpatched edge devices and invoice fraud
Accepting vendor capability claims without testing against your own data, particularly for detection and guardrail products
Confusing fluent output with authoritative content, and allowing model-generated analysis into products without source verification
Ignoring the shadow estate of business-procured tools, which is where sensitive data most often leaves the organisation

Legal and ethical considerations

The regulatory picture is tightening quickly: the European Union artificial intelligence regime imposes obligations by risk tier, and biometric processing triggers special category data rules almost everywhere. Training data raises intellectual property and data protection questions that a security assessment cannot resolve alone. Synthetic media evidence needs careful handling, since authenticity claims will be contested in court. Synthetic sexual imagery of children is illegal in most jurisdictions regardless of how it was produced, and must be reported through the same authorised channels as any other such material.

Data integrity: no fabrication, no drift, no hallucination

Intelligence that cannot be traced back to a source is not intelligence, it is assertion. Everything in this entry — and everything in the platform behind it — is built on a small number of non-negotiable rules.

Provenance on every record

Every indicator carries the source that supplied it, a first-seen and last-seen timestamp, and a sighting count. Where several feeds report the same artifact, each contribution is recorded separately rather than collapsed, so you can see whether a finding rests on one source or twelve. Source attribution travels with the data into every export, so a recipient can audit a claim without asking you for the working.

Nothing is invented to fill a gap

If the platform has no data for Emerging Technology & AI Security, it says so. Empty is displayed as empty — never padded with plausible-looking placeholder values, sample records or illustrative examples that a reader might mistake for observations. A dashboard with no rows is a true statement about collection coverage, and it is treated as a gap to close, not a blemish to hide.

Scoring is deterministic and reproducible

Threat scores, reputation grades and risk tiers are computed from stated inputs with fixed weights, not estimated. The same inputs always produce the same output, and the formula is visible rather than a black box. Aggregates are cached with an explicit time-to-live so a figure on screen is never silently stale — and when a heavy query exceeds its time budget the platform serves the last known-good value and labels it, rather than inventing a fresh number or hanging.

Where AI is used, and where it is not

Language models summarise and explain. They do not create indicators, assign attribution or manufacture relationships. No IP address, wallet, hash or identity in the platform originates from a model — every one is ingested from a named feed, resolved from a reference dataset, or entered by an analyst with a source recorded. Copilot output is presented as narrative alongside the underlying records, never in place of them, so a reader can always check the summary against the evidence.

Guarding against drift

Enrichment is additive and timestamped rather than overwriting. Reference data — sanctions lists, allocations, taxonomies — is re-synchronised from the authority on a schedule instead of being edited in place, so local copies cannot quietly diverge from the source of truth. Attribution is recorded with a confidence level and the reporting it rests on, and inferred relationships are labelled as inferred. When a source retracts or corrects, the correction propagates rather than leaving a stale assertion behind.

What this means for you

You can put a finding from this platform in front of a regulator, a court, a board or a partner agency and show where each element came from. That is the standard the tooling is built to — because in this work, being confidently wrong is more damaging than being usefully uncertain.

By the numbers

The taxonomy this entry belongs to is not a marketing list — it is the actual structure of the platform: 52 mission domains, 52 intelligence disciplines and 65 data points, each with a live dashboard behind it. Supporting that: 18 indicator types, 14 playbooks, 16 AI skills, 18 export formats and a 30-step automated pipeline.

This particular entry connects directly to 6 intelligence disciplines, 6 data points, 5 closely related entries — every one of them a tag you can follow, and a dashboard you can open.

Questions analysts actually ask

Can synthetic media be reliably detected?

Not reliably enough to stand alone. Detection classifiers degrade badly across unseen generators, and ordinary compression, re-encoding or screen recording defeats many of them. They also produce confident errors in both directions, which is the worst property for evidential use. Practical authentication combines source chain analysis, provenance metadata where it exists, contextual corroboration such as weather, shadows and independent recordings, and technical examination by a specialist. Report residual uncertainty honestly. Where a determination cannot be made, saying so is a legitimate and often correct result.

What is the most serious practical risk from deploying assistants internally?

Indirect prompt injection against a system with real privileges. If an assistant reads email, documents or web content that an outsider can influence, and it can also send messages, call internal systems or modify data, then attacker controlled text can trigger those capabilities. The mitigations that work are architectural: least privilege on tools and data, separation between ingesting untrusted content and acting, allowlists on outbound destinations, and human confirmation for consequential actions. Instruction based defences reduce casual attempts but should never be the control you rely on.

How should an organisation respond to a deepfake of its executive?

Assume it is designed to trigger a payment or a market reaction and act on both fronts. Verify through an out of band channel immediately, alert finance and treasury to expect a fraudulent instruction, and preserve the media and its distribution path for investigation. Publish a clear, dated denial through the organisation's own verified channels rather than engaging with the content in detail. Report to law enforcement and to the platforms hosting it. Afterwards, test whether existing verification procedures would have stopped a payment, because most organisations discover that they would not.

Are provenance standards going to solve this?

They help where they are present, and they will not be present on most content for a long time. Signed provenance is most valuable at the point of documentation: human rights organisations, journalists and investigators capturing evidence can adopt it now and materially strengthen later authentication. For content arriving from the open internet, absence of provenance says nothing, since most genuine media also lacks it. Treat provenance as a positive signal when valid, never as a negative signal when missing, and do not let its existence displace conventional verification.

How do we do vendor diligence on an AI supplier?

Ask questions with verifiable answers. What model versions are used and how are changes notified; what data is retained, for how long and is it used for training; what evaluation was performed for scenarios like ours and can we see the results; what happens when the system is uncertain; who is accountable for an incorrect output that harms a customer. Require contractual notification of material model changes. Marketing claims about safety are not evidence, and a supplier unable to describe its own evaluation methodology is telling you something important.

What is the correct handling of generated child sexual abuse material?

Exactly the same protective handling as any such material: stop, do not view further, do not download, copy or retain, and report immediately through the designated national hotline or police channel. Generated material is criminal in a growing number of jurisdictions and, regardless of legal classification, it is a referral matter and never a research or moderation exercise for a general team. Ensure staff are briefed on this before they can encounter it, and provide immediate welfare support. Organisations must have this route written down in advance.

Standards, frameworks and further reading

Work that references a recognised framework is easier to defend, easier to hand over, and easier for a partner to consume:

NIST AI Risk Management Framework, providing the govern, map, measure and manage structure for AI risk.
EU AI Act, establishing risk tiers, prohibited practices and obligations for providers and deployers.
ISO/IEC 42001, the management system standard for artificial intelligence governance.
MITRE ATLAS, the adversarial technique taxonomy for machine learning systems.
OWASP Top 10 for Large Language Model Applications, cataloguing common deployment vulnerability classes.
C2PA content provenance and authenticity specification, defining signed provenance metadata for media.
ISO/IEC 23894 guidance on AI risk management, complementing sector specific requirements.
GDPR and equivalent regimes, governing automated decision making, profiling and personal data used in training or inference.

References

Primary sources and authoritative references for this entry. Publishers revise and retire material, so treat the retrieval date as part of the citation and re-check before relying on any of it in a formal product.

AI Risk Management Framework — National Institute of Standards and Technology. Voluntary framework for governing artificial intelligence risk.
MITRE ATLAS — MITRE. Adversarial threat taxonomy and case studies for machine learning systems.
Top 10 for Large Language Model Applications — OWASP. Community catalogue of vulnerabilities in language model deployments.
Content provenance specification — Coalition for Content Provenance and Authenticity. Technical standard for signed media provenance.
AI Incident Database — Responsible AI Collaborative. Repository of documented real world AI harms and failures.
EU Artificial Intelligence Act — European Commission. Regulation establishing obligations by risk tier for AI systems.
Guidelines for secure AI system development — National Cyber Security Centre UK. Government guidance on securing machine learning systems across the lifecycle.
Reporting on synthetic child sexual abuse material — Internet Watch Foundation. Assessment and trend analysis with authorised reporting routes.

Link integrity: every reference above was verified with a live request when this page was generated. Where a publisher had moved or withdrawn a document, the link was repointed at a preserved copy in the Internet Archive and marked as archived. Anything with no reachable copy anywhere had its link removed rather than left to rot — the source is still credited, it simply cannot be linked.

Put it into practice

The Quantus Intel threat intelligence platform operationalises this entry: tracks synthetic media abuse, model supply chain exposure and injection risk across deployed artificial intelligence systems. Explore the platform, or browse the rest of the library by following any tag above.