Romance scams have become a multi-billion-dollar industry, using AI and forced labour to defraud victims on an unprecedented global scale.
ROMANCE scams have entered a “dark age”, evolving from disorganised individual actors into a multi-billion-dollar criminal enterprise.
New research and firsthand accounts reveal that scammers are now leveraging “frontier” and open-source AI to scale predatory operations within specialised forced-labour compounds.
According to the Federal Trade Commission, investment scams, the primary “endgame” for romance fraud, resulted in US$5.7 billion (RM22.28 billion) in losses in 2024, a figure experts believe is a conservative estimate.
The availability of powerful frontier AI models has provided digital gold for scammers. For the price of a cup of coffee, predators can now leverage these tools to generate linguistically perfect, emotionally resonant messages designed to ensnare victims across the globe.
Industrialisation of deception: Four key trends
There are four critical pillars currently driving this new era of fraud:
AI frontier: Scammers now use Large Language Models (LLM) to eliminate traditional “red flags” like broken grammar and inconsistent narratives. By automating the “grooming” phase, they can maintain dozens of highly persuasive, persona-driven conversations simultaneously.
AI room: Sophisticated operations now utilise dedicated “AI rooms” where deepfake technology enables real-time, face-swapped video calls. This allows a scammer to “prove” their identity visually, effectively dismantling the old advice to “just hop on a video call” to verify a match.
Investment pivot: Romance is now simply the hook for “pig butchering” schemes. Victims are systematically “fattened” with trust and staged financial success on fraudulent platforms before being “slaughtered” for their life savings.
Open-source threat: While frontier models have guardrails, free open-source models like DeepSeek and Qwen allow scammers to operate without ethical restrictions. These models now reach near-parity with paid services, providing a powerful, unrestricted toolkit for malicious use.
These scams are the engine of a multi-billion-dollar industry often built on the backs of trafficked individuals.
Inside these compounds, victims are forced to work sales floors governed by strict quotas. They even ring bells and gongs to celebrate when a victim’s life savings are stolen.
While the technology is new, the psychological manipulation is as old as time; it just happens at a scale we have never seen before.
Consumers are urged not to be swayed by screenshots of earnings or claims of insider expertise. If a match brings up investments, whether aggressively or coyly, it is a scam. If the conversation turns to money, the solution is simple: cut contact, unmatch and report.
Technical mechanics of the AI room
Scam compounds utilise specialised AI rooms to bypass visual verification. These setups use virtual camera software to intercept video feeds on platforms like WhatsApp or FaceTime.
Real-time face-swapping software, such as DeepFaceLive, maps the scammer’s facial movements onto a high-resolution target persona. Technical artifacts or glitches in the AI are often masked by the scammer claiming a poor internet connection or using low-light environments.
Role of open-source models (DeepSeek and Qwen)
While frontier models – for example, ChatGPT, Gemini – have built-in safety filters to refuse requests related to fraud, open-source models like DeepSeek and Qwen can be run locally on private servers. This allows scammers to remove all guardrails, enabling the generation of unlimited, unrestricted predatory content without fear of account suspension or detection by the AI provider.
Statistical context
The US$5.7 billion figure cited for 2024 refers specifically to investment fraud, which is the secondary phase of a romance scam, known as “pig butchering”. Total losses are estimated to be significantly higher as many victims do not report losses due to the perceived social stigma associated with romance-based deception.
Satnam Narang is a senior staff research engineer at Tenable with over a decade of experience in identifying and exposing social engineering trends. He is a leading expert on the intersection of AI technology and modern cybercrime. Comments: letters@thesundaily.com
The Sun Malaysia
