No man is an island, but it’s great to be one if you’re an AI company

Source: Dall-E, Datafund

There’s a concerning shift in the AI industry that we’ve already described in more detail in our previous blog “OpenAI’s latest drama: What are the larger currents underneath?”.

The industry is now in a race to capture an increasingly lucrative market, instead of prioritising a more cautious development of a highly potent and disruptive technology. And they’re not shying away from using social media 1.0’s bag of tricks either, while adding new ones of their own. “I worry that we are heading to a social media 2.0 like world,” commented Malwarebyte’s General Manager of Consumer Business Unit Mark Beare.

This blog will explore the tactics employed by major players in the AI industry to build deep moats around their AI business and the possible consequences of these actions:

1. Tapping into people’s real-time data to train personal AI agents

2. Anthropomorphising chatbots to create emotional dependency

3. Promoting trusted computing to impose selective use of AI technology

Let’s begin.

Tapping into real-time data: The shovel that can deepen the moat

Last week, Microsoft unveiled its latest feature, ‘Recall’, which will soon roll out to a new line of laptops. The problematic aspect is that this feature will indiscriminately capture snapshots of all user activity on their computers. While Microsoft claims that these snapshots will be encrypted and stored locally, ensuring user privacy, this doesn’t mean that they will not be able to extract immense value from this feature.

Here’s why:

• Vast user base: As discussed in our previous blog, AI companies are keen to tap into real-time data flows. With Recall, Microsoft gains indirect access to an enormous amount of live data. Windows 11 already boasts 400 million users, and Windows 10 has around one billion users. With Windows 10 support ending in 2025, a significant number of users will likely transition to Windows 11, the only version with Recall. The latter allocates 25 gigabytes of space for itself on the disk by default, potentially resulting in an influx of data measured in exabytes.

• Telemetry and Usage Data: Although Microsoft won’t have direct access to the snapshot content, they could still collect valuable metadata on how Recall is used. This includes information on how often it is accessed, which types of queries are made, and the general performance of the AI in responding to those queries. Such telemetry data can be used to improve underlying algorithms and models, even without direct access to sensitive content.

• Inference and Local Model Updates: The AI models running locally on devices could be updated using federated learning techniques. Federated learning enables model training on user devices, with only learned model updates (not raw data) sent back to Microsoft servers. This approach allows AI to learn from user data indirectly, ensuring sensitive data remains on-device.

The consequences

So, what does this mean for Microsoft’s position in the market and its competition? The release of Recall gives Microsoft access to an enormous user base, creating a deep moat that rivals will find challenging to cross. By integrating this feature into peoples’ daily flows, Microsoft theoretically gains access to vast amounts of free, high-quality training data.

This data can be used to enhance their AI agents and personal assistants, potentially creating a lock-in effect that competitors will find difficult to break. History has shown that such lock-ins leave users at the mercy of the one holding the keys.

The security and privacy implications of Recall are also cause for concern. Just a few days after its announcement, Recall was dubbed a potential privacy and security “nightmare” by experts. Cybersecurity expert Kevin Beaumont highlighted the risks, particularly regarding the potential for infostealer malware to exploit locally stored data. Microsoft itself has fallen victim to trojans, Beaumont warns, so one can easily imagine what months’ worth of all personal activity stored on millions of devices across the globe means. The scale of the problem is immense!

With Recall, Microsoft just might have built the biggest shovel to dig a moat or has handed one to its customer to dig the graves of their privacy. The only question is, who will do the digging.

Human-like chatbots: Playing a dangerous tune on the heartstrings

Anthropomorphising, the act of ascribing human-like traits to inanimate objects or phenomena, has been a common theme in robotics. However, the latest advancements in AI, particularly with OpenAI’s latest model featuring advanced audio and voice capabilities, have taken this to a new level. This model can engage in conversations that are strikingly human-like, creating a powerful hook that can latch onto a person’s subconsciousness.

The issue with these seemingly human-like machines is that as humans we have a natural tendency to project human characteristics onto inanimate objects. This happens if there are cues that prompt us to view them as something similar to humans. So if an AI exhibits lifelike behaviours, we tend to humanise it.

This is backed by extensive research. As noted in “The Ethics of Advanced AI Assistants”, users who interact with digital voice assistants (DVAs) with realistic voice reproduction “exhibit a concerning inclination to generalise purely human concepts to digital assistants”. Additionally, when they feel affinity and familiarity with a DVA, which is reinforced via jokes, chat, comforting advice etc. coming from a DVA, they are also reluctant to “replace their digital assistant with an equally capable substitute”.

Another prominent fact, emphasised in “The Ethics of Advanced AI Assistants”, is that “people suffering from persistent loneliness are likely to seek out and form human-like attachments to virtual companions to cope with their lack of social connections”.

This deep emotional entanglement not only creates a powerful product lock-in but also raises dangerous ethical and psychological concerns. Because emotionally vulnerable individuals can form such a strong connection to humanised AI systems, a break in this relationship could lead to genuinely traumatic experiences.

The consequences

We don’t know yet the full ramifications this technology will have on human emotions and internal worlds. However, researchers have already identified several near-term and distant future concerns:

Near-term future

• Privacy concerns — Emotional attachment brings with it the feeling of security and individuals may be inclined to share deeply personal facts with an organisation or an unknown actor that can leak or exploit this knowledge in unpredictable and harmful ways.

• Manipulation and coercion — By trusting and being emotionally dependent on an AI assistant, individuals may hand it excessive influence over their lives.

• Overreliance — Through trust people may be inclined to take advice from the assistant even if it is harmful to them.

• Violated expectations — Feelings of profound disappointment, frustration and betrayal may arise when a non-feeling and unpredictable AI fails to meet a person’s expectations.

• False notions of responsibility — Individuals may develop a notion that they are responsible for “their” AI’s feelings.

More distant future:

• Degradation — Relations with AI prevail over human-to-human relations.

• Disorientation — AI takes over an individual’s opinion space.

• Dissatisfaction — The feeling of unfulfillment with the majority of relations being taken over by AI.

(The Ethics of Advanced AI Assistants, 2024)

Limiting access to hardware: Who gets to decide

In early May, OpenAI published a blog post outlining “six security measures for advanced AI infrastructure”. The first measure, “Trusted computing for AI accelerators”, proposes the introduction of cryptographically attested GPUs that support encrypted model weights. In this scenario only selected organisations and/or individuals would be able to run AI models on selected GPUs.

This proposal highlights third potential strategy for AI companies to create a moat around their technology through hardware restrictions. By controlling access to the underlying hardware necessary to run advanced AI models, companies like OpenAI could limit competition and exert significant influence over the AI landscape.

The consequences

In terms of successful moats it doesn’t get better than controlling access to infrastructure. This is analogous to restricting access to bricks for building factories. While the proposal does have several benefits, like protecting sensitive data and inspiring greater trust among stakeholders, it also leaves a wide open space for negative externalities.

Advanced AI infrastructure and cryptographic GPUs are expensive. As such, they would be difficult to obtain by smaller and less developed countries or companies. Consequently, it will have a detrimental effect on the digital divide and well functioning markets. A company using trusted computing hardware would get security certifications far easier, a crucial component should regulatory compliance become a critical requirement. It’s an effective way to keep competition at bay.

Limited access to cutting-edge technology can also lead to a talent drain, forcing skilled labour to countries with better resources and opportunities. Companies that control key components, such as GPUs and model weights, would become more attractive to investors and syphon limited funds away from its competitors, withering the market and stifling innovation.

Because of the distorting effects such a decision would have on everyone outside of the moat, the glaring question becomes: Who gets to decide?

Here’s just a limited set of issues that arise from this path:

• Who attests the GPUs?

• Who gets to access the infrastructure?

• Who controls these systems?

• Who removes the certification and under what circumstances?

• Who decides the penalties?

• Who enforces them?

• Are there any exceptions and who defines them?

• Which models get included in the limitations?

The list of concerns is extensive, and all the crucial questions remain unanswered. The concentration of power and the potential for distortionary effects on the wider ecosystem demand further and careful considerations.

We hope this blog has provided some insight into the evolving dynamics in the AI industry.

If you like what you just read, keep following us for more on our social channels:

X

Linkedin

or get in touch via:

[email protected]

---

No man is an island, but it’s great to be one if you’re an AI company was originally published in Datafund on Medium, where people are continuing the conversation by highlighting and responding to this story.

---

First published on Medium on 2024-05-30.