In the past few years, artificial intelligence (AI) has taken the technology world by storm in a way few innovations before it ever have. What started out as an entertaining distraction for consumer masses in the first few weeks of ChatGPT quickly morphed into a transformational power, with applications across every known industry and business. With the ability to help businesses drive efficiency and innovation through automation, streamlining complexity and targeted analysis (amongst many other things), AI has quickly become table stakes for any business looking to maintain relevance and a competitive edge in the modern era.

In this multi-part series, we will explore practical strategies for adopting AI into your business, starting with how customer data interacts with AI applications and how companies can mitigate the legal and regulatory risks that arise.

Risks and considerations

With this tidal wave of powerful new technological advancement comes a myriad of risks and complexities that can overwhelm and stagnate progress for those businesses that are unprepared to meet these challenges in a practical way. For any business seeking to incorporate AI into its operations and/or its product or service offerings, the key risks or issues to consider include:

• Potential impact to privacy/personal information;
• Data that is being used with any AI application;
• The legal and regulatory regimes that govern the use of AI applications; and
• Commercial implications of AI—including impacts on contractual relationships with customers and vendors.

AI laws and regulations

The European Union is the only current jurisdiction that has enacted any regulation of AI. Canada has tabled a proposed act to regulate AI, and some other jurisdictions are also considering regulations. While keeping an eye on the development of regulations is vital, the practical approach for any business seeking to manage AI risk is to develop a risk framework for its AI applications.

Many organizations likely already utilize such a framework as part of their security protocols. Layering in AI applications into that framework is a prudent and efficient approach. However, for those looking for a more targeted AI framework, several have been developed. ISO/IEC 42001 standard and the AI Risk Management Framework by the National Institute of Science and Technology (NIST) both set out principles to manage AI-related risk. Both these organizations are widely known to most businesses, as they are widely relied upon in establishing security and privacy frameworks.

Any company seeking to take a targeted approach to managing AI risk should start with one of these two frameworks.

Issues arising from the commercial use of AI applications

If a framework represents the holistic, meta approach to managing AI risk conceptually, what are the real-world practical issues that need to be addressed at the commercial frontlines of any business? There are currently several primary issues that our clients experience:

1. Customers seeking to understand how an AI application uses customer data and what data the AI application is trained on.
2. Data vendors seeking to restrict how a company purchasing its data is used with its AI applications.
3. Buyers in M&A transactions seeking representations and warranties around how a seller approaches privacy and personal information with respect to its AI applications.
4. Companies creating AI applications seeking to understand how legal or regulatory barriers will impact their product development.

In Part 1 of this practical framework, we will conduct an in-depth examination of the first issue.

Customers seeking to understand how an AI application uses customer data and what data the AI application is trained on

A great way to approach this issue is by way of a real-world hypothetical.

One of our technology clients has a new AI application that it has rolled out to customers. The application is free. Our client includes it as a part of a bundle in a packaged offering.

In this case, our client’s AI application helps their customers draft presentations and scripts for their customers’ investors. The common questions their customers have are:

• What data is the AI application using in order to draft the scripts? 
• Is their data used to help train the AI application? and by extension;
• Is their data therefore being used to help draft other customers scripts and presentations?

Answering these questions requires an initial understanding of how an AI model is trained.

Training an AI model requires an algorithm that is fed a large amount of relevant data to create an output result. That output result is examined for accuracy and the data and algorithm are adjusted accordingly to continue to optimize the accuracy and effectiveness of the AI model’s output.

The most common commercial AI models currently used include ChatGPT’s GPT line of models, Anthropic’s Claude and Google’s Gemini and Notebook, amongst several others. 

These models are trained on vast amounts of public data, as well as other data that users have provided to the models.

Because of the amount of work, level of technical expertise involved and vast amounts of data required in creating an AI model, most companies typically license a commercially available AI model from the aforementioned companies rather than build their own. 

If that’s the case, how do we reconcile this with the questions our client’s customers have regarding their data and the client’s AI application?

First, our client licenses a commercially available AI model. That’s a model that has already been trained on large amounts of data, as previously discussed.

But in order to help their customer generate relevant presentations and scripts, they need the customer to provide information.

So how is this done? 

It’s done by requesting information from the customer through a series of relevant questions that the customer responds to. These are known as Inputs. For example, if a customer wants to utilize the AI application to generate a presentation and script about its annual financial results, the relevant questions may be:

• What was your revenue for the last fiscal year?
• How much profit did you generate in the last fiscal year?
• What were your operating expenses compared to last year?

The answers to these questions would be the Inputs.

These Inputs are then fed into the AI model the client has licensed and the presentations and scripts are generated from there by leveraging the reams of publicly available data related to financial results presentations and scripts. These generated responses by the AI model are known as Outputs.

So what happens to the Inputs? Do they now become part of the data set used to train the AI model, which is the big concern of most customers?

No, Inputs are designed to be permanently deleted once they have been used to generate Outputs. They are not used for any future training of the AI model, nor are they used for generating Outputs for any other client customers, because each customer has its own instance of the AI model siloed off from the other customers.

Now that we know how data is handled via the Inputs and Outputs, how are legal concerns overlaid and addressed in this type of scenario?

In these scenarios, where customer data is being used for Inputs into the AI model to generate Outputs, the primary concerns will be three factors:

1. Ensuring you have the right to use the customer’s data for Inputs;
2. If the customer’s data include personally identifiable information (PII), that any required consents to process such PII have been obtained; and, as noted above
3. The customer data will not be used to train the AI model and will not be made available to any other customers.

In the next part of our series, we will examine the considerations businesses should be aware of when data vendors seek to restrict how purchased data can be used with AI applications.

For more information on this topic, please reach out to the author, Kenneth Szeto.