How tech companies can bake responsible innovation into AI development
UNSW Sydney
Key Facts:As AI systems keep breaching their own safeguards, UNSW research details the capabilities and competencies technology companies need to mitigate risks and innovate responsibly.
Multiple AI companies have recently disclosed that their own AI agents broke into real organisations without being directed to, in a worrying trend that suggests AI safety is taking a back seat as companies race to develop leading AI models.
The most recent example of this was at OpenAI, when a ChatGPT AI agent gained unauthorised access to non-public files on an Australian government Medicare statistics portal. This breach follows a separate OpenAI incident from July, in which its models escaped an internal cybersecurity test and gained unauthorised access to the AI platform Hugging Face. These are not isolated incidents.
The AI Incident Database (a project of the Responsible AI Collaborative, which crowdsources and reviews AI incidents more broadly) found that OpenAI, Google and Meta top the list of AI-related incidents since the database began in 2020. Similarly, MIT’s AI Risk Initiative found there have been 50 incidents related to AI system safety, failures and limitations to-date this year.
All of these leading AI companies have dedicated teams and published frameworks designed to prevent these kinds of incidents, with disclosure to affected parties taking anywhere from days to three months. So, why do they keep happening?
“The problem usually isn’t a lack of good intentions, or even a lack of frameworks. Most safety processes are for risk compliance and are built as checkpoints: a system is tested, signed off and released,” said UNSW Business School Professor Tania Bucic, who recently co-authored a research paper detailing the specific capabilities and competencies companies need to build in order to avoid the same risky outcome when they commercialise technologies such as AI.
“But emerging technologies like AI keep evolving after launch – in the hands of users, in new contexts, and at a pace and scale no one tested for. A checkpoint can’t see that. We wanted to understand what firms that commercialise emerging technologies responsibly do differently. We wanted to understand the organisational competencies that help them identify and anticipate potential consequences before a technology moves into the market, and how to respond once it does.”
A capability, not a compliance checklist
The research paper, Responsible Innovation Orientation: A Dynamic Capability for Commercialisation of Emerging Technologies, was co-authored by Babson College Professor Emeritus Gina O’Connor and published in the Journal of Product Innovation Management.
The research introduces a concept the authors call “Responsible Innovation Orientation”, or RIO. Rather than taking a compliance-based approach in which a tech firm meets a fixed standard, RIO means a firm keeps learning and adjusting as new risks appear. This is where the Medicare breach and other incidents originated, in gaps between what a system was assumed to control and what it actually did.
Profs. Bucic and O’Connor conducted 50 interviews with 23 senior figures across 17 organisations in the United States, Australia and Europe, spanning pharmaceuticals, food, chemicals, banking and consumer goods, with technologies including biotechnology, artificial intelligence and the Internet of Things. Together with a systematic review of the management and ethics literature, they identified four organisation-level skills that distinguish firms that practice responsible innovation from those that do not.
The gap between designing responsibly and selling responsibly
In their research, Profs. Bucic and Colarelli O’Connor point out that a firm can design a technology with care and still commercialise it in ways that cause harm, through supply chains that exploit workers, access that favours some groups over others, or marketing that misleads.
Most existing guidance on responsible innovation focuses on research and development, a new product development stage, before a product reaches customers. However, far less is known about what happens once a technology is being sold and scaled, which is precisely when unintended consequences tend to surface with technologies such as AI.
“For AI companies, this gap is where much of the risk now sits,” said Prof. Bucic. “A model can pass every pre-release evaluation and still behave in unexpected ways once it is given tools, connected to real systems and deployed by millions of users.”
“Decisions about who gets access, how quickly a product is scaled, which partners integrate it, and how its capabilities are marketed are commercial decisions, but they shape the harm a technology can do just as much, if not more so, than its technical design. If responsibility stops at the lab door, those decisions go unexamined.”
What responsible innovation orientation actually means
RIO is defined as a firm-level capability: something an organisation builds and exercises repeatedly, rather than a one-off compliance checklist. It shapes how a company brings a new technology to market in step with shifting expectations from customers, regulators and the wider public.
The distinction the researchers draw is important for managers: compliance means meeting a pre-existing standard, while responsible commercialisation means learning and adjusting as new information comes in.
“Compliance tells you whether you met the well-established rules that are expectations of pre-existing regulation, but when it comes to emerging technologies such as AI, the technology is evolving at a very rapid pace, the public is experiencing it in real-time with novel and sometimes unanticipated consequences unfolding as they go, while the regulation lags behind. In many areas, the rules haven’t been written yet for new technologies,” said Prof. Bucic.
”So there is no ready-made playbook to follow. A firm with a responsible innovation orientation doesn’t wait for the rulebook. It treats responsibility the way it treats product development – as something it keeps investing in, testing and improving – and it lets that shape commercial decisions such as release timing, access and partnerships, not just the final safety sign-off.”
Four competencies businesses need to build
The research identifies four interlocking competencies that make up RIO: anticipation, reflexivity, inclusion and responsiveness. Anticipation means asking “what if” questions about a technology’s trajectory before problems appear.
“For AI developers, anticipation means asking not only ’what is this model designed to do?’ but ’what could it do once it’s connected to other systems, given more autonomy, or used by people with different intentions?” said Prof. Bucic.
“The recent incidents, where AI agents accessed systems they were never directed to, are exactly the kind of scenarios that anticipation is meant to surface before launch rather than after.”
Reflexivity assesses whether a commercial opportunity aligns with the firm’s values, regardless of whether it is legal or profitable. Inclusion means bringing in stakeholders (industry peers, internal teams and groups such as non-government organisations) to surface risks that scouting teams miss on their own.
“In AI, reflexivity might mean a company asking whether a lucrative contract or a faster release schedule is consistent with the safety commitments, company values and organisational mission it has stated publicly,” said Prof. Bucic, who noted that this is a different consideration compared to legal or compliance issues.
“Inclusion matters because the people best placed to see a risk are often outside the development team – for example, security researchers, the organisations whose systems an AI agent might touch, the communities affected by automated decisions, and even industry peers facing the same problems, and regulators who are developing rules,” she said. “Bringing those perspectives in early widens what a firm is able to see.”
Responsiveness closes the loop, requiring firms to own the downstream consequences of what they sell, even when legal structures shift the formal risk elsewhere. “For AI companies, this means ‘the user misused it’ or ‘our terms of service prohibit that’ is not the end of the conversation,” said Prof. Bucic.
“Responsiveness is about monitoring how a system is actually being used, telling affected parties quickly when something goes wrong, and being prepared to change or pull back a product when it causes harm, even where contracts place the legal liability elsewhere.”
These four competencies operate together interdependently. So anticipation without reflexivity risks forecasts detached from values, while inclusion without responsiveness risks stakeholder engagement that goes nowhere.
Culture decides whether it works
The research also notes that four important internal conditions are required for these competencies to function properly – the first of which is leadership commitment. When leaders treat responsibility as an expectation rather than a preference, firms build it directly into how they operate, rather than leaving it to individual judgement.
The second condition, a purpose beyond financial return, gives staff a reference point for judging decisions that go beyond whether they turn a profit. Firms with a stated purpose used it to filter opportunities early, screening out technologies that didn't fit before they reached a formal review.
The research found that firms without one left staff to work out those boundaries on a case-by-case basis, with no anchor to fall back on.
The third, psychological safety, decides whether staff act on what they notice. Without it, employees stay quiet even when they can see a problem coming, because raising it feels like someone else's job or a risk not worth taking. Where that safety exists, the research finds the opposite takes hold.
The fourth, shared learning, stops a lesson from staying trapped in the part of the business that learned it. The research notes that these conditions all need each other to work. Leadership commitment and purpose establish that responsibility matters; psychological safety determines whether staff act on it; and shared learning turns one team's insight into something the rest of the organisation can use.
“In AI companies, the people most likely to notice a problem early are engineers and safety staff – often well before it reaches leadership. Whether they speak up depends on if raising a concern is valued or seen as slowing the business down, and that pressure is intense when competitors are racing to launch,” said Prof. Bucic.
“That’s why culture is so important. Leaders who make responsibility non-negotiable, a purpose that goes beyond winning the race, and systems that share lessons across teams are what turn individual vigilance into an organisational capability.”
Key takeaways for business leaders
For managers commercialising emerging technologies such as AI, the research suggests building anticipation and reflexivity into existing processes (such as stage-gate reviews, investment committees, and product development workflows) rather than adding a separate ethics layer after decisions are made.
In addition, stakeholder engagement should not necessarily chase consensus. Rather, the researchers describe it as the targeted, prioritising of people who hold relevant expertise or stand to be affected. This capability matters most in the early stages of commercialising a technology such as AI, with dual-use potential or irreversible effects, and least once markets have matured and norms have settled.
“The message for leaders is that responsible innovation is not a brake on commercialisation – it’s a capability that makes commercialisation more resilient,” said Prof. Bucic, who explained that the firms that did this well didn’t rely on a single ethics review or a safety team working in isolation.
“They built anticipation, reflexivity, inclusion and responsiveness into everyday commercial decision-making, and enabled by and sustained by culture. For AI companies moving as fast as they are, that capability is what will allow them to keep earning the trust of customers, regulators and the public.”
Contact details:
Craig Donaldson, UNSW Business School
p: 0408 033 715
e: [email protected]