Panel: Governance & Ethics

I've got all of my questions written down, but I I I did something before this, and I'm curious to see if it worked or not, which is I've got my claw been watching this stream, and I've got it coming up with some questions for the panel. So we'll see what its questions are like. I'm gonna ask some of my own first, and then I'm gonna give my open claw its opportunity to shine.

My open claw is called Marvin for obvious reasons for those of a hitchhiker's bent. So we'll see what Marvin has to say. And then I've got some questions that other people asked. If we have time at the end, then we'll throw to the audience for a couple maybe. But we are a bit tight on time. So thank you everybody.

There were some amazing talks. We've covered the topics of governance and ethics and privacy, which I think is not a conversation that we can say too much about in the current way the industry is heading right now. So thank you all for doing that. My questions, I'm gonna kind of bounce around a little bit. I'm gonna ask some direct questions specific people, but if you have something to say, just just chip in.

This is this is a little bit of an informal conversation. So my first question, I guess, is around how we're thinking about the ethics of how these AI models are getting used and how they're trained and thought about. One thing that has been a conversation for the past three or four years is that the people who are most likely to be targeted by misuse of these models are also the people who are least likely to be in the room making the decisions on governance and ethics and privacy.

So I'm thinking women, I'm thinking of a marginalized groups. What are the things we can do as an industry? This is a room full of people who have come to a leadership track. So I'm expecting a lot of people in this room to be the people who are making a lot of those decisions. What can the people take back to their organizations, to their communities, and actually start doing to make sure those underutilized viewpoints are put forward?

So does anybody want to just kick off on that?

I mean, I'll take it because that was my job for ten years. Again, I took Atlassian. By the way, when they appeared, they were 6% women in engineering, which I hope everyone feels physical shame about that. We got them to 17%, which was industry leading at the time. When I left Culture Amp, Culture Amp was 32% female engineers.

So don't tell me it's impossible. I know how to do it. But But I think one of the big things that I would say is things are moving really fast. And so I would actually put a bit of responsibility on folks that have technical skills to open a door and pull up a seat. And what I mean by that is that ethics and governance is actually more female, more queer, and more brown than the technical side of it. I don't have hard stats on that, but vibes, right? We know that's true.

And also, ethicists tend to be seen as less credible or given intellectual respect compared to someone who's a coder. And so what I would say for anyone who is on the technical side of the house is actually creating space in the room for those people and intentionally watching the way that they're treated in dialogue. So some classic kind of techniques for that is one, tech folks bringing them in early.

Literally opening the room to the door is really powerful. But then also, when you see people either kind of contradicting or moving on, bringing the discussion back to the ethical point can be really powerful. I often teach that for men in allyship for women, but then also making sure that idea attribution happens. So cultures develop intellectual respect by whose ideas get adopted.

And a common pattern that's often really unconscious on the part of folks from majority groups is that if I, as the woman in the room, am saying x, the room kind of moves on. And then a dude says the same thing in slightly different language five minutes later. Everyone's like, yeah, John. Amazing. And the solution to that is not to shit on John.

The solution is to say, hey, John. I'm so glad you agree with Aubrey. It sounds like we're moving in that direction. So it's not about shaming anybody. It's about acknowledging and kind of noticing patterns of who gets in the room and then whose ideas get to have influence. And I think if you do have those technical skills, use that power for good, both because it's the right thing to do.

But also it means that you as a group are going to settle on better ideas. So again, it's one of those win win things. But if you're not subject to those kinds of patterns, you probably don't notice them. And so I kind of bring that forward to say, this is an option or this is something that usually happens.

But I would also say for those folks, I get that your lived experience doesn't give you this knowledge. Think about your social media feeds and how you can kind of construct your algorithms to make those patterns and issues more visible to you so that you can come up with your own ways to solve them. So I actually think tech can be really helpful in widening the aperture of figuring out how you can be useful without expecting you to do everything all the time.

Thank you. I guess a follow on question or a follow on point of view. There's a lot of discussion. So I work for a AI health tech company and we use models all the time. And it's fascinating to me to see different levels of bias that actually go into the models themselves.

So we record clinical sessions. You can imagine some of those clinical sessions have quite sensitive discussions. But also, a lot of our clients use them for things like discussions of gender affirming care. And when we do our evals on the different models, you will be unsurprised to know that one particular model from one particular provider seems to just stop producing output tokens when gender affirming care is discussed. And another model from another provider seems to stop producing output tokens when Beijing in June 1989 is discussed.

How can we kind of think about the way these models are used? And is this to the point of do we need to build an Australian model for Australian sensibilities? Is it not just data sovereignty? Do we need model sovereignty in Australia?

If I was looking at me. Look, I think that two years ago, there was a no, it's too expensive time, and there was probably a reasonable argument for that. I think that people have proven since then that it's not as expensive as it used to be. I don't think that a single Australian model is a particularly desirable thing, like something that's dictated from the top that this is the Australian government model and should be used for government business or something.

But I do think that more Australian models themselves would be great and as many of them as possible. Diversity is the strength. You know, to go to the previous question, one of the things that I've used as an argument where people are arguing against that is, you know, in traditional machine learning, you have ensemble approaches that always outperform single approaches because of the diversity of the input.

And, you know, you get the most nerdy person and they have trouble arguing against that. So, like, diverse models, including Australian models would be fantastic. I think that we're not thinking through some of our the structures of ways we're thinking about intellectual property in Australia.

I think we haven't thought through the potential of restricting training and losing Australian culture because of that completely. And I'm not arguing for free slather, but I also think that we don't want to lose Australian culture to because of the lack of Australian data sources.

And that's a point that has been lost in the debate, and I think needs to be considered as well.

Sure. It might have been when

you mentioned 1989.

That's my fault. Sorry about that. I'll use this and

not I will do a thing that I probably get in trouble free on livestream. Just in case anyone knows, main code has sloughily opened up the beta sign up for Matilda. So if you are interested in Australian AI, you can put your name on the list and they'll they'll, I don't know, vet you or something. I don't know how it works.

But anyway, the beta's open. It's on their website.

Somebody somewhere will share that link, I'm sure.

Main code's training on AMD as well, so if you want diverse hardware, you know.

Jensen's quaking in his boots.

No. I'm sure he's quaking in his million dollar boots.

You know, Jensen is willing to pay his taxes.

This is fair.

I just have to put put that on him. We we appreciate people who pay taxes.

Has he paid more on taxes or leather jackets? Yeah.

You know, do you think he has more than one?

Oh, yeah.

Like he seems to me like the dude who'd have like one leather jacket for 20 years.

But multiple of one, not just one.

Oh, you think he has multiple copies?

Yeah. Okay. So for people who aren't aware, Jensen is the CEO of NVIDIA, and he always does his presentations in leather jackets. It's actually from foreign caps. Oh. K. Yeah. Someone's gonna get him.

I I saw you shaking your head after making that joke. It genuinely wouldn't have surprised me if that if that was actually true.

It's also I don't know what is my former professor, if he's not a philosopher at GDM doing that also often shows up in a leather jacket. So I I don't know what it is about like AI elite like as a look.

Is is this like early two thousands where everybody wore the black turtleneck to

Maybe or like I don't know, I come from San Francisco so like I have trauma about like oh my god khakis and a Patagonia vest. I mean that means you were giving money not building anything but

Or you get funded.

Oh yeah, you got funded and now you can afford your puffer vest. Buy it in a vending machine at SFO.

How are we doing on the livestream?

More comedy, please.

More comedy. I can ask one of my Marvin questions.

Yes.

Shall I ask one of my Marvin? Yes. Because these are probably gonna be so bad that the livestream livestream people are happy that they've not heard them. So this is from Marvin the Paranoid Android. I'm just gonna read this verbatim. I've not read this before, so if this doesn't even make sense, I apologize. Aubrey argues governance should be culturally specific.

Hamish wants lightweight frameworks that scale. Nick's building privacy tech that works across jurisdictions. Where does local values end and universal engineering discipline begin?

Good question. That's bad. It's not

a bad question.

Look, mean, think good engineering is not is orthogonal to ethics. And I think that good engineering should be an enabler for good ethics. You can have local values wherever you are, and good engineering is the thing that makes them enforced and useful.

Yeah. You need a mechanism to verify that the values and the ethics that you're intending your organization to follow are being followed, I think is the key. Different organizations are going to have different postures on these types of things and you want to make sure that you have frameworks that are lightweight, that people can digest and sort of work with it as rapidly as the technology is available, but allows that, I would say personalization, but I think that's maybe the wrong term, but that that customization to the organizations or nations sort of needs.

What do you think about vibe ethics? Like, so I saw a thing the other day about the governor of Hong Kong during the nineteen sixties and seventies. British appointed colonial, all the all the bad things, but nonetheless saw Hong Kong grow enormously. And one of his things was that he refused to collect economic data because he said, as soon as we start collecting the economic data, that's the thing that will concentrate on on optimising.

And so instead, he walked around and tried to fix problems that he saw. And I certainly have been guilty of that. You hill climb on the thing that you can measure because it's easy to measure rather than, you know, if you started off saying, I wish ethics should be measurable. Can you measure ethics? Like you can measure parts of them, you can measure some outputs, but that's not the ethics themselves, right?

Yeah. And We are back on the livestream, by the Awesome. And apologies for you watching the livestream. I had a completely throwaway question and it ended up turning into a fascinating conversation. I expected it to be a humorous two minutes and it turned into, as you have when you have amazing speakers, a very interesting five minutes.

So apologies for that.

Well, so there's a book that I read back in 2020 talking about the ethical algorithm, but there's a very extensive body of research looking at actually how do you use good engineering practice to operationalize ethics. Because one of the kind of failure modes I see with my clients is they build this beautiful set of RAI principles. And then it sits in confluence and does jack shit in terms of engineering decisions. And I think one of the things that's really essential, and I actually learned this from a guy that I'm in grad school with, talks about this essential nihilism that the engineers must accept in building this technology.

And that bad shit's going to happen with what you do and you can't solve it, but soldier on and do your best. And I think vibe ethics is actually more like morality. So people's morality is more vibes. But to me, ethics is about structured decision making, which is something that engineers are actually really exceptional at. And so in the work you do, there's a level of risk acceptance. And so I think part of it is going ethical risk is a thing. And how do we actually conceptualize it within the context of what tools are?

And so the discipline and the obligation is to move moral vibes into ethical frameworks and then articulating the risks and trade offs that you accept. And I think that's the thing people don't like to do with ethics because they keep it fuzzy because it is kind of this ego feeding exercise of, look, I'm a good person.

And so I actually think one of the most helpful postures to have when you're trying to do ethical engineering is that I cannot resolve all of the potential harm that I might cause in doing this. But I can put in good faith effort and rigor, like what you're doing, to intentionally and own the choices that I'm making.

Like that's kind of the boundary of what you can control. And I offer, again, that engineers actually have 80% of the skills to do this without touching ethics. And so I think there's a lot of intimidation of like, oh, I don't know anything about that. But you actually already have if you thought about risk, you thought about technical issues. You have 80% of the mental frameworks to do it. You just need someone to support you with that 20% of thinking how to think about it in practicality.

One of the things that differential privacy does that I mentioned was that it gives a quantifiable measure of privacy, and that allows you to do these exact decisions around, are we gonna run this report? Are we gonna add this extra attribute to this report? Because you can measure exactly how much privacy you lose from that, which is quite interesting.

So all of this stuff we're talking about, this is a good segue to to my next question. All of this stuff takes time. And, you know, the the model providers, the VCs, the whole industry is saying to us right now, if if you're not running, you're walking backwards. And just go quicker, go quicker, go quicker. What message can we take back to our organizations to encourage them that this is something that is in our organizational interests to go slow and to take the time and consider and to make the right decisions on rather than running so fast that we trip over our own feet into a pile of our own excrement.

I with the premise of the question. I think we're in a K shape. So you have some companies which are aggressively running fast. And for those companies, I think you need to take the time to ensure you have governance and tooling that allows you to rationalize with the risks you are accepting.

I think there's a lot of blind acceptance of risk and I think, yeah, big problem on that sort of upwards trajectory, but they are not running the risk that someone else is going to outcompete them and be fitter for the commercial game. The flip side is the other half of the businesses that I've spoken to which are too timid to even start taking the first steps and they are like really daunted by, you know, but what if it does this and what if it does that?

And we haven't gotten a sort of position or a governance paper on that, and those are the ones that I think actually you need to go and experiment with this and you do need to accept the risk of the unknown. And the world is now about evolving quickly. Like it's get the right people that allow you to make the right decisions very quickly.

It's get the right tools that allow you to see the right information to make the right decisions very quickly.

I totally agree. I would also, I'm rejecting a different premise but I agree with your rejection. Oh, I'm

getting very rejected.

Like there there are more than two speeds. There there like there's not nothing and sprinting. And so one thing I would say is that again, it goes back to my idea of optimizing over longer time horizons, I think is really important. I'm literally announcing to staff on Monday the Ethics Center's AI adoption plan, which is a six month plan that starts with governance but allows sandbox play in a defined thing.

So you get the tools right away, but you have some guardrails about what you're allowed to do before we get the policy in place and all of that. And it's because, again, we're saying, go for a jog forward right now. And in six weeks, you'll get to sprint. And the reason we're doing that one, we're literally an ethics center.

So if we fuck it up, it's really existential. But with the clients that I talk to that are doing responsible transformation, part of it is I have to say, yes, I think in general the economy is K shaped, but you don't have to fall to the ends of the K. And so be aware that speed is a spectrum.

And again, depending on if you're working in health care, the necessary care is much higher than if you're doing calendar scheduling. And not to make light of that, but I think your safest responsible speed is really determined by the use case, the part of the business, and what the business does in a way that, again, this goes back to make your choices deliberately.

I, in general, find that I really don't like the rhetoric around you're going to get left behind. I do think that is true. I think empirically it is true in some cases. But I think it delegitimizes other possible futures that are not drunk on technology. Like I personally like AI. I want to adopt it. I find it useful.

Like I'm very much on like the pro but careful side. But I see legitimacy. And someone's saying, fuck no, I don't want to do that. That's also an acceptable choice. Now, there are consequences and things of that. But one thing I would encourage us in the room is thinking about and what I work on with clients is actually a conscientious objection and opt out kind of principles and policies.

Every organization will have to make their own decision about whether you actually allow yourself to opt out for ethical reasons. And I think Pope Bob coming out and laying some groundwork that I do think there's actually going to be some regulation because there are jurisdictions where religious objection is a thing. And I think that's going to be something that if you're not thinking about it, it might catch you unawares in ways that could be expensive or embarrassing.

So that's it. It's like, I'm pro but I think it's okay if someone says no for whatever reasons makes sense to them.

What if someone wanted to conscientiously object to using a computer?

Yeah.

That at some point the employability of the person Totally.

Oh yeah yeah. No I think that's a great point. So and I'm actually quoting my boss right now because we just had this conversation Monday. So Doctor. Simon Longstaff is like a ethical philosopher. But the thing about conscientious objection is that conscientious objection doesn't accept you from repercussions for the objection. Right? There are potential things. This is actually, we are wrestling with this and what we've said is for experimental phase of AI, you have a right topped out if you want to.

We have determined that at a set point in our adoption plan, we as an organization will develop a point of view on conscientious objection and also on job requirements such that it is a possible like, it's possible in the set of options that for I'm also a former HR lady. I think something that becomes more critical when you are running at AI is to be really clear about what outcomes people are responsible for. Not task lists, but outcomes.

Because the idea, you know, in theory, you could have two conscientious objectors. One can actually hit the objectives without AI. Great. You can stay. Someone else cannot achieve the outcomes required of their role without artificial intelligence but opts out. That would be a conversation about an inability to carry the duties. And there's legislation around reasonable accommodation that already exists in legislation to cover this. So think that's It actually requires the HR and human rigor underneath the systems to get better in order to be able to deal with these in a way that's actually achievable for business.

But yeah, some businesses will say AI are out. And I think the ethical obligation there is to be clear about what your stance is.

In the example you gave there, wouldn't that be a situation where you're changing somebody's job expectations under their feet?

Potentially. So there's labor regulation around this. Like with government clients, if someone's job changes more than 10%, you actually have to go through a labor consultation process. Tech businesses, that generally doesn't apply at that kind of knowledge work that we're talking about. But, yeah, the reality is actually most businesses just fucking suck at having job descriptions on file anyway, and no one knows what objectives they're being measured on.

So, like, you were theoretically correct, but, like, trust me. I've consulted with hundreds of startups. And they're like, I'm like, they wanna they're like, I wanna do a performance process. I'm like, cool. What are your performance criteria? They're like, we need those. It's not just manager vibes. I'm like, yeah. And you're making compensation decisions based on that.

So like, what I say is the reality is messy. And to Hamish's point, you need governance. And governance isn't just about your agents, even though that's really crucial. You actually need to improve your people governance systems so that you can have a better integration of this because they are sociotechnical systems. So anyway, HR is still useful.

Don't lay them aloft.

We can just automate that.

Oh my god. That Bolt dude, just for the record, such an idiot. Like, just because you don't know about the problems doesn't mean they go away, which also applies to agents. Yeah.

So I I have a question from Marvin the paranoid android. Marvin says, How can we 10x our token usage so we can pay Anthropic more money? Just joking. Sorry. His actual question, and I don't know if this is gonna be good or not. I'm literally just reading it verbatim.

Ew. It's for you, Nick. So you mentioned in your presentation that your co founder went to work for Anthropic.

I personally did Piper with.

Okay. Sure. There you go. Hallucination. Do you see that as an indication that these companies are investing in privacy? Do we see the two companies you mentioned, Google and Apple, are really spending a lot of time and money on this, one of them builds their own AI model, the other one doesn't. Are the companies that are building their models investing in this?

Or is this going to be other companies that are kind of spending their time on it?

For Tobin specifically, no, it was not related. He had an MIT PhD and so went to someone that pays more money than God. On the other thing, yes and no. Like, I mean, OpenAI has released models to do PII redaction because, you know, that is one of the primary objections to using PII identification identifying information through their models.

What do you do? Well, solve that problem. In terms of some of the other technologies, I haven't seen a lot from the the Frontier Labs outside of DeepMind at Google to done some work in federated learning. And Meta's done quite a lot of work in it. In both of those cases, they've generally been related to ad tech applications where there is concern around the privacy implications of advertising, astonishingly.

And a lot of that has been sort of imposed on them by regulation. Should I move this out? I don't know. Yeah. So, like, I haven't seen anything from anthropic around that, and OpenAI is limited in what they're doing in that area.

The Chinese labs, of course, are pushing most of their models out as open waves, so you can run them yourselves. The motivations behind that are mixed and who knows how long they'll last. And some of the American ones like Google Gemma models are great if you're usable up to that scale. Like, they don't release their largest models.

But their smaller models are very, very good. And in some cases, like, think there are new architectures possible. I've built a a full data engineering platform that runs in a browser using browser based models. So there's no data that goes to the cloud. Like, that's interesting, but you can do that twenty twenty six and twenty thirty. Who knows how good that will be?

So I don't know if that answered Marvin's question.

I know Marvin, you'll have to send me a Slack DM and tell me if that answered your question. I guess speaking about these providers and the models they build, the providers are starting to lock some of their models behind, I guess, gateways. Mythos is not available for general usage.

What do those types of decisions from the model providers do in terms of the ethics of how these models are used?

I mean, I think the mythos is coming in a couple weeks. Like, glasswing is almost over. I I would also say glasswing was given to other private companies for the most part. And like, I'm much more concerned about people hacking the power grid just like in general. Like, I assume everything I've written down is gonna be on the internet at some point.

But yeah, I think the locking down to me is more like an economic thing. Mythos, I think Anthropic also has a brilliant comps team in that they're realizing that OpenAI and Grok are evil ones. And so they're trying to position themselves. So trust in that is there. I'm not saying they're not sincere. I'm just saying I think they also have economic arguments that point them in that direction.

And I think it's related to model features. And more higher order models are getting locked behind paywalls because they have to figure out how to pay for the data centers. Glad the ethics is in the conversation, but I don't think it's the primary motivator for the most part.

I mean, OpenAI said something very similar when GPT-three came out. They said we're delaying the release of GPT-three because it's so powerful. We're afraid what people can do about it.

Yeah. And I know they've had people on podcasts lately talking about some of their internal models that they're not releasing. Yeah. Look. I mean, I have concerns that this trend of limiting intelligence to approve people, I I think that's problematic. I don't. And I I am aware of the issues around the cyber potential of these things, and I do think it makes some sense, Anthropics approach of, you know, giving people a head start to try and solve some of these problems.

But, you know, really, the problem is the insecure software, not the model that's finding it.

Yeah. I'm I'm hoping I'm hoping that WordPress was one of the companies that got access to it. Probably just used up all the tokens straight away.

It's already hacked. Don't worry, like Yeah. I I I think that the trend of trying to restrict stuff like that is worry about that. I think, you know, in particular in Australia, like, what do we do if we can't get access to these these tools because we're not on the approved list from China and from America?

The open weight model solve this problem like we saw in the in the keynote this morning? Open weight models are trailing three to nine months behind state of the art. I don't know if Mythos is is gonna book that trend, but is that the potential solution to that?

Look, there's a really interesting argument here. Depending on who you talk to about the benchmarking, that gap is increasing or decreasing. If it's decreasing, then it's one message. If it's increasing, it's another message. And it's unclear what the answer is. And it's unclear if it stops at some point because of money.

So look, it's a start. The fact that those open models are available means there's a level of intelligence that's always gonna be available, but we don't know how high the roof is on it. So

Cool. Yeah. I think we got time for one last question before we finish up. So what is the most common mistake that a company will make when rolling out AI into their org? And and I guess the question I'm asking because I'm sat with three people who just talked about governance, ethics and privacy and this is a room full of leaders.

I'm not talking about, you know, what technical mistakes people make with their agents MD. I'm talking about governance and rollout and securing the organizations and the customers data. What's the main mistake to look for?

My

take unsurprisingly is I think for those organizations that are being aggressive with it, there is a high risk that they're being naive to the risks that they are taking. I think there are a plethora of risks from the spectrum of technical all the way through to ethical and they often sort of see that some discrete uses are using like the ISO, you know, forty two point zero point zero point one framework and so they assume that because they've done the ISO framework for that implementation, that extends to all of the other users in the organization and it just doesn't hold true.

They're incentivizing people to experiment and sort of push the boundaries, but they're often not watching what's happening with it.

My spicy but strongly reasoned take is they choose productivity and efficiency as an objective rather than a tactic. So if you choose productivity or efficiency, one, you freak people out that you're gonna try to automate away their jobs. And there is a lot of talk about psychosocial risk litigation to do that. So basically suing the executives for scaring the shit out of the employees. But you also encourage people to make slide decks instead of customer value.

So productivity is output measured, and so you create a lot of slop. And so I'm not saying productivity is bad, but I'm saying with nonprofits, it's often mission impact. Or for profits, it's customer value. But tell people why they might want to be more productive and encourage their critical thinking about where AI is the best tool to do or accomplish something and where it's actually just not needed.

You also don't do the weird like the token maxing. Like, what? Stop. So that's what I would say is you need to teach people the why you're doing it. And it is just much more interesting to talk about innovation or customer value creation because then you involve individuals and trust their experience and their expertise. And it's more engaging to experiment with.

And so I think that sometimes automating something, sometimes making it more efficient is, in fact, the strategic and correct decision. But it is one of a suite of approaches to get to a meaningful outcome. And I don't think make more stuff is actually an outcome that's particularly defensible, whether you're concerned about ethics or your fiduciary duty as a board director?

I have two answers. One is vibe based rollouts where they implement whatever they saw yesterday on Twitter, and that's the thing we have to roll out. And at the other end of the spectrum sorry if anyone here is gonna be offended, but rolling out Microsoft Copilot and thinking you've got AI. It's the cosplay hero of

victimized by Copilot this week.