Pandora's Box: What Happens When You Give Your Users a Terminal in the Metaverse

by Lazy Eye Of Sauron

Metaverses, or, well, let's call them what they really are, walkable chat rooms, surged in popularity during the China COVID-19 pandemic.

VRChat, Roblox, hell, even Second Life saw growth during the pandemic.  Additionally, we saw other companies show interest in creating metaverses of their own, with varying degrees of success.  Zuckerberg dove head first into the shallow end and bonked his head, rebranding into Meta, and creating Meta Horizon Worlds, for example.

This article is for those wanting to jump on the metaverse bandwagon, so you know what you're getting into regarding securing your metaverse.

Maybe the biggest hurdle you are going to come across is users creating third-party content.  A few metaverses allow this, and it is a cool feature.  Your users get to create custom avatars, worlds, assets, and ensure that you have a unique and varied world with infinite customizability.  This is a double-edged sword, however.

In a world where the worst thing that will happen to you is your account being banned, you can bet someone is going to make some annoying griefing tool instead of something useful.  If you have ever been in a VRChat public lobby and been hit with someone trying to force you to have a seizure, or just logged into Second Life and been greeted by a physics crasher, you know what I'm talking about.  Preventing this is effectively impossible.

Your moderation team is basically on the ropes, in a reactionary role, and the best thing you can do is keep up to date on what your users are making.  You can attempt to blanket ban the object, but it's impossible to prevent a new one from being created, with the barrier for entry being lowered every time ChatGPT gets an update and a new DAN variant is created.  The best way to handle this is to make sure your team can not only identify these objects on sight, but also know what they look like on a code level.  If your metaverse allows for the creation of user-created content, your team needs to be able to create their own, or at least be familiar with the programming languages used to create objects in your world.

For example: Say you need to test an object.  You load it into your test environment, but it doesn't run.  It's coded in such a way that it will not run outside of specific circumstances.  If you have no idea how to modify the object to make it run and understand the conditions it requires, you can't effectively make a case that it is malicious.  This is part of the reason why VRChat asks for unity experience for their trust and safety team.

Now, of course, you can just take the code you see, drop it into the AI of your choice, and have it explain it to you, but even now it's not consistent with its results.  AI still lacks the instinct and creativity required to look at an object, its code, and think about potential opportunities for misuse; Which leads me to my next point...

I know that AI is tempting to use as a replacement for humans in a moderation team.  It's cheap, you don't need to pay for therapy because it got exposed to nightmare fuel for the fifth time this week (you'll pay for that later when it starts monologuing about hate, or takes over an abandoned Peugeot factory or something), it can work 24/7/365 with no breaks, and I am here to tell you to resist the siren call of the glorified Markov chain that it really is.

I'm not saying that it doesn't have its uses, but it will not do the job for you, and even present security risks of its own.  After all, how else is it going to get trained, not using the data that you are feeding it?  You need humans to be at the helm at every step in the process, humans who know what they're doing, know what they're looking at, know what can be safely be input into the AI, and can tell when the AI they have to help them is smelling toast.

One more thing you should be on the lookout for when attempting to protect your shiny new metaverse is anything involving voice.  Text chat is easy to look at.  But voice, well, not as easy to review.  Now, of course, someone using hate speech in voice, or being a general nuisance, is important and you should know how to deal with that, but it's not exactly what I'm hinting at here.  In some metaverses, it is a common tactic to use voice as an attack vector.

For example, using it to force a crash.  Voice is an essential function for realism and immersion, but is a can of worms in and of itself, one that again you can only really react to unless you want to annoy your userbase by forcing delay so a program can check what you're saying.

These are all just baseline things you should think about before creating something new.  Look at the problems that older metaverses deal with, and still have.  Know that you being new means that hackers are going to come and try to break your world over their knee, and try to avoid mistakes that older metaverses and online communities had in the past (Looking at you, Meta...  If you know, you know.)

Metaverse require very different teams to protect them than standard chat rooms or forums.  They need people who can think offensively, like the people they are protecting your users from.  They need to be able to act proactively, and have time to hone their skills and research.

In a sense, your new metaverse is a dungeon, your team is the dungeon master, and all these new hackers and trolls and degenerates are your neverending stream of adventurers, eager to cause all sorts of chaos.  Your team needs to be the masters of their domain.  You should find those who can think like hackers, think like trolls, think outside the box and find the weaknesses you know full well you overlooked or at least didn't have the budget to fix (because we all know this was way more expensive than you thought it was going to be), in addition to having a diverse and empathetic team, and one that can handle context and gray areas (rules lawyers are their own special breed of hacker).

If you would like to talk more about metaverse security, policy, or perhaps even recruit me for your team, I can be found on Bluesky (@lazy-eye-of-sauron.bsky.social) and on X (@SauronLazy).