Pretend a Mouse: VR-GUI in Godot

How I cheated my way to VR-friendly UI in Godot by raycasting into a mesh, projecting into a viewport -- and then faking mouse events.

TL;DR

For the Godot VR Toolkit I wanted proper GUI interaction in VR without rewriting or reinventing the wheel. So, I kept using normal Godot Control nodes and made them think they were being used with a mouse:

Result: 3D VR hands, laser pointer, hovering, clicking, dragging – but under the hood it’s still just mouse events and a viewport.

The implementation can be found on Github.

The 2D vs. 3D UI Problem

Godot is fundamentally stable and user-friendly at 2D GUIs: Control hierarchies, themes, focus handling, signals – all the usual desktop UI goodies.

VR, on the other hand, lives entirely in 3D: controllers/hand meshes, raycasts, spatial interactions and depth perception. What you don’t have is a mouse pointer.

Traditionally, user interface interactions in VR are handled by casting a ray from the controller (like a “laser pointer”) or by directly pinching, hence touching objects with a virtual hand/finger. Naively, you could:

  1. Implement a new VR-specific widget set (buttons, sliders, etc.),
  2. Or try to hack Controls to understand “rays” and “fingers”.

Both approaches quickly turn into a rabbit hole of “oh right, I also need hover, drag, scroll, focus, keyboard…“.

So instead I went for a “cheaper” trick that reuses Godot’s existing 2D GUI system as much as possible;:

What if we keep the entire 2D GUI stack as-is and just pretend we’re a mouse?

That leads to a very simple mental model:

Everything in between is just coordinate transformations and synthetic input events.

Overview of the Setup

The three main scripts in play are:

In a slightly simplified picture:

flowchart LR
  H[VR Hand / Controller]
  H --> R[Raycast]
  H --> G[Finger Collider]
  R -->|collide| M[GUI Mesh]
  G -->|collide| M[GUI Mesh]
  M -->|interpret collision point| M
  M -->|pass faked mousevent| C[Viewport]
  C --> E[Godot GUI system]

The resulting faked mouse events are then processed by Godot’s normal GUI system, triggering hover states, button presses, etc..

ViewportToMesh - putting UI into the world

The starting point is trivial – rendering a UI to a texture and show it on a mesh:

As a node-hierarchy this might look like:

Spatial (World)
    ├── ViewportToMesh (MeshInstance)
    │    ├── Area
    │    │    └── CollisionShape
    │    └── Viewport (instanced PackedScene)
    ...

Conceptually, when a ray hits the mesh, we get:

  1. The hit position in local coordinates
  2. The UV coordinate of that point on the mesh
  3. The corresponding pixel in the viewport

Ray-based Interaction – the VR laser pointer

The ray-based interaction lives in GuiInteraction.gd; a ray originates from the VR controller (or camera), which – every frame – it casts into the world. This gives a classic laser-pointer style UI:

The interesting bit is not the raycast itself (that’s standard Godot), but what happens in _send_mouse_motion and _send_mouse_button – that’s where we fake mouse events.

Finger-based Interaction – poking buttons

GuiFinger.gd is the second interaction mode: direct hand interaction.

Instead of a laser pointer, we have:

The flow is essentially:

  1. The finger collider overlaps with the GUI mesh or a dedicated interaction volume.
  2. From the collision, we again get:
    • Contact point on the mesh,
    • UV coordinate at that point.
  3. From UV we compute viewport coordinates.
  4. From those coordinates we synthesize:
    • Hover events while finger is near the surface,
    • A “click” when the finger crosses some threshold (e.g. penetration depth / velocity / pinch gesture).

Where the ray pointer feels a bit like a laser pointer/remote, finger interaction feels like actually poking the UI – but internally, both boil down to the same InputEventMouse-machinery.

Faking Mouse Events

Now the fun part: lying to Godot’s GUI.

The goal: from the GUI’s perspective, nothing special is happening. It just sees mouse movement and clicks.

1. Function overview

func ray_interaction_input(position3D: Vector3, event_type, device_id, pressed=null):

This function turns a 3D hit position on the GUI mesh into a 2D mouse-like input event for a viewport.

The rest of the function:

  1. Converts coordinates into viewport space.
  2. Builds the appropriate mouse event.
  3. Sends it into the viewport so regular Godot Control nodes react to it.

2. World -> local -> 2D panel coordinates

	position3D = area.global_transform.affine_inverse() * position3D
	var position2D = Vector2(position3D.x, position3D.z)

At this point, position2D still uses the panel’s own local units, centered on its origin.

3. Local panel coordinates -> normalized UV -> viewport pixels

Centered local range -> 0-based quad range

	position2D.x += quad_mesh_size.x / 2
	position2D.y += quad_mesh_size.y / 2

Quad range -> normalized 0..1

	position2D.x = position2D.x / quad_mesh_size.x
	position2D.y = position2D.y / quad_mesh_size.y

So position2D is now in the range (0..1, 0..1) and basically matches the UVs of the quad.

Normalized UV -> viewport pixel coordinates

	position2D.x = position2D.x * viewport.size.x
	position2D.y = position2D.y * viewport.size.y

This is exactly the coordinate system Godot’s GUI expects for mouse events.

4. Creating and configuring the mouse event

	var event = event_type.new()

Here we instantiate whichever mouse event type was passed in (InputEventMouseMotion or InputEventMouseButton).

Mouse motion

	if event is InputEventMouseMotion:
		if last_pos2D == null:
			event.relative = Vector2(0, 0)
		else:
			event.relative = position2D - last_pos2D

Mouse button

	elif event is InputEventMouseButton:
		event.button_index = 1
		event.pressed = pressed

5. Final bookkeeping and sending the event

	last_pos2D = position2D
	event.position = position2D
	event.global_position = position2D

	event.device = device_id
	viewport.input(event)

From this point on, Godot treats it like a normal mouse event:

The nice part is that dragging a slider via a VR ray is literally the same code path as dragging it with a real mouse.

Edge Cases, Trade-offs and Thoughts

This approach works surprisingly well, but it’s not magic. A few gotchas and design decisions:

1. Coordinate precision & panel curvature

If you bend or distort your mesh (curved UI panels), you have to ensure:

ViewportToMesh acts as the contract: its job is to make sure “UV -> viewport pixel” stays consistent.

2. Focus and keyboard input

Mouse is only half the story:

The nice thing: you don’t have to change how LineEdit or TextEdit work, you just feed them events.

3. Multiple pointers / hands

In VR you might have:

The mouse model assumes a single pointer. In practice, I make a conscious decision:

For more complex setups you could emulate multiple mice by tagging events, but that quickly diverges from Godot’s standard assumptions.

4. When not to fake mouse events

Not every VR interaction needs to pretend to be a mouse:

Closing Thoughts

Start from the existing machinery, then adapt the edges.

By bending the inputs to look like mouse events, I can:

The triad of ViewportToMesh.gd, GuiInteraction.gd, and GuiFinger.gd is essentially just:

From the user’s perspective it feels like “of course I can point at that button and press it”.
From Godot’s perspective, they just moved a mouse and clicked.

And that’s exactly the kind of cheating I enjoy.

Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • Engaging the local community: WIMBY workshops in Styria
  • Start der BioPV-Labs: Gemeinsam die Zukunft gestalten! | BioPV
  • Extending Django Knox by secure refresh-tokens
  • Istanbul: the bridge between Europe and Asia
  • Vietnam: my first culture shock (positive)