face_mouse: a hands-free mouse for people who can't use their hands

The business problem

Think of a man with no hands who wants to use a computer on his own. Mice, trackpads and keyboards all assume hands, and so do most "accessible" setups: someone still has to start the app, press a hotkey or fix things when they go wrong.

Specialist options exist, such as eye-tracker hardware and head-mounted pointers. But they cost money, need extra devices and usually need someone to set them up.

The goal: use what every laptop already has, a webcam, so a person can move the cursor with their head and click with their eyes, with no extra hardware, no internet and nothing to hold.

Why existing solutions failed

• Hardware trackers work well but cost a lot, and they're one more device to mount, charge and calibrate.
• Simple "nose = cursor" demos map head position straight to screen position. Reaching the screen edges then takes extreme neck angles, and nobody can hold their head perfectly still, so the cursor never settles.
• Blink-to-click sounds natural but fails in practice. People blink involuntarily 15–20 times a minute, so a naive detector clicks at random while you read.
• Most tools assume one kind of face. Blink strength and wink ability vary a lot between people, and many people can't wink one eye at all. Fixed thresholds work for the developer and nobody else.

The solution

face_mouse is a Windows app that turns a normal webcam into a hands-free mouse. It runs entirely on the PC.

1. Head movement steers the cursor, by speed rather than position

The cursor uses velocity control: head angle sets how fast the cursor moves, not where it is. Tilt slightly to creep, tilt more to cross the screen, and return to centre to stop. A deadzone keeps it still while you hold your head steady.

• Two signals, blended: head rotation (steady, ignores you shifting in your seat) and nose position (better for small corrections). They're measured in eye-widths, so moving closer to or further from the camera doesn't change the speed.
• A One Euro filter smooths heavily when the head is still and opens up when it moves. Beta 6 was chosen from measured trade-offs: 6× less jitter at rest, with low lag when moving.
• A 120 Hz cursor loop, separate from the 30 fps camera, with sub-pixel accumulation. Without it, slow movements round down to zero and fine positioning is impossible.
• Auto-recenter slowly follows the user's resting pose, so slouching over a long session doesn't cause drift.

2. Winks click, and blinks don't

This is the hard part: telling a deliberate wink from a natural blink. Three checks do it.

1. Co-occurrence veto. A natural blink closes both eyes. If the other eye also closes past the veto level while one is shut, it's a blink and nothing fires.
2. Duration gate. Natural blinks last about 100–150 ms. Anything shorter than the first tier is ignored.
3. Refractory period. Only one action per 300 ms, so a flickering eyelid can't fire a burst of clicks.

The chart above is the real detector running on a synthetic eyelid stream. It rejects the blink, then gives a left click, a right click and a double click on the longer wink.

Winking also tugs the face, which would drag the cursor off target at the moment of clicking. So when an eye closes, the cursor freezes and rewinds 100 ms, and the click lands where the user was actually pointing.

3. A small gesture vocabulary, learned by watching

How long a wink is held decides what it does: click, double click, or drag on/off. Holding too long cancels, which is an escape hatch for mistakes. While the eye is shut, the on-screen HUD shows a live tier bar, so the user can see which action they'll get before opening their eye.

• Drag latches: one long wink presses the button, the user moves their head freely, and a second long wink releases. Nobody can hold an eye shut for the several seconds a drag takes while also watching where they're going.
• Jaw open toggles scroll mode, where the head drives the scroll wheel instead of the pointer.
• Both eyes held 800 ms pauses or resumes everything.
• Dwell menu: for people who can't wink one eye on its own, holding the head still opens a radial menu, and tilting towards an item selects it. No winking needed.

4. Tuned to the person, not the developer

Every face is different, so the app measures the user instead of guessing:

• Calibrate records resting, natural blinks, and left and right winks for about 40 seconds. It then suggests thresholds and warns when winking won't be reliable for this person, pointing them to the dwell menu.
• Diagnose records 25 seconds of normal use, runs it through the real detector, and names the exact setting to change, for example "your winks are rejected as blinks: raise the veto threshold".

These tools found a real problem with the original defaults. One-eyed winks peaked at 0.48, under the original 0.50 threshold, so winks never registered at all. The thresholds now come from measurements.

5. Safe by design

A head-driven pointer that misbehaves is hard to stop with the head-driven pointer. So there are several independent ways to stop it:

• Both-eyes gesture or a hotkey pauses everything.
• Touching the physical mouse pauses control at once.
• Losing the face stops movement. If the face is lost for 3 seconds while dragging, the button is released.
• Every exit path releases any held button, so a drag can't survive a crash or a quit.

Key features

• No extra hardware. A built-in laptop webcam is enough.
• Fully offline. No video leaves the computer.
• Click, double click, right click, middle click, drag and scroll, all hands-free.
• Blink-proof clicking: 0 false clicks from 36 natural blinks in a simulated 2-minute reading session, and 10 of 10 deliberate winks detected.
• Wink-free mode (dwell menu) for people who can't wink one eye.
• Personal calibration and diagnosis instead of fixed thresholds.
• Portable Windows build: unzip and run, with no Python install and no internet needed.

Technical architecture

Stack: Python, MediaPipe, OpenCV, NumPy, pynput and PyInstaller, on Windows. Every tunable value lives in one config.yaml that sits next to the exe, so a person's tuning travels with the folder.

Testing: 77 automated checks across 6 suites. Most need no webcam: blinks and winks differ only in timing and in whether both eyes close, so synthetic eyelid streams can test the detection logic thoroughly. They also cover the signal maths, actions, calibration, diagnosis and the cursor driver.

Challenges

Blinks vs winks. A threshold alone can't separate them. The combination of veto, duration tiers and refractory period does, and the veto threshold is the single most important setting.

Signals that disagree. The rotation and nose signals had opposite vertical signs (correlation −0.97 before the fix), so blending them partly cancelled vertical motion. After flipping one axis, they agree at +0.88 horizontally and +0.95 vertically.

Fine control at slow speeds. Below one pixel per update, cursor movement rounded down to zero. Sub-pixel accumulation in a separate 120 Hz loop fixed it.

Packaging for non-technical users. MediaPipe and OpenCV make a large bundle (about 90 MB zipped). Choosing a folder build over a single exe gave fast start-up and a config file people can actually edit.

The outcome

• A working hands-free mouse that runs on any Windows laptop with a webcam.
• Clicking that ignores natural blinking, tested with synthetic reading sessions and real recordings through the diagnose tool.
• Per-person tuning built in, because the first defaults failed on a real face, and measuring fixed it.
• A portable build for people who will never install Python.

What's next

Pointing works. The next goal is a full day at the computer without anyone else touching the keyboard or mouse. The plan is written and staged:

1. Remove every remaining key press: starting control, recentering, settings and quitting all move to face gestures and voice. Add a status pill and sound cues.
2. Offline English voice commands and dictation, so the user can type, press keys and switch apps by speaking.
3. "Show numbers" to click any button by saying its number, plus a mouse grid for apps without accessibility support.
4. Video calls, by sharing the camera as a virtual camera, so face_mouse and Teams or Zoom can both use it.

Each stage ends with a test by a real user who can't use their hands.

What this means for other businesses

Accessibility doesn't have to mean expensive hardware. A webcam and careful signal processing go a long way.

Measure the user. The biggest improvements came from measuring real faces, not from guessing better defaults.

Design the escape routes first. For anyone who can't reach the mouse, a tool that can't be stopped is worse than no tool at all.

If you're building software that should work for everyone, including people who can't use a keyboard or mouse, I'd be glad to help.