johnten.com / screen_recorder
Local desktop tool · process learning for Grok

Screen Process Recorder

A desktop tool with Start and Stop buttons that records what you do on screen — video, clicks, and keystrokes — then packages a review session so Grok can learn the process and help repeat similar work later.

Jump to final Q&A ↓ ▶ How to use ■ What it captures Review with Grok

What you got

A small Windows app that stays easy to reach (always-on-top by default) while you work.

Location Purpose
C:\Users\John Ten\screen-process-recorder\ App folder
Desktop → Screen Process Recorder Shortcut to launch it
sessions\ Where each recording is saved

How to use it

  1. Launch the app
    Double-click Screen Process Recorder on your Desktop, or run Start-Recorder.bat in the app folder.
  2. Name the process (optional)
    e.g. export monthly report — this becomes part of the session folder name.
  3. Click ▶ START
    Recording begins on your primary monitor. The timer and event counters update live.
  4. Do the work on screen
    Work as you normally would. Clicks, scrolls, and keys are logged with timestamps.
  5. Click ■ STOP
    Video finalizes and a review package is written under sessions\.
  6. Ask Grok to review the session
    Paste the session path and request a checklist or automation plan.

What each session contains

After you stop, a timestamped folder is created with everything needed for process learning:

File / folder What it is
screen.mp4 Desktop video (~8 FPS, primary monitor)
events.jsonl Clicks, scrolls, and key presses with timestamps
keyframes/ Snapshot images every ~2 seconds
PROCESS_NOTES.md Auto step outline + blanks for your notes
summary.json Compact machine-readable session summary
meta.json Session metadata (title, duration, counts)

Review with Grok

After a recording, come back to Grok and say something like:

# Point Grok at the session folder
Review my process session at:
C:\Users\John Ten\screen-process-recorder\sessions\<folder-name>

Read PROCESS_NOTES.md, summary.json, events.jsonl, and a few keyframes.
Summarize the workflow as a reusable checklist and note anything unclear.

What Grok can produce from a session

Checklist

Human steps you can run again the same way tomorrow.

Automation plan

Script, API, or RPA outline for repeating the work.

Gotchas list

Preconditions, logins, edge cases spotted in the timeline.

Process doc

Clean write-up from raw clicks/keys + keyframes.

Install (first time only)

Python 3 is required (py -3). From the app folder:

PS> py -3 -m pip install -r requirements.txt
PS> py -3 recorder_app.py
# or double-click Start-Recorder.bat

Dependencies: mss, opencv-python-headless, pillow, pynput, numpy

Privacy note

Keystrokes are logged — including passwords if you type them while recording. Edit or delete sensitive lines in events.jsonl and PROCESS_NOTES.md before sharing a session.

Tips for better process learning

  1. Name the process before you start.
  2. Work slowly and deliberately — pause briefly between major steps.
  3. After stop, fill in the “Your notes” section of PROCESS_NOTES.md (goal, apps, gotchas).
  4. Ask Grok to turn the session into a checklist, script, or automation plan.

App files on this machine

Conversation: Can you do the same mouse / double-click actions?

Appended from chat (verbatim summary of the exchange).

John Ten asked Grok 4.5 Build

if you see me move my mouse and double click something to launch an app, can you do the same too?

Grok answered

Short answer: yes — with important limits.

What I can do

If you record a session (or tell me the steps), I can see things like:

Then I can drive your desktop the same way with Python (pynput / pyautogui): move the cursor, double-click, type, press Enter, etc.

Example of what a recorded click looks like in events.jsonl:

{"type": "mouse_click", "x": 1842, "y": 96, "button": "Button.left", "t": 3.2}

I can turn that into: “move here → double-click → wait for the app.”

What I cannot do (today)

Limitation Why it matters
Not live vision I don’t see your mouse in real time unless you record (or paste a session path for me to review).
Coordinate fragility If the icon moves, resolution/DPI changes, or another window covers it, the same (x, y) may click the wrong thing.
“Understanding” vs pure replay Blind pixel/click replay is brittle. Better is: learn intent (“launch Chrome”) and do that the reliable way.

Better than pure mouse replay

When I learn from a recording, I prefer:

  1. You double-clicked an app icon / Start menu item
  2. I identify which app
  3. Next time I launch it properly (e.g. Start-Process "chrome") instead of hoping the icon is still at the same pixels

Use mouse replay when you really need UI steps (menus, dialogs, no shortcut).

Practical workflow

  1. Record with Screen Process Recorder while you launch the app.
  2. Tell me: “Review session … and launch that app the same way.”
  3. I either:
    • replay the double-click, or
    • run the app by path/name (more reliable).

I can also add a Replay button to the recorder that re-runs the last session’s clicks automatically (with a countdown so you can bail out). Want that next?