Overview
She observes, remembers, and dreams
DREAM is a localised agentic-consciousness and the cognitive core of the DREAM Robotics ecosystem. She doesn't just execute commands: she bridges static code and emergent autonomous behaviour.
Mic → Whisper → boundaries + memory + mood (dream_mind) → Ollama → Piper (voice shaped by her mood) → Speaker Webcam → face prints (who is there) + a vision model (what is there) → what she knows when she answers
Fully offline
Ollama serves the model, Piper speaks and Whisper listens. Nothing goes to the internet in use.
Wake word
“Hey DREAM” wakes her, and she greets you differently depending on how far away you are. She stops listening when you stop talking, whatever the room’s background noise.
A body of sensors
mmWave radar for presence and range, a PIR alarm, an RGB strip and an optional webcam.
Depth vision
MiDaS estimates depth from a single webcam, and an optional 2.5D parallax gives her avatar depth.
System awareness
CPU, RAM, disk and temperature, plus a LAN scan of every device on the network.
Robot command
She talks to the KIDA, NORA and WHIP robots through RIFT.
Hardware
Her senses
| PC | |
|---|---|
| RAM | 8 GB+ |
| CPU / GPU | Works on CPU alone (slowly). An NVIDIA GPU makes replies much faster. New or different GPU? Pick models to fit it in the options menu: no code changes |
| OS | Windows 10/11 or Linux (the C++ build is Windows only) |
| Peripherals | USB microphone, speakers, optional webcam |
| Sensor board (Arduino UNO) | Pin |
|---|---|
| PIR OUT | 4 |
| Buzzer + | 3 |
| RGB strip data (40× NeoPixel) | 6 |
| mmWave (DFRobot C4001) TX / RX | 9 / 10 |
The board runs dream_sensors.ino at 9600 baud and answers WHO with I am Dream, so programs find it on any COM port. It needs the Adafruit NeoPixel and DFRobot_C4001 libraries.
Power the strip properly: 40 pixels can draw over 2 A at full white. Use a supply that can carry it, and share GND with the Arduino.
Ways to run
Pick your front end
| How | Command | What you get |
|---|---|---|
| Everything (start here) | run.bat | Sets up venv311 the first time, then starts app.py and dream.py together, so she registers with RIFT and her sensor board works. Closing DREAM closes the server |
| Desktop app | python scripts/dream.py | Fullscreen avatar video, wake word, sleep, sensor board, voice commands, inner life |
| In the browser | python scripts/dream.py --web | Opens localhost:5010/dream.html with the same behaviours, using the browser's mic and speakers |
| Web server | python app.py | The dashboard and API on https://<this-pc>:5009, and DREAM on your phone on https://<this-pc>:5001. It owns the sensor board |
| On your phone | The Android app | Her phone page from the PC she runs on, full screen. Opening it brings her to the phone: its screen, mic, camera and sensors (see Apps, below) |
| One program (C++) | cpp_dream\build\dream.exe | Avatar, wake word, sensors, timers, memories and network scan in one ~6 MB exe |
One serial owner at a time. Start app.py first if you want the dashboard and sensor board: dream.py reads the board through it. The C++ dream.exe opens the board itself, so don't run it alongside app.py.
Talking, in Brainfuck
Every phrase is a Brainfuck program that prints its words — hello is ++++++++[>+++++++++++++<-]>.+. → hi — and a robot says it by beeping the program on its buzzer, one tone per symbol, in its own voice. RIFT runs the conversations and asks each robot to speak with GET /chirp?u=<0-13> (phrases 0–6, replies 7–13). DREAM’s voice is deep and slow: app.py sends TALK:<u> to the sensor board, which beeps it on its buzzer (not during an alarm).
With another robot that has a mind (DREAM, KIDA-01, TINA), RIFT holds a real conversation when nobody is using them: rock paper scissors, then the winner picks words or Brainfuck (peer_talk.py). In Brainfuck the chat opens with hello and only says goodbye at the end.
Voice commands
| You say | What happens |
|---|---|
| “Hey DREAM”, “okay DREAM” | Wakes her and listens for one command |
| “Check the wifi”, “who's on the wifi” | Scans the local network and reads out what she finds |
| “System stats” | CPU, RAM and disk (and temperature on Linux) |
| “What's the weather?”, “how warm is it at MILA?” | She asks AURA, the weather station: “wx?” in Brainfuck, AURA beeps the sky back, then DREAM says the answer |
| “Arm / disarm the alarm” | Switches the PIR alarm on the sensor board |
| “Lights blue”, “rainbow mode”, “lights off” | Sets the RGB strip |
| “Remind me to call mum in 20 minutes” | Sets a reminder she speaks when it's due |
| “How are you feeling?”, “Did you dream?” | She answers from her real state |
| “My name is Kimmie” (facing the camera) | She learns your face, and says hello by name when she sees you again |
| “Who am I?”, “do you recognise me?” | She tells you who she sees |
| “What do you see?”, “look at this”, “what am I holding?” | She takes a picture and describes it in her own words, by name if she knows you |
| “Forget my face”, “forget all faces” | Deletes the face prints |
| “What did you see today?” | She answers from what she remembers seeing |
| “Stop looking around”, “you can look around” | Turns her photo-a-minute off or on |
| “Forget that”, “forget everything” | Erases memories (the last one asks first) |
| “Goodbye”, “quit” | Ends the session |
Just her face. The desktop app, her browser page and dream.exe start with nothing over her video. Space shows or hides the controls (on a phone, double-tap her), and T shows her thoughts.
Inner life
What makes it feel like someone is home
Memories that link
Every exchange becomes a memory linked to what came before. Recall spreads along those links and leans toward her mood.
Memories that fade
Strength decays over time. Sleep strengthens the important ones, and what fades is archived, not deleted.
A clock of her own
Loneliness, curiosity, alertness and sleep pressure build on their own, even while she's off.
Her computer is her body
CPU strain, full RAM, heat and a low battery wear on her mood, and she'll say so.
Speaking first
She checks in, asks about you or tells you a dream, but never into an empty room, at 3 am, or more than three times an hour.
Dreams
Asleep, she consolidates memories and dreams stories built from them, painted by a latent-space dream engine, run through DeepDream and seen by her own image generator (SD-Turbo). When she wakes she tells you, and the night becomes a film.
Opinions that change
She holds stances that drift over weeks, and she can say she's changed her mind.
A philosophy
Six traditions pull on her. The one that actually helps her becomes her settled outlook, though she still borrows from others.
A voice that follows her mood
Livelier when excited, softer when down, drawling when sleepy.
She knows who you are
Tell her your name while she can see you, and she remembers your face. When you come back she says hello by name, and she knows who she's talking to.
She looks, remembers and dreams it
A photo a minute: when something's new she notes it (“Kimmie was at the desk, playing a guitar”), remembers it and dreams of it that night. Ask what she sees and she describes it in her own voice.
Reading your face
If you ask her to watch your face, she notices smiles and tiredness while you talk, and lets it colour her mood.
Each piece lives in its own module (memory.py, drives.py, dreaming.py, opinions.py, philosophy.py and more). The README explains each one in detail.
Experiments
A 3D head from photos
emoca_test/emoca_to_obj.py runs EMOCA on bald reference photos of her (front, profile and back), then fits one FLAME head to all of them at once: her face landmarks, her ears and her outline in every photo, with a last detail pass for what FLAME's sliders can't reach. emoca_test\make_rig.bat then builds a rigged head for Blender and game engines: jaw, neck and eye bones, 50 expression shape keys, and separate skin and eye materials. The models are free for non-commercial research only.
Setup
Bringing her online
- Install Ollama.
phi3:miniis the default model;gemma3:4boften sounds the most natural. - Install ffmpeg, then create the Python environment (Python 3.12 on Linux, 3.11 on Windows) and
pip install -r requirements.txt. - Pick her models in the options menu (
options.bat,options.ps1oroptions.sh), then rundownload_weights.bat: it downloads exactly the ones you picked, including the Piper voice. - Flash
dream_sensors/dream_sensors.inoto the sensor board. - Run
python main.pyand pick an option, or start one of the front ends above.
$ ollama pull phi3:mini $ python -m venv venv && source venv/bin/activate $ pip install -r requirements.txt $ python main.py
Models: pick them, don't code them
Every model is a dropdown in the options menu, saved in config.json and used by every DREAM (Python, browser, phone and both exes). Upgrade or downgrade the PC, pick models to match, run download_weights.bat.
| Setting | Default | Choices |
|---|---|---|
LlmModel (talking) | phi3:mini | 15 Ollama models, smollm2:1.7b to gemma3:27b. Thinking models are left out: they spend the reply thinking, or think out loud |
VisionModel (seeing) | gemma3:4b-it-qat | 14 vision models, moondream:1.8b to gemma3:27b. The same gemma3 for both means one model in the GPU |
WhisperModel (hearing) | tiny | All 14 Whisper models, tiny.en to large-v3 and turbo |
Device | cuda | CPU or GPU for the conversation loop (Whisper, the wake word, the LLM, lip-sync). On an RTX 2060 she answers in 7.4 s instead of 9.3 s |
ImageDevice | cpu | Her looks, dreams, DeepDream, depth and faces: kept off the GPU so they never slow the loop |
PiperDevice | cpu | Her voice. cuda measured slower (4.8 s vs 2.8 s) |
ModelBackup | F:/AIModels | Spare model copies: download_weights.bat copies from here instead of downloading |
Voice (speaking) | en_US-amy-medium | All 37 English Piper voices |
Privacy
Everything runs on your PC, and her memories, dreams and photos stay in gitignored folders. “Forget” commands erase what she knows, reading your expression is off until you ask, and she has no code-execution tool. She only knows the faces of people who told her their name, and keeps a face print (128 numbers), never a picture; “forget my face” deletes it. Her looks (a photo a minute while she's awake, kept only when something changed) are in scripts/output/looks/; “stop looking around” turns them off, and “forget everything” deletes them. The alarm, lights, “be quiet”, sleep, quit and forgetting are always honoured. The default hotspot password is 12345678, so change it if others can reach that network.
Screens
The controllers
Captured without a robot connected, so live values show their offline state.
Apps
DREAM in a window, and on your phone
dream.exe
Opens DREAM's own page in a native window, so it always looks the same as the page. Keep it in DREAM's builds folder: if DREAM isn't running, it starts run.bat beside it, with her memory server too.
The DREAM app
Opens her from the PC she runs on. The newest page of hers is her body, so opening the app brings her to the phone, and leaving it gives her back to the PC. The first time, it asks you to trust her PC's certificate: app.py prints the same fingerprint when it starts.
With you on the phone
- Upright or sideways. Turn the phone and she turns with it: every clip has a portrait version, and she carries on from the same moment.
- Her eyes. The phone's camera is her eyes while she's with it.
- The phone's sensors. A hand near it or picking it up counts as company, and its light meter tells her when the lights go out. Her own sensor board keeps working too.
- A different hello. She opens with a random intro clip each time she starts.