Report
Measuring Engagement, 2014: how the walk is drawn
Part of Measuring Engagement, a 2014 study by the Iota team at SapientNitro. The walk page says what the study was and why; start there if it is new to you.
This report. It records how the walk page was rebuilt in 2026 from the original files: where each stream comes from, how the path was placed from the video, how the readings were fitted to that path, and what the result can and cannot support. It is an illustration and not a measurement of engagement.
Two walks
The team’s 2014 figure lays the route, head movement, footsteps, light and sound along one line through a 3D model of the streets. The original files show that these come from two trips.
The video is from 3 April 2014: one uncut clip from Glass, and a second from the camera that followed. No sensor readings from that day have been found, after a search of the archive. TJ: “I think we failed to log the data, thus the repeat trip to simulate.” The readings are from that repeat trip, on 15 May. The Glass log’s own clock says so, and a phone app, Moves, logged the same walk and stop to within about 30 seconds.
So no reading can be matched to a moment in the video. There is a test for this: a head turn seen in a video should show as the same turn in its sensor log. On 7 April 2014 it lined a Glass clip up with its own log in all 19 two-minute windows. On the walk’s video it finds nothing consistent.
What joins the two walks is where they were. TJ: “I think the real anchor is position, which we get from the video.” So the video gives the path and the head, and the readings are fitted onto that path. A head turn placed on the plaza was not a look at the plaza on 3 April.
Where each stream comes from
| On the page | Walk | Source |
|---|---|---|
| The dot and the path | 3 April | The Glass video, placed as the next section describes. |
| Head, the red line, head turns | 3 April | The Glass video: the camera sits on the head, so each camera position says which way the head faced. It is the head, not where the eyes looked. |
| Sound | 3 April | The clips’ own audio, with Glass’s microphone as the layer. |
| Transcript | 3 April | Both cameras’ sound, transcribed by machine (Whisper) and not checked by ear. TJ has read it: “harmless and modestly correct”. |
| The two clips | 3 April | Side by side and in step, in place of the 2014 edit. Faces are blurred, except TJ’s, and the clips play without sound. |
| Steps and the movement graph | 15 May | Glass’s accelerometer, fitted to the path. TJ: “Leave the existing accel data.” |
| Light | 15 May | Glass’s light sensor, fitted to the path. It cannot come from the picture: the camera’s auto-exposure holds its brightness steady while the real light changes about 30-fold. |
How the path is placed
A camera on someone’s head does not know where it is. GPS wanders among tall buildings and fails indoors, and in 2014 the team placed the walk by hand. In 2026, software called COLMAP worked out how the camera moved from the video itself, by following details as they shift from frame to frame, a method called structure from motion. That gives movement with no real-world size or direction, so each stretch is tied to a 3D model of where it happens: the office, the lobby, the team’s 2014 model of the streets, and the Monadnock Building.
TJ set the method, after first attempts looked right in some frames and far off in others:
- Rebuild from still, sharp frames. A turning head smears the picture and bends it, because the camera reads each frame line by line.
- Tie to crisp fixed edges. Each stretch takes one size, one turn and one shift, from edges read by hand: floors meeting walls, window mullions, desk edges, door frames and curbs. Furniture, wall art and people do not count.
- Fill in between. Points tracked from one frame to the next carry the head’s direction across the frames a rebuild lost.
- Check by eye. The model’s low edges are drawn over the frames, which are graded one by one, naming the misses. There is no automatic score for this.
Real-world size has three separate checks, which agree within about 3%: TJ’s measured eye height, 69.25 in; the published height of Alexander Calder’s Flamingo, 53 ft; and the block from Adams Street to Jackson Boulevard in the 2014 model. Placing the Flamingo sculpture gives the working.
Video times are minutes and seconds into the 2014 edit, which the page’s clock also uses. Straight lines between known places are named as such.
| Video | Stretch | How it is tied | Weakest point |
|---|---|---|---|
| 0:00–2:46 | Desks, corridor, big screen, open floor, hall, the car down | A stretch at a time, to the office model’s fixed edges. Reads miss by a median of 2 to 8 pixels. | 1:54–2:00 and 2:15–2:21 are straight lines. Separately tied stretches meet 12 to 36 in apart. |
| 2:46–4:00 | Lobby, out | One fit to the lobby model: 33 reads in 9 frames, median 5.2 pixels. | Eye height comes out at 69.0 in, but that rests on a 5 in sill estimated by eye; 65 in without it. |
| 3:42–7:00 | Street, out | Three rebuilds: two onto the tied track from 5:13 (median 3 in and 19 in off), one onto the first corner’s two curbs and those. | At the first corner the 2014 model’s outline stands on open sidewalk, 10 to 15 ft from the glass, probably because it follows the tower and not a recessed ground floor. Not proved. |
| 7:00–14:00 | Crossing and café | The tied track, then the Monadnock model, in the café to about 6 in. | Differences of up to 18 ft along Jackson that nothing in the model settles. |
| 14:00–18:14 | Street, back | Four stretches, cut where the rebuilds’ scale jumps, tied to the Monadnock’s door and the curbs TJ crosses. | No scale around 15:15 or from 17:14 to 17:28, so the dot goes straight there. A wrong single-frame rotation at 16:24. |
| 18:14–18:38 | Lobby, back | Seven frames solved alone from crisp corners. A dense rebuild gives the head between them, and the place only where its motion is believable. | About 16 of 22 s is straight lines between solved frames. Three frames rest on two reads. |
| 18:38–19:17 | The ride up | Direction from a rebuilt piece on the car’s own lines, moved 4° to agree with door reads. | About 8° of doubt. Which car is a reading from the frames. |
| 19:17–21:36 | Back in the office | A dense rebuild at ten frames a second, each part fitted through the way out’s edges. | The corridor’s far end is a few feet uncertain. About a ninth, 15.6 s of 139, is straight lines. |
The head was tested by hiding stretches of known frames and predicting them. With 8 s hidden in the office, the way out has a median error of 3° and the way back 7°, with one trial in twenty worse than 11° and 30°. With 14 s hidden in the café the median is 1.3°, and the worst stretch reaches 44°. A fast turn that the known frames miss can be lost. As a further control, head turning measured from the picture scored 0.81 against the real sensor log of the 7 April clip.
How the readings are fitted to the path
Both walks are measured by distance along one route, 3,493 ft from the office floor and back, with the elevators counted by their height.
- Markers are places. A marker is a door, a corner or a crossing on that route. Twenty-one have a time in the readings, five of them inside the café by order only. Forty have a time read from the video’s frames.
- The readings say when. A sharp change of light marks a door, and a still spell marks a wait at a curb. Glass logged no pressure, so an elevator ride is inferred from a still spell of the right length between stretches of walking.
- Steps set the pace. Between two markers a reading moves forward in proportion to the steps since the last marker. During a wait it does not move, so a stop stays a stop. On the three long street stretches this comes to 3.07 to 3.17 ft per step, a plausible length.
- The video leads. It is never stretched. The page’s clock, slider and graphs run on the video’s time, with the readings plotted where they fall on it, so the readings’ time runs unevenly against it: between 0 and 3.1 s per second over any 20 s.
How the data is drawn
Every layer is drawn now from the data, and none is a drawing from 2014. The 3D marks take the forms of the team’s figure, at the proportions of its meshes: light a flat green band, the acceleration a flat blue sawtooth, sound a magenta curtain, and the head’s direction black arrows, red or green when it turns faster than 80°/s, the bound at which the count is about the figure’s.
The page opens on the whole walk, as the figure did. While it plays it shows the recent past, 20 s by default, fading, so that a long wait does not pile up.
The buildings, the office floor and the elevator are light outlines from the team’s 2014 model. Flamingo is the scene’s one solid, full size. Close in on the Monadnock, and its revised model fades in.
The transcript’s lines are shaded by an estimate of how likely each is someone talking with TJ: 60% from reading the words and 40% from the sound, that is, how far Glass’s microphone rises above the room, how much louder it is than the other camera’s, and whether the line has a pitch at all. Whisper invents lines in silence, and those are among the palest.
The 2014 version
Set Path to the team’s 2014 curve for the earlier version. Each reading stays where the team’s model placed it: their Grasshopper script spaces the readings evenly along a hand-drawn curve, so the walker’s apparent pace comes from how the curve was drawn. The video is warped by hand onto the readings’ timeline at eight moments that look alike. TJ compared this version’s drawing with the figure and agreed that it matched. The video-led version has not been compared.
What it can and cannot support
It can support these:
- Descriptions of each walk from its own streams: the 15 May readings, and the 3 April video and sound.
- Where TJ was on 3 April and which way the head faced, for the whole video, as firmly as the table above says for each stretch.
It cannot support these:
- Any statement tying a sensor reading to something visible in the video.
- Where TJ was on 15 May at a given moment. The readings are placed by the fit, and the light edges and steps constrain them to tens of feet at best.
- Reading the fitted log as a measurement.
- Any finding about engagement. It is one person and two walks, and the results are exploratory. TJ: “This particular study is more illustration than scientific; it is a provocation.”
The page does not mark the three moments the team’s papers report. It takes the idea and the figure from those papers, and not the details of the walk. All positions are relative to the team’s 2014 model, which has not been checked against a survey.
Appendices
Each appendix gives what is particular to one model or check, with its sources, fit and open points.
- The model of the office. The floor where the walk starts and ends.
- The model of the lobby. The ground floor the walk crosses, both ways.
- The model of the Monadnock Building. The coffee shop and the north end of the building around it.
- Placing the Flamingo sculpture. The check on the path along the street, and on its size.
The study, the walk and its instrumentation were a team effort by Iota, at SapientNitro. This rebuild was made in 2026 from the original files, at TJ’s direction and with Claude Code. Reconstruction: COLMAP. Transcription: Whisper.