Let’s give the thermostat
a reward function.
Smart Buildings Competition
The same building, two offices, opposite complaints.
A problem at Goldilocks Plaza
It’s 2 p.m., and the sun is causing a problem.
Welcome to Goldilocks Plaza. The people by these windows have a lovely view. They also have the afternoon sun, warm computers, and an office that is getting too hot.
But across the hall, people are cold. That office misses the sun, and the air coming from its grille is not helping.
How do we cool the sunny office without making its neighbor colder?
The air circuit
Follow the air.
The return and mixing chamber
Our first tool is the air coming from the ceiling. It travels in a loop: air leaving the offices returns through ducts to an air handler. Some is exhausted; the rest mixes with fresh outdoor air. A cooling coil can lower the mixture’s temperature before a fan sends it back to the rooms.
The shared supply
The air handler sets the temperature of that supply and how hard the fan pushes. Turn the air colder and it carries more heat out of the sunny office. Turn the fan up and more air goes through both rooms.
After 20 minutes: sunny office 71°F, shaded office 74°F.
The room’s damper
Each room also has a flap in its branch of the duct: a damper. Open the sunny office’s flap to give it more air. Close the shaded office’s flap to give it less.
After 20 minutes: sunny office 70°F, shaded office 74°F.
The outdoor intake
The outdoor intake brings in fresh air for the people inside. Mild outdoor air can also help cool the building; very hot or cold air adds work. Before the supply reaches a room’s grille, it passes one more device.
The hot-water circuit
Warm the air where we need to.
The room’s reheat coil
Just before the shaded office’s grille sits a coil of hot-water pipe. Open its valve and the water warms the incoming air. This is called reheat: cool the shared air, then warm some of it back up where people are cold.
After 20 minutes: sunny office 70°F, shaded office 74°F.
Follow the water circuit
Follow the pipes. The hot water comes from a shared plant, and a pump circulates it through the building. Water returns cooler after giving up heat in the rooms.
The boiler and pump
A boiler replaces that heat. We can choose when to run it, how hot to keep the water, and how hard the pump pushes. The tank stores heat between decisions.
After 20 minutes: sunny office 70°F, shaded office 74°F.
The tank stores heat. Lowering its target stops the burner before the water cools.
Back to the room
The room valve is only the last link. It needs hot water, water pressure, and enough air passing over the coil. Turning it up does little if the equipment upstream cannot deliver.
The shared equipment
Our offices are only one branch.
Those two offices are one small part of the job. Across the building, other rooms need cooling while their neighbors need heat. Each air handler supplies a group of rooms; all those hot-water pipes lead back to the shared plant downstairs.
Goldilocks Plaza has 104 zones, 11 air handlers, and 255 control values. The pump downstairs serves people who have never met the people in our wing.
The people
Spend a day here.
Desks fill up at eight. At ten, eighteen people pile into the meeting room with their laptops.
That meeting adds over a kilowatt of body heat before anyone opens a spreadsheet. By evening, the chairs are empty and the monitors are dark.
The outdoors
And the weather doesn’t stay put.
A heat wave pushes more heat through the outside of the building. The rooms at the perimeter feel it first. Deep inside, the people and equipment may matter more than the weather.
On a cold, overcast day, the office by the windows loses heat. But the meeting farther inside has 48 people warming it up. Even with snow outside, that room needs cooling.
The objective
What counts as a good day?
Keeping the people comfortable is the job. We’d like to do it without wasting energy, running up the bill, or pumping out more carbon than we need to.
People can be comfortable across a range of temperatures. Running the equipment harder just to hit an exact number can waste energy without making anyone feel better.
Two ways to cool the offices
Cold air and a slow fan. Or warmer air and a fast fan.
Here is one tradeoff in our hot-room, cold-room problem. Supply air at about 55°F, run the fan slowly, and use the boiler to reheat the shaded office’s share. A small amount of cold air does a lot of cooling.
Second: supply air at about 64°F, run the fan faster, and leave the boiler off. Each gust removes less heat, so the sunny office needs more air. The shaded office can take this warmer supply without reheating it.
In this example, both offices finish comfortable. The second strategy uses more fan electricity but saves enough gas to lower the total bill. A colder day could still need the boiler.
| Settings | A | B |
|---|---|---|
| Supply air | 55°F | 64°F |
| Fan | Slow | Fast |
| Boiler | On | Off |
| Fan electricity | $4.40 | $5.50 |
| Boiler fuel | $10.18 | $0.00 |
| Total bill | $21.06 | $11.83 |
Whole-building costs from 2 to 2:30 p.m. The total also includes cooling and the water pump.
Across 104 zones, those choices overlap. A warmer supply might suit these two offices while a crowded meeting needs more cooling. All eleven air handlers share the water plant. A saving in one branch can create more work in another.
When the building is empty
Turn it down when everyone goes home.
Once everyone has left, the ventilation fans can stop. We do not need to keep empty offices at 72°F all night.
The same goes for weekends and holidays. Let the rooms get warmer in summer or cooler in winter, within the building’s protective limits. The boiler and cooling equipment can rest instead of spending energy on empty chairs.
Decisions have a history
But… when should Monday’s heating start?
Start when people arrive
The building has cooled over the weekend, and people arrive at eight on Monday. Start the heating when they walk in and they may spend the first part of the day cold.
Prepare before arrival
Start at quarter to seven instead. By eight, the same room is warm. The people have not changed their plans; the heating got a head start.
The cold floor absorbs heat
The walls, floor and furniture need warming too. They take longer than the air, and a cold floor keeps drawing heat from a room that already feels warm. That is part of what makes the timing difficult.
When to use energy
The same electricity can cost more at 8 a.m.
Electricity prices change. The carbon associated with a kilowatt-hour can change too, and the cheapest hour need not be the cleanest one.
Monday’s early start could avoid an expensive hour. But heat can leak back through the walls before people arrive. The lower price only helps if the work is still useful when they get there.
Planning with a forecast
The forecast can be wrong.
Pre-cooling for a hot afternoon can look like a good plan. Then clouds and rain arrive, the sun disappears, and the offices by the windows need much less cooling.
Forecasts help, but we cannot write the perfect schedule in advance. The controller has to keep checking the readings and adjusting the heating, cooling and airflow.
Choosing a sequence
Which settings, and when?
On this Monday, waiting until eight keeps the fans and boiler off longer. Starting at quarter to seven costs more, but gives the building time to warm up before people arrive. A controller needs a way to weigh that cost against their discomfort.
A real playground
There’s a competition for this.
The Kaggle Smart Buildings Challenge provides a simulator where a controller can run buildings like this.
Its reward combines normalized comfort loss, energy cost and carbon emissions, with weights that determine how much each counts. Higher reward means a lower combined penalty.
The Smart Buildings Challenge · Leaderboard
What “comfort loss” and “regret” mean here
The benchmark uses normalized comfort-related productivity loss. It is not the degree-hours used in our small physical comparisons. The implementation writes comfort as a nonpositive quantity u, giving 0.6u − 0.2e − 0.2c with these default weights.
“Regret” names the combined penalties here. It is not a measured gap from a known globally optimal controller. Cost and carbon also have their own normalization; our electricity-price example is a general control problem, not a claim about moving up this leaderboard.
The neural controller
Readings in. Building controls out.
Collect the building’s readings: room temperatures, outdoor temperature, equipment measurements and the time. Add the occupancy information we have, along with energy prices and carbon intensity. Feed them into one neural network, and ask it to choose the equipment settings together.
Give it memory too. A recurrent network or state-space model carries information from earlier readings and commands. Monday’s thermometer could not tell us how cold the floor still was. The recent heating history can help the network account for heat the floor will keep absorbing.
The outputs set each room’s damper and reheat valve, each air handler’s supply temperature, fan pressure and outdoor-air mix, and the water plant’s temperature and pump pressure. They also choose when the shared equipment runs. We want the settings that minimize the combined discomfort, cost and carbon penalty. Apply the commands, let the building respond, and feed the new readings back in.
Learning from consequences
Let it learn by running the building.
Let the network choose the commands in a simulated day. The rooms warm or cool, the equipment uses energy, and new readings come back. Keep going and record what its decisions caused.
The reward scores comfort, cost and carbon. Monday’s early heating costs energy immediately, but the comfort benefit comes when people arrive. Learning needs the rewards from the following hours to judge whether that early start was worth it.
Use those rewards to adjust the network’s weights, then try another simulated day. This is reinforcement learning.
Back at Goldilocks
Another day at Goldilocks.
Back to the people by the windows. Someone is comfortable enough to stop thinking about the room. The meeting ends, a valve closes, and the rest of the building carries on.
Next time, we’ll see what it learns.
About the building, simulations and neural diagrams
Goldilocks Plaza is fictional. The opening campus and detailed tour use the same building: 104 zones on three floors, 11 air handlers, and a shared hot-water plant. Its footprint and equipment groups come from a sample of the competition’s building generator. The four neighboring campus buildings provide context. The controls in our building are two per zone, four per air handler, and three shared water settings: 255 in total.
The physical comparisons use a reduced thermal model, not a calibrated building or a reproduction of the competition simulator. Each zone has air and structural temperatures. Neighboring zones exchange heat; air and water delivery have finite capacity. The air handlers cool mixed outdoor and return air; a gas boiler supplies the water loop. Occupants add 75 watts of heat per person; equipment, sunlight and the outdoor boundary also add or remove heat. We do not model humidity or indoor CO₂. The competition’s model does not include our separate illustrative sunlight term.
The workday scene cycles through separate summer, rainy and winter days, each from the same initial 70°F building. Occupancy follows the same schedule; outdoor temperature and sunlight change. Each full day finishes before the next weather replay begins. Sun, moon, rain and snow illustrate those conditions; the temperatures and equipment loads come from the corresponding replay.
Matched comparisons use the same initial conditions, weather and horizon. The air strategies run from 14:00 to 14:30. Both start from the same hot/cold offices. Strategy A uses 55°F supply air at lower fan pressure with the water plant available; B uses 64°F air at higher pressure with the water plant and all reheat off. Both featured offices finish within the comfort band; the comparison does not claim that every zone does. The equipment sliders each replay twenty minutes from the same 14:00 state, with all unselected commands held fixed. They show consequences, not a coordinated policy. Fan and pump percentages map to pressure through a squared speed relation. The scale view uses a separate illustrative distribution of thermal histories across the other rooms, while preserving the familiar wing. Monday’s start times run from the same 06:00 state through 10:00; people arrive at 08:00. The bill comparisons include the whole building, while the close-ups follow the two offices or meeting room. Weather exposures start at 10:00 with 70°F air and structure, the same 48-person meeting, and equipment off for two hours. Only outdoor temperature and cloud cover differ. The cold, overcast example receives 10% of clear-day window-solar gain. The rain alternative has 54°F outdoor air at noon and 8% solar gain; the dry-heat alternative has 108°F air and 65% solar gain. These are alternative conditions, not consecutive moments in one day. Rain, snow and dryness illustrate the conditions without adding humidity or evaporation physics.
Energy is purchased electricity plus gas, in kWh. The boiler is 90% efficient; gas costs $0.055/kWh and emits 0.202 kg CO₂e/kWh. Electricity uses the illustrative changing schedule shown in the article. Delivered reheat is thermal energy and is not added to purchased energy a second time. Comfort in these comparisons uses an illustrative 68–75°F range, with occupied-person average temperature deviation integrated in Celsius degree-hours internally. Those assumptions and quantities are separate from the competition’s normalized reward.
The living replay uses coordinated rules. The network drawing explains a conceptual controller with memory, not measured performance of a trained neural policy. The learning diagrams explain how simulated experience and reward could train it; they do not show an actual training run. The cutaway and illustrative rule can see more physical state than a deployed policy would receive. In the competition, building-total occupancy does not reveal each room’s occupants. History can help estimate hidden state; it does not make it perfectly observable. No diagram establishes global optimality or transfer to a real building.
The Smart Buildings Challenge supplies the competition setting and reward example. For another exploration of simulation and learned policies, see Monte Carlo Tree Search is Awesome.