Home Assistant MQTT: How to Control Custom DIY Devices Without ESPHome

You can integrate any microcontroller with Home Assistant over MQTT by publishing sensor data to discovery-compatible topics and subscribing to command topics. This gives you complete control over the firmware without ESPHome’s abstraction layer. The approach works with any language and any chip - ESP32, RP2040, STM32, or even a Raspberry Pi Pico W - and it is the right choice when your device needs custom protocols, bare-metal timing, or firmware features that ESPHome simply does not support.

The following covers when raw MQTT makes sense, the discovery protocol that auto-registers devices in Home Assistant, concrete firmware examples on the ESP32 and RP2040, bidirectional control patterns, and security hardening.

When to Skip ESPHome and Go Raw MQTT

ESPHome works well for most home automation sensors and switches. You write a YAML file, flash it, and the device shows up in Home Assistant. For 80% of projects that is all you need. But there is a remaining 20% where ESPHome gets in the way, and knowing when to reach for raw MQTT saves you hours of fighting the framework.

Custom communication protocols are a common reason. If your device needs to speak a proprietary RF protocol - decoding 433 MHz weather stations with precise timing, bit-banging I2C to an unsupported sensor, or implementing custom Modbus registers - ESPHome’s component model may not expose the low-level control you need. Writing your own firmware lets you control every microsecond of the signal.

Bare-metal performance is another. High-frequency data acquisition like vibration monitoring, audio FFT, or oscilloscope-like sampling demands tight loop timing. ESPHome’s main loop runs at its own pace and the YAML abstraction adds overhead that makes sub-millisecond timing unreliable.

Multi-core ESP32 usage comes up often too. The ESP32 has two cores, and dedicating one to real-time motor control or LED animation while the other handles MQTT communication is a clean architectural pattern. ESPHome’s single-threaded component model does not support this cleanly without fighting the framework.

Then there are non-ESP platforms: the RP2040 (Raspberry Pi Pico W), STM32 with Ethernet, nRF52 with Thread/BLE bridge, or any board where ESPHome support is limited or nonexistent. If you are already writing C or MicroPython for these chips, adding MQTT is straightforward.

Firmware size can also matter. ESPHome’s compiled firmware for a full-featured device easily exceeds 1.5 MB. Hand-written firmware with just MQTT and sensor code can fit in 200 KB, leaving plenty of room for OTA partitions on chips with limited flash.

And finally, understanding the MQTT protocol and Home Assistant’s discovery mechanism teaches you how smart home integration actually works under the hood. That knowledge makes you better at debugging ESPHome devices too.

MQTT Discovery: Auto-Registering Devices in Home Assistant

The most common misconception about raw MQTT devices is that you need to manually configure each entity in Home Assistant’s YAML. You do not. Home Assistant’s MQTT discovery protocol lets your device announce itself automatically, creating entities without any configuration on the Home Assistant side.

Home Assistant listens on a specific topic pattern for JSON discovery payloads:

homeassistant/<component>/<node_id>/<object_id>/config

Publishing a properly formatted JSON message to this topic creates the entity automatically. The <component> can be sensor, binary_sensor, switch, light, climate, cover, and many others.

Here is a minimal temperature sensor discovery payload:

{
  "name": "Workshop Temp",
  "state_topic": "workshop/sensor/temperature/state",
  "unit_of_measurement": "\u00b0C",
  "device_class": "temperature",
  "unique_id": "workshop_temp_01",
  "device": {
    "identifiers": ["workshop_node_01"],
    "name": "Workshop Sensor",
    "manufacturer": "DIY",
    "model": "ESP32-WROOM",
    "sw_version": "1.0.0"
  }
}

The device block is important. It groups multiple entities under a single device in the Home Assistant UI. If your board reports temperature, humidity, and has a relay, all three entities appear under one device card when they share the same identifiers value. Include name, model, sw_version, and manufacturer for a polished device page.

Availability and Last Will

Use availability_topic with an MQTT Last Will and Testament (LWT) message so Home Assistant marks the device as unavailable when it disconnects. When connecting to the broker, set the LWT to publish offline to workshop/sensor/availability. After the connection succeeds, publish online to the same topic. Add this to your discovery payload:

{
  "availability_topic": "workshop/sensor/availability",
  "payload_available": "online",
  "payload_not_available": "offline"
}

If your device loses power or the network drops, the broker publishes the LWT message automatically, and Home Assistant updates the entity status within seconds.

Controllable Entities

For controllable entities like switches and lights, add command_topic to the discovery payload. Home Assistant publishes ON or OFF to the command topic when someone toggles the switch in the UI, and your firmware subscribes to that topic to actuate the relay or LED.

{
  "name": "Workshop Relay",
  "state_topic": "workshop/switch/relay1/state",
  "command_topic": "workshop/switch/relay1/set",
  "unique_id": "workshop_relay_01",
  "device": {
    "identifiers": ["workshop_node_01"],
    "name": "Workshop Sensor",
    "manufacturer": "DIY"
  }
}

One important detail: retain the discovery config message with retain: true so that Home Assistant picks it up even if it restarts after your device. Retain state messages too so the dashboard shows the last known value immediately instead of “unknown” until the next sensor reading.

Firmware Implementation on ESP32 and RP2040

With the discovery protocol understood, let us look at concrete firmware patterns for the two most popular hobbyist platforms.

ESP32 with Arduino Framework

The PubSubClient library (v2.8+) is the most common choice. For non-blocking operation, the AsyncMqttClient library is more robust but adds complexity.

Here is a minimal ESP32 Arduino sketch that connects to WiFi and MQTT, publishes a discovery payload, and reads a temperature sensor:

#include <WiFi.h>
#include <PubSubClient.h>
#include <DHT.h>

#define DHT_PIN 4
#define DHT_TYPE DHT22
#define MQTT_BROKER "192.168.1.100"
#define MQTT_PORT 1883
#define NODE_ID "workshop_node_01"

WiFiClient espClient;
PubSubClient mqtt(espClient);
DHT dht(DHT_PIN, DHT_TYPE);
unsigned long lastPublish = 0;

void publishDiscovery() {
  String payload = "{\"name\":\"Workshop Temp\","
    "\"state_topic\":\"workshop/sensor/temperature/state\","
    "\"unit_of_measurement\":\"C\","
    "\"device_class\":\"temperature\","
    "\"unique_id\":\"workshop_temp_01\","
    "\"availability_topic\":\"workshop/sensor/availability\","
    "\"device\":{\"identifiers\":[\"workshop_node_01\"],"
    "\"name\":\"Workshop Sensor\",\"manufacturer\":\"DIY\"}}";

  mqtt.publish(
    "homeassistant/sensor/workshop_node_01/temp/config",
    payload.c_str(), true  // retain = true
  );
}

void reconnect() {
  while (!mqtt.connected()) {
    if (mqtt.connect(NODE_ID, "mqtt_user", "mqtt_pass",
        "workshop/sensor/availability", 1, true, "offline")) {
      mqtt.publish("workshop/sensor/availability", "online", true);
      publishDiscovery();
      mqtt.subscribe("workshop/switch/relay1/set");
    } else {
      delay(5000);
    }
  }
}

void callback(char* topic, byte* payload, unsigned int length) {
  String msg;
  for (unsigned int i = 0; i < length; i++) msg += (char)payload[i];

  if (String(topic) == "workshop/switch/relay1/set") {
    if (msg == "ON") {
      digitalWrite(5, HIGH);
      mqtt.publish("workshop/switch/relay1/state", "ON", true);
    } else {
      digitalWrite(5, LOW);
      mqtt.publish("workshop/switch/relay1/state", "OFF", true);
    }
  }
}

void setup() {
  pinMode(5, OUTPUT);
  dht.begin();
  WiFi.begin("YourSSID", "YourPassword");
  while (WiFi.status() != WL_CONNECTED) delay(500);

  mqtt.setServer(MQTT_BROKER, MQTT_PORT);
  mqtt.setCallback(callback);
}

void loop() {
  if (!mqtt.connected()) reconnect();
  mqtt.loop();

  if (millis() - lastPublish > 30000) {
    float temp = dht.readTemperature();
    if (!isnan(temp)) {
      mqtt.publish("workshop/sensor/temperature/state",
        String(temp, 1).c_str(), true);
    }
    lastPublish = millis();
  }
}

A few things to note. The LWT is set in the mqtt.connect() call - that fourth argument is the will topic, and the last argument is the will message. The reconnect() function re-publishes discovery payloads after each reconnection because the broker may have lost the retained messages during a restart. Sensor readings publish every 30 seconds using a non-blocking timer instead of delay(), which would block MQTT keepalive packets and cause disconnections.

ESP32 with ESP-IDF

For production devices, the ESP-IDF framework’s built-in esp_mqtt_client component is more reliable. It supports TLS, automatic reconnection, and an outbox that queues messages when the connection is temporarily lost. The API is event-driven:

esp_mqtt_client_config_t config = {
    .broker.address.uri = "mqtt://192.168.1.100:1883",
    .credentials.username = "mqtt_user",
    .credentials.authentication.password = "mqtt_pass",
    .session.last_will.topic = "workshop/sensor/availability",
    .session.last_will.msg = "offline",
    .session.last_will.retain = true,
};
esp_mqtt_client_handle_t client = esp_mqtt_client_init(&config);
esp_mqtt_client_register_event(client, ESP_EVENT_ANY_ID, mqtt_event_handler, NULL);
esp_mqtt_client_start(client);

The event handler receives MQTT_EVENT_CONNECTED, MQTT_EVENT_DATA, and MQTT_EVENT_DISCONNECTED events, letting you structure the firmware as a clean state machine.

RP2040 Pico W with MicroPython

The Raspberry Pi Pico W running MicroPython is a low-cost alternative. Use umqtt.simple2 from the micropython-lib package:

from umqtt.simple2 import MQTTClient
import network, json, time

wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect("YourSSID", "YourPassword")
while not wlan.isconnected():
    time.sleep(0.5)

client = MQTTClient("pico_node", "192.168.1.100",
                     user="mqtt_user", password="mqtt_pass")
client.set_last_will("workshop/sensor/availability", b"offline", retain=True)
client.connect()
client.publish(b"workshop/sensor/availability", b"online", retain=True)

discovery = json.dumps({
    "name": "Pico Temp",
    "state_topic": "workshop/sensor/pico_temp/state",
    "unit_of_measurement": "C",
    "device_class": "temperature",
    "unique_id": "pico_temp_01",
    "availability_topic": "workshop/sensor/availability",
    "device": {
        "identifiers": ["pico_node_01"],
        "name": "Pico Sensor",
        "manufacturer": "DIY"
    }
})
client.publish(b"homeassistant/sensor/pico_node_01/temp/config",
               discovery.encode(), retain=True)

while True:
    temp = read_onboard_temp()  # your sensor reading function
    client.publish(b"workshop/sensor/pico_temp/state",
                   str(round(temp, 1)).encode(), retain=True)
    time.sleep(30)

For the lowest latency and smallest footprint on the Pico W, the C SDK’s lwIP MQTT client is available, but it requires manual memory management and callback-based programming that most hobbyists will find harder to work with than MicroPython.

Bidirectional Control: Commands, State Feedback, and OTA Updates

A useful DIY device does more than report sensor data. It accepts commands and confirms state changes. Getting this bidirectional communication right is what separates a working prototype from a reliable home automation device.

Command Handling and State Confirmation

Subscribe to command topics in your connect callback. When a message arrives, parse the payload, actuate the hardware, and then publish the confirmed state back to the state topic. The key word is “confirmed” - always publish the actual state after actuation, not the commanded state. If the relay fails to switch because overcurrent protection triggers, the state topic should reflect reality.

For dimmable lights, Home Assistant sends JSON payloads to the command topic:

{"state": "ON", "brightness": 180, "color_temp": 350}

Your firmware parses this with ArduinoJson (on ESP32) or json.loads() (in MicroPython), applies the PWM values, reads them back, and publishes the actual state. This round-trip confirmation pattern prevents the UI from showing a state that does not match the hardware.

Remote Management

Three additional topics make remote management practical.

A restart command topic triggers ESP.restart() or machine.reset() for remote recovery of stuck devices. This saves a walk to the garage when a sensor hangs.

A config topic accepts runtime parameter changes - polling interval, calibration offsets, WiFi credentials - stored in NVS (ESP32) or flash (RP2040) so they persist across reboots without reflashing.

An OTA update topic can work two ways. The simpler approach is to publish a firmware URL over MQTT and let the device fetch it via HTTP OTA. The more involved approach streams firmware binary chunks directly over MQTT, which works when the device cannot reach an HTTP server. On ESP32, the esp_ota_ops.h API handles writing chunks to the OTA partition. On Pico W, you can write to flash and reboot into the new firmware.

Reconnection Strategy

MQTT connections drop. WiFi routers restart. Brokers go down for updates. Your firmware needs a reconnection strategy with exponential backoff: retry after 1 second, then 2, then 4, then 8, up to a maximum of 60 seconds. After each successful reconnection, re-publish discovery payloads and re-subscribe to command topics. Without this, a brief network blip turns into a permanently offline device.

unsigned long backoff = 1000;
void reconnectWithBackoff() {
  if (mqtt.connect(NODE_ID, user, pass, lwt_topic, 1, true, "offline")) {
    backoff = 1000;  // reset on success
    publishDiscovery();
    mqtt.subscribe("workshop/switch/relay1/set");
    mqtt.publish("workshop/sensor/availability", "online", true);
  } else {
    delay(backoff);
    backoff = min(backoff * 2, 60000UL);
  }
}

Security and Reliability Best Practices

DIY MQTT devices are only as secure as their weakest link. A few practical measures separate a hobby prototype from something you can trust on your home network.

Authentication and Encryption

Use MQTT authentication with per-device usernames and passwords. In Mosquitto , create a password file with mosquitto_passwd and reference it in mosquitto.conf. Never use anonymous access on a production network - a single compromised device could publish to any topic.

Enable TLS on the MQTT broker. Mosquitto supports TLS with Let’s Encrypt certificates. Configure your firmware to verify the server certificate. ESP-IDF has built-in TLS support with CA certificate bundles, and MicroPython’s ssl module can wrap the socket connection. The overhead is minimal on modern microcontrollers - the ESP32’s hardware crypto accelerator handles TLS handshakes without noticeable latency.

A Mosquitto configuration with TLS and per-device access control looks like this:

listener 8883
certfile /etc/mosquitto/certs/fullchain.pem
keyfile /etc/mosquitto/certs/privkey.pem
cafile /etc/mosquitto/certs/chain.pem

password_file /etc/mosquitto/passwd
acl_file /etc/mosquitto/acl

per_listener_settings true
allow_anonymous false

The ACL file restricts which topics each device can read and write:

user workshop_node_01
topic readwrite workshop/#
topic write homeassistant/sensor/workshop_node_01/#

Network Isolation

Isolate IoT devices on a separate VLAN so a compromised sensor cannot reach your main network. The MQTT broker bridges the VLAN boundary - it sits on both networks or on a management VLAN with firewall rules that allow only MQTT traffic (port 1883/8883) from the IoT VLAN. This way, even if someone exploits a vulnerability in your ESP32 firmware, they are stuck on a network segment with no access to your workstation or NAS.

Watchdog Timers

Hardware watchdog timers are your safety net against firmware bugs. On ESP32, enable the task watchdog with a 30-second timeout so the device reboots automatically if the main loop hangs:

#include "esp_task_wdt.h"
esp_task_wdt_init(30, true);  // 30s timeout, panic on trigger
esp_task_wdt_add(NULL);       // add current task

// In your main loop:
esp_task_wdt_reset();         // feed the watchdog each iteration

On RP2040, the hardware watchdog is available through machine.WDT(timeout=30000) in MicroPython.

Telemetry Buffering and QoS

When the MQTT broker is temporarily unavailable, buffer telemetry readings locally. Store them in a ring buffer in RAM or on flash (SPIFFS on ESP32, littlefs on RP2040) and flush the buffer when the connection is restored. This prevents data gaps during broker maintenance.

For QoS levels, use QoS 1 for state and discovery messages to guarantee at-least-once delivery. Use QoS 0 for high-frequency telemetry where occasional packet loss is acceptable - this reduces broker load and keeps latency low for time-series data that gets averaged anyway.

Wrapping Up

Raw MQTT integration with Home Assistant takes about the same effort as ESPHome, just in different places. You trade YAML convenience for complete firmware control. For standard sensors and switches, ESPHome remains the faster path. But when you need custom protocols, multi-core processing, non-ESP platforms, or firmware that fits in 200 KB, writing your own MQTT firmware and using Home Assistant’s discovery protocol makes more sense.

Start with a single sensor publishing temperature over MQTT. Get discovery working so the entity appears automatically in Home Assistant. Add a relay with command handling and state confirmation. Then layer on TLS, watchdog timers, and OTA updates. Each step builds on the last, and before long you have a template you can adapt to any microcontroller and any sensor in your workshop.