blog.tedder.dev

Building a METAR aviation weather map on an ESP32

tedder — Sun, 16 Aug 2026 16:00:00 GMT

I fly, which means I look at weather constantly. METARs — aviation routine weather reports — tell you ceiling, visibility, and flight category at any reporting station. The colors VFR/MVFR/IFR/LIFR map directly to green/blue/red/magenta in most flight planning apps. A while ago I decided to make that visual and physical: an LED strip that stays on my desk and shows current flight conditions at airports along my usual routes.

Here's how it evolved, and what I learned.

Hardware

The platform is an Adafruit QT Py ESP32-S2. It's small, has native USB, runs CircuitPython or ESP-IDF, and has enough flash and RAM for a WiFi-connected LED controller. A WS2812B strip plugs into one of the analog pins; an SSD1306 OLED 128×32 hangs on I2C for status readout.

The strip has one LED per airport. Right now I track a handful of stations — local fields and a couple of en-route ones. The OLED shows the last update time and a count of how many airports had valid METARs.

The backend: don't do METAR parsing on the microcontroller

Aviation weather APIs return JSON blobs or text, require TLS, and occasionally change schema. Parsing that on a microcontroller is painful — you burn flash on JSON libraries, you fight TLS certificate chains, and every API hiccup means you're debugging on a device with no debugger.

My approach: offload all of that to a Lambda@Edge function. The microcontroller sends a single HTTP request with a list of airport codes and gets back a simple CSV:

KPDX,VFR,0-127-0,8,0
KTTD,VFR,0-127-0,12,0
KHIO,MVFR,0-0-127,24,0

Fields: ICAO code, flight category, RGB color tuple, age in minutes, stale flag. The microcontroller splits the line, applies the color directly. No parsing, no JSON, no TLS from the device side (Lambda handles it outbound). This architecture has been solid — the device just asks "what color are these airports" and gets an answer.

Flight categories to LED colors

Category	Condition	Color
VFR	> 3000ft ceiling, > 5sm vis	Green (0, 127, 0)
MVFR	1000–3000ft or 3–5sm	Blue (0, 0, 127)
IFR	500–1000ft or 1–3sm	Red (127, 0, 0)
LIFR	< 500ft or < 1sm	Magenta (255, 22, 199)

The stale flag comes back set if the METAR is older than a threshold. I dim stale airports slightly to indicate "this data is old."

CircuitPython phase

The first version ran CircuitPython. CircuitPython is genuinely great for a project like this: the REPL is fast, you edit files over USB mass storage without any flashing ceremony, the Adafruit ecosystem has everything you need, and iteration is just save-file-watch-reload.

It held up well through about a year and a dozen feature additions. I added a web server for live airport configuration, NTP sync, a debug flag bitmask stored in NVM (persistent across reboots), a watchdog timer, and structured logging.

Then CircuitPython's networking started biting me.

The DNS problem

On certain builds of CircuitPython for the ESP32-S2, lwIP's DNS resolver would silently fail. The device appeared connected — it had an IP, it could ping — but getaddrinfo() would hang or return errors on the first few calls after boot. The workaround I ended up with was a SmartPool wrapper around socketpool.SocketPool that intercepts getaddrinfo() and falls back to raw UDP DNS against Cloudflare and Google if lwIP's resolver fails:

class SmartPool:
    def __init__(self, pool):
        self._pool = pool

    def getaddrinfo(self, host, port, *args, **kwargs):
        try:
            return self._pool.getaddrinfo(host, port, *args, **kwargs)
        except Exception:
            # Raw UDP DNS fallback to 1.1.1.1, then 8.8.8.8
            return self._raw_dns_lookup(host, port)

All the Adafruit libraries (adafruit_requests, adafruit_ntp, etc.) receive SmartPool instead of the raw pool, so the fallback is transparent. It worked — but it's the kind of thing that makes you wonder if you're approaching the floor of what CircuitPython's network stack can reliably support.

WiFi roaming detection

The device is always-on and needs to survive WiFi disconnects. CircuitPython doesn't give you a clean reconnect event — you discover the connection is dead when the next HTTP call throws. My workaround: catch the exception, set requests = None, detect None at the top of the fetch loop, reconnect, reinitialize the session. Works, but fragile.

Production reliability: debug flags and watchdog

One pattern I've carried across all my embedded projects: a 32-bit debug bitmask stored in persistent memory, and a hardware watchdog.

The bitmask lets me POST the device's health history to a monitoring endpoint. Each bit tracks a category of failure — WiFi connect failures, HTTP exceptions, NTP failures, NVM write failures. Even if the device reboots and the exception is gone, the bit stays set until explicitly cleared. When I look at the debug page later, I can see "WiFi connect failed 3+ times" even if the device is healthy now.

The watchdog is set to 60 minutes (this is an always-on device, not deep-sleep). It gets fed in the WiFi check, the METAR fetch, and the main loop. If anything hangs — which on CircuitPython occasionally happens with adafruit_requests on flaky connections — the device reboots rather than sitting stuck forever.

ESP-IDF rewrite

The CircuitPython workarounds accumulated enough that I started an ESP-IDF C rewrite. The motivating factors:

ESP-IDF WiFi is reliable in ways CircuitPython's isn't on this chip. No SmartPool, no requests = None dance, no lwIP DNS workarounds.
OTA updates. CircuitPython devices update by copying a file over USB. ESP-IDF devices can pull firmware from an HTTPS endpoint and self-update. For a device that sits on my desk, this matters less — but it's a pattern I use across all my ESP32 projects now.
Performance. The OLED rendering in CircuitPython is noticeably slow at higher refresh rates. In C it's instant.

The backend CSV format and airport list format stay identical. The behavior from outside — what the LEDs show, what the debug page returns — is the same. The rewrite is a platform swap, not a feature change.

What I'd build differently

The Lambda@Edge backend was the best decision I made early. Keeping the wire protocol simple (plain HTTP, CSV responses) meant the firmware stayed simple. Anytime the METAR API changed, I updated the Lambda — the device didn't know anything changed.

If I were starting over, I'd skip CircuitPython for a project that's always-on and WiFi-dependent. CircuitPython is excellent for sensors, USB gadgets, interactive prototyping. For a device that needs to run unattended for months on a network connection, ESP-IDF's networking stack is just more reliable.

The debug flag pattern is worth copying to any embedded project. Persistent error history — even across reboots — is invaluable when you're trying to figure out why a device that's been running for a week started acting up.

ESP-IDF 6.0: what changed, and whether to upgrade from 5.5

tedder — Sun, 12 Jul 2026 21:41:00 GMT

After running a uniform fleet of ESP32 devices on ESP-IDF v5.5.4 for months, I evaluated the jump to v6.0.2. The short answer: no urgency for existing projects, but the upgrade is clean and v6 is worth choosing for new work.

What got better in v6

Picolibc replaces Newlib — This is the headline win. Picolibc is smaller (flash), faster, and uses less RAM. Free win on all chips, especially memory-constrained S2 and original ESP32. See the Picolibc vs Newlib comparison.

New chips and matured C6 support — C5, C61 (preview), and full ESP32-C6 support. The C6 in particular is where v6 shines: Wi-Fi 6 (802.11ax) with TWT (Target Wake Time) for connected low-power operation, native 802.15.4 (Thread/Zigbee/Matter), and a genuine low-power core (not a tiny ULP helper like S2/S3 have).

Recovery bootloader (C5/C61 only) — Safe OTA of the bootloader itself, not just the app. If the bootloader write fails or loses power mid-flash, the chip boots a recovery copy from a dedicated partition instead of bricking. See ESP-IDF bootloader OTA docs.

Wi-Fi driver robustness — WPA3 mixed-mode AP (SoftAP serves WPA3 but still accepts WPA2-PSK clients), USD (peer discovery), general reconnect/roaming improvements.

Tooling — Installation Manager (EIM) for managing versions side-by-side, MCP server support, idf.py CLI extensions.

Active development — v5.5.x is trending maintenance-only. v6 is where new features land.

What breaks (or requires attention)

Warnings are now errors by default. This is the most likely migration friction. Each of your 13 projects may need warning cleanup on the first v6 build—either fix them or re-disable the flag. See the official migration guide.

Toolchain bumps — CMake 3.22.1 minimum, esp-idf-kconfig v3 (Kconfig syntax changes, only matters if you have custom Kconfig).

idf.py requires -p flag for eFuse commands. If you have automation that does idf.py burn_efuse or similar without specifying the port, it breaks.

Legacy drivers removed — ADC, DAC, I2S, Timer, PCNT, MCPWM, RMT, temp-sensor drivers are gone. I scanned my fleet; none use them (though RMT's esp_led_strip component is NOT removed, and driver/ledc.h for PWM is fine). Check your projects for raw driver/adc.h, driver/dac.h, etc.

Deep sleep GPIO wake on C6: no EXT0, and the wakeup-cause API changed. Hit this building real esp_deep_sleep() support for a battery doorbell button on C6. Two separate things:

esp_sleep_enable_ext0_wakeup() doesn't exist on your build if you're targeting C6 — it's compiled out behind #if SOC_PM_SUPPORT_EXT0_WAKEUP, and that capability is ESP32/S2/S3-only. C6 (and H2) only get esp_sleep_enable_ext1_wakeup_io(), the multi-pin bitmask API, even for a single GPIO: esp_sleep_enable_ext1_wakeup_io(1ULL << GPIO_NUM, ESP_EXT1_WAKEUP_ANY_LOW). Every ESP-IDF example and blog post written against classic ESP32 uses EXT0 for "wake on one pin" — none of that code compiles unmodified on C6, and the failure is a build error (implicit declaration of function), not a runtime surprise, so at least you find out immediately.
esp_sleep_get_wakeup_cause() (singular, returns one enum) is deprecated in v6 in favor of esp_sleep_get_wakeup_causes() (plural, returns a uint32_t bitmask — sleep can now report multiple simultaneous wake reasons). Old code still compiles, just warns. Fix is mechanical: causes & BIT(ESP_SLEEP_WAKEUP_EXT1) instead of cause == ESP_SLEEP_WAKEUP_EXT1. IDF's own examples/system/deep_sleep_wake_stub already uses the new form if you want a reference.

Picolibc differences — The libc swap can surface subtle printf, locale, float-formatting, or stdio differences. Low-risk, but test rather than assume.

Effort — Multiplied by the number of projects. Re-running idf.py set-target, rebuilding, re-OTA'ing 13 devices takes time. Not a blocker, just real.

My migration strategy

Staged trial: install v6.0.2 side-by-side with v5.5.4. Start with one low-stakes project (e.g., a development board or a device you can easily recover), clear warnings-as-errors fallout, confirm OTA works, then roll the rest opportunistically.

Keep v5.5.4 until all projects are confirmed running on v6.0.2. That gives a fallback if something unexpected surfaces.

For new projects, start directly on v6.0.2. No point dragging v5.5.4 into new work.

Bootloader OTA: the chip-specific caveat

If you're planning safe OTA of the bootloader itself (not just the app), watch chip support:

C5 / C61: Full recovery bootloader with eFuse fallback. Safe. Use it.
C6, S2, S3: No recovery bootloader. Only app OTA is safe. Bootloader must be flashed once over USB and left alone—OTA-ing the bootloader on these chips is risky (no fallback if the write fails).

C6 practical rule: App OTA + rollback = safe/supported. Bootloader OTA = still risky. If bullet-proof bootloader OTA is a hard requirement, prefer C5 or C61, not C6.

C6 vs S2 vs S3 at a glance

For battery devices that need to stay Wi-Fi-connected on battery, the C6 is a different animal from S2/S3.

ESP32-S2

Cores: 1× Xtensa LX7 @ 240 MHz
Low-power: ULP-RISC-V (tiny helper, limited fixed functions)
WiFi: Wi-Fi 4 (802.11n)
BLE: None
Connected power: ~mA (frequent DTIM wakeups)

ESP32-S3

Cores: 2× Xtensa LX7 @ 240 MHz
Low-power: ULP-RISC-V (same as S2)
WiFi: Wi-Fi 4 (802.11n)
BLE: BLE 5
Connected power: ~mA (frequent DTIM wakeups)

ESP32-C6

Cores: 1× RISC-V HP @ 160 MHz
Low-power: Real LP core (full RISC-V with LP UART/I2C/ADC/GPIO)
WiFi: Wi-Fi 6 (802.11ax)
BLE: BLE 5
802.15.4: Yes (Thread/Zigbee/Matter)
Connected power: ~µA with TWT

Key differences:

TWT (Target Wake Time) is a Wi-Fi 6 feature (C6-only). The chip negotiates sleep windows with the AP while staying associated — can cut average current ~10× vs S2/S3 which need frequent DTIM wakeups.

LP core means real sensor/comms loops can run on the low-power core while the HP core sleeps — not a "tiny helper" like S2/S3 ULP, but an actual programmable MCU with peripherals.

What I'm doing

Existing fleet (13 projects on v5.5.4): Staying put unless a specific reason to upgrade surfaces (bug fix, new feature, hardware issue). Low urgency.

New work: C6 on v6.0.2 from the start, taking advantage of TWT, LP core, and the full v6 tooling.

Test upgrade (Q3 2026): Pick one S3 project (attic-temperature-sensor or rain-sensor), do the staged trial, confirm OTA, then roll out to the rest if no surprises.

The migration is not risky — it's just not urgent for devices already working on v5.5.4.

Automating Nikola blog deployments with GitHub Actions

tedder — Sun, 12 Jul 2026 21:40:00 GMT

Building a Nikola static site is fast. Pushing to S3 is fast. But running them manually every time you publish a post is tedious. GitHub Actions automates both in a few minutes.

The setup we built

A workflow that: 1. Listens for pushes to main 2. Installs Nikola and dependencies 3. Builds the entire site 4. Pushes the output to S3

Every post goes live the moment you push.

Create limited AWS credentials

First, create an IAM user with S3-only access to your bucket. This follows the principle of least privilege:

aws iam create-user --user-name blog-deploy-user
aws iam put-user-policy --user-name blog-deploy-user \
  --policy-name s3-bucket-deploy \
  --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket",
        "arn:aws:s3:::your-bucket/*"
      ]
    }
  ]
}'
aws iam create-access-key --user-name blog-deploy-user

Save those credentials. You'll need them in a moment.

Add secrets to GitHub

In your repository settings, go to Secrets and variables → Actions.

Add the following secrets:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

The workflow file

Create .github/workflows/deploy.yml:

name: Build and Deploy Blog

on:
  push:
    branches:
      - main

jobs:
  build_and_deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install Nikola
        run: pip install 'nikola[extras]' watchdog aiohttp

      - name: Build blog
        run: nikola build

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-west-2

      - name: Deploy to S3
        run: |
          aws s3 sync output/ s3://your-bucket/ \
            --delete \
            --exclude "*.md" \
            --exclude ".git/*"

The --delete flag removes files from S3 that aren't in your build output (useful when renaming posts). The --exclude flags skip markdown source and git files.

A note on the official Nikola action

The getnikola/nikola-action is designed for GitHub Pages deployments (github_deploy), not S3. If you're using S3 (or another deployment method), a custom workflow like this is simpler and more transparent.

The action also had a subtle issue: running inside a Docker container, it couldn't find the repository due to git's safe.directory check (a security feature added in Git 2.36). The workaround was to create .gitconfig in the runner's home directory before the container ran, but at that point, using a direct Python install was cleaner. (See the related runner issue for details.)

Next steps

Customize the S3 sync command for your bucket and path
Add CloudFront cache invalidation if you use CloudFront as a CDN
Add Slack or Discord notifications on deployment success or failure using GitHub Actions workflows
Consider --cache-control headers to bust caches on HTML files (e.g., aws s3 cp ... --cache-control "max-age=300")

Deploy from the comfort of your editor. No manual builds, no SSH tunnels.

ESP-IDF code patterns: OTA, diagnostics, and power management

tedder — Tue, 07 Jul 2026 15:37:00 GMT

You've set up your project structure and partition table. Now what? The next layer is the actual firmware code — the patterns that keep your device running reliably. I've learned these through deployed devices in the field: solar-powered sensors that need to run for months, OTA updates that can brick a device if they stall, and the peculiar debugging nightmare of a device you can't reach without logging in to a VPN jump host.

OTA update strategies: when to check and how to download

OTA updates are great until they're not. The first time a firmware update stalls halfway through (on a slow cellular connection, say) and the device hangs forever because it can't finish downloading, you'll understand why this section exists.

Start OTA checks with a delay after boot. Don't check on startup. If the firmware you just flashed is bad, you'll immediately try to update back to the broken version, and your device is toast. Instead, use a delay of at least a few minutes:

#define OTA_INITIAL_DELAY_MS (5 * 60 * 1000)  // 5 minutes

void app_main() {
    // ... initialize, WiFi, etc ...

    // Don't check OTA immediately; let the device stabilize
    vTaskDelay(OTA_INITIAL_DELAY_MS / portTICK_PERIOD_MS);
    xTaskCreate(ota_task, "ota_task", 4096, NULL, 5, NULL);
}

This gives you a window to manually reflash if the new firmware is broken. On OTA success, the bootloader sets the new partition as active on next boot, so if something is wrong you have a chance to stop it.

Space out OTA checks to hours, not minutes. Polling every 5 minutes wastes power and network bandwidth for no benefit. Once every 4-6 hours is reasonable for most devices. If you're on solar or battery, make it longer:

#define OTA_CHECK_INTERVAL_MS (4 * 60 * 60 * 1000)  // 4 hours

void ota_task(void *pvParameters) {
    TickType_t last_check = xTaskGetTickCount();

    while (1) {
        TickType_t now = xTaskGetTickCount();
        if ((now - last_check) > pdMS_TO_TICKS(OTA_CHECK_INTERVAL_MS)) {
            perform_ota_update();
            last_check = now;
        }
        vTaskDelay(pdMS_TO_TICKS(60000));  // Check the clock once per minute
    }
}

Download in chunks and call esp_task_wdt_reset() per chunk. This is the hard-won lesson. If you use the ESP-IDF's esp_https_ota_perform() in blocking mode (which is easy), it will sit in one function call for the entire download. On a slow connection, that's minutes. The watchdog timer gets angry and reboots your device mid-update.

Instead, use the chunked OTA API:

#include "esp_https_ota.h"

esp_err_t ota_with_watchdog_reset(const char *url) {
    esp_https_ota_config_t config = {
        .http_config = &http_config,
    };

    esp_https_ota_handle_t https_ota_handle = NULL;
    esp_err_t err = esp_https_ota_begin(&config, &https_ota_handle);

    if (err != ESP_OK) {
        return err;
    }

    while (1) {
        err = esp_https_ota_perform(https_ota_handle);

        if (err == ESP_ERR_HTTPS_OTA_IN_PROGRESS) {
            // Reset watchdog on each chunk
            esp_task_wdt_reset();
            continue;
        } else {
            break;
        }
    }

    if (esp_https_ota_is_complete_data_received(https_ota_handle)) {
        err = esp_https_ota_finish(https_ota_handle);
    } else {
        esp_https_ota_abort(https_ota_handle);
    }

    return err;
}

Yes, you'll need to tune the watchdog timeout itself, but resetting per chunk is the key. Without it, slow downloads fail mysteriously.

Defensive programming: tracking reset reasons and device state

If a device reboots out in the field, you need to know why. Logging doesn't help if you can't reach the serial console. Build diagnostics into the firmware itself.

Capture the reset reason on every boot:

#include "esp_system.h"
#include "nvs_flash.h"

typedef struct {
    uint32_t reset_count;
    uint32_t last_reset_reason;
    uint32_t last_ota_attempt_ms;
    uint32_t last_data_post_ms;
    uint32_t wifi_fail_count;
} device_state_t;

void track_reset_reason() {
    esp_reset_reason_t reason = esp_reset_reason();

    device_state_t state = {};
    nvs_handle_t nvs_handle;
    nvs_open("device", NVS_READWRITE, &nvs_handle);
    nvs_get_blob(nvs_handle, "state", &state, sizeof(device_state_t));

    state.reset_count++;
    state.last_reset_reason = (uint32_t)reason;

    nvs_set_blob(nvs_handle, "state", &state, sizeof(device_state_t));
    nvs_commit(nvs_handle);
    nvs_close(nvs_handle);

    ESP_LOGI(TAG, "Boot #%lu, last reset reason: %d", 
             state.reset_count, state.last_reset_reason);
}

Implement a safe-mode escape hatch. If your device has rebooted 10 times in the last 5 minutes, something is very wrong. Entering a safe mode that doesn't run the main logic prevents a reboot loop from destroying data or spinning through CPU cycles:

#define UNSAFE_REBOOT_THRESHOLD 10
#define REBOOT_WINDOW_MS (5 * 60 * 1000)

void check_for_reboot_loop(device_state_t *state) {
    uint32_t time_since_reset = esp_timer_get_time() / 1000;

    // If we've rebooted many times in a short window, enter safe mode
    if (state->reset_count > UNSAFE_REBOOT_THRESHOLD &&
        time_since_reset < REBOOT_WINDOW_MS) {

        ESP_LOGW(TAG, "Too many reboots! Entering safe mode.");

        // Don't run main logic; just sit in a loop and wait for OTA
        while (1) {
            vTaskDelay(pdMS_TO_TICKS(1000));
        }
    }
}

Expose device state via HTTP. Add a simple debug endpoint that returns JSON with diagnostic info. You can query it without serial access:

esp_err_t debug_handler(httpd_req_t *req) {
    device_state_t state = {};
    nvs_handle_t nvs_handle;
    nvs_open("device", NVS_READONLY, &nvs_handle);
    nvs_get_blob(nvs_handle, "state", &state, sizeof(device_state_t));
    nvs_close(nvs_handle);

    cJSON *root = cJSON_CreateObject();
    cJSON_AddNumberToObject(root, "reset_count", state.reset_count);
    cJSON_AddNumberToObject(root, "last_reset_reason", state.last_reset_reason);
    cJSON_AddNumberToObject(root, "wifi_fail_count", state.wifi_fail_count);
    cJSON_AddNumberToObject(root, "uptime_ms", esp_timer_get_time() / 1000);

    char *json_str = cJSON_Print(root);
    httpd_resp_set_type(req, "application/json");
    httpd_resp_sendstr(req, json_str);

    cJSON_Delete(root);
    free(json_str);

    return ESP_OK;
}

Then register it during HTTP server setup. Now you can SSH into a jump host, curl your device, and get a full picture without touching a serial cable.

NVS (NVRAM) diagnostics: what to track and why

Not every value is worth persisting, but the right ones save you hours of debugging. Here's what I track:

Reset count: how many times has this device rebooted?
Last OTA attempt timestamp: did it try to update recently? (Stuck devices often have an old timestamp here.)
Last successful data-post timestamp: when was the last time this device actually transmitted data?
WiFi connect failure count: is WiFi flaky?

typedef struct {
    uint32_t reset_count;
    uint32_t last_reset_reason;
    uint32_t last_ota_attempt_ms;      // Milliseconds since epoch
    uint32_t last_data_post_ms;        // Milliseconds since epoch
    uint32_t wifi_fail_count;
} device_state_t;

void update_ota_timestamp() {
    device_state_t state = {};
    nvs_handle_t nvs_handle;
    nvs_open("device", NVS_READWRITE, &nvs_handle);
    nvs_get_blob(nvs_handle, "state", &state, sizeof(device_state_t));

    state.last_ota_attempt_ms = (uint32_t)(esp_timer_get_time() / 1000);

    nvs_set_blob(nvs_handle, "state", &state, sizeof(device_state_t));
    nvs_commit(nvs_handle);
    nvs_close(nvs_handle);
}

void update_data_post_timestamp() {
    device_state_t state = {};
    nvs_handle_t nvs_handle;
    nvs_open("device", NVS_READWRITE, &nvs_handle);
    nvs_get_blob(nvs_handle, "state", &state, sizeof(device_state_t));

    state.last_data_post_ms = (uint32_t)(esp_timer_get_time() / 1000);

    nvs_set_blob(nvs_handle, "state", &state, sizeof(device_state_t));
    nvs_commit(nvs_handle);
    nvs_close(nvs_handle);
}

Now your /debug endpoint tells you: - "This device last sent data 3 hours ago" → likely a WiFi issue - "Reset reason is POWERON, reset count is 50, OTA timestamp is from this morning" → firmware is stable but something else keeps power-cycling the board - "WiFi fail count is 200 but uptime is only 2 hours" → authentication problem

Power savings for solar+battery devices

If your ESP32 is sitting outside powered by a solar panel and a battery, every milliamp-hour matters. Deep sleep is your friend, but there are sneaky current draws you need to minimize.

Use deep sleep liberally. If your device only needs to transmit once per hour, sleep for 59 minutes and wake for 1 minute:

#define SLEEP_TIME_US (59 * 60 * 1000000)  // 59 minutes

void app_main() {
    // ... initialize, WiFi, transmit data, update OTA ...

    // Deep sleep consumes ~10 µA (vs ~80 mA active)
    esp_deep_sleep(SLEEP_TIME_US);
}

Minimize WiFi active time. WiFi startup takes a second or two and draws ~200 mA. Shave every millisecond:

Use a static IP instead of DHCP. DHCP adds 100-500 ms to startup, depending on your router.
Use WPA2_PSK, not mixed WPA2/WPA3. WPA2 negotiation is faster.
Connect to the closest/strongest AP to minimize retries.

wifi_config_t wifi_config = {
    .sta = {
        .ssid = (uint8_t *)CONFIG_WIFI_SSID,
        .password = (uint8_t *)CONFIG_WIFI_PASSWORD,
        .threshold.authmode = WIFI_AUTH_WPA2_PSK,
        .pmf_cfg = {
            .capable = true,
            .required = false,
        },
    },
};

// Use static IP
tcpip_adapter_ip_info_t ip_info = {
    .ip = { .addr = ipaddr_addr("192.168.0.100") },
    .netmask = { .addr = ipaddr_addr("255.255.255.0") },
    .gw = { .addr = ipaddr_addr("192.168.0.1") },
};
tcpip_adapter_set_ip_info(TCPIP_ADAPTER_STA, &ip_info);

(Substitute your actual IP range, of course.)

Consider ESP-NOW for ultra-low-power sensors. If you have a receiver (another ESP32, always-on gateway, etc.) nearby, ESP-NOW uses 150-200 ms of active time versus 2+ seconds for WiFi. The tradeoff is that you need infrastructure:

void send_via_espnow(uint8_t *payload, size_t len) {
    // ESP-NOW transmit: ~100-150 ms active time, much less power
    // Requires a paired receiver
    esp_now_send(broadcast_mac, payload, len);

    // Deep sleep immediately after
    esp_deep_sleep(SLEEP_TIME_US);
}

Budget your power envelope. If you have a 500 mAh battery and 500 mW solar charging on a cloudy day, you can sustain about 5 mAh per hour (rough math). If each transmit-and-OTA-check cycle takes 500 mA for 10 seconds, that's 1.4 mAh. Do that once per hour and you're breaking even or losing charge. Do it twice per hour on a cloudy day and your battery drains. Pick your sleep interval based on what your solar panel can actually charge:

#define SLEEP_INTERVAL_MS (2 * 60 * 60 * 1000)  // 2 hours for marginal solar
// vs.
#define SLEEP_INTERVAL_MS (30 * 60 * 1000)      // 30 min for good sun/large battery

Test in the actual sun conditions you'll deploy in. A device that runs fine in your garage with good WiFi will fail miserably in the field.

These patterns have saved me from devices that reboot in loops, OTA updates that hang forever, and batteries that drain in a week instead of a month. They're not ESP-IDF-specific — most of these apply to any embedded system — but they're the things I reach for in every firmware project now. The defensive diagnostics pay for themselves the first time you need to debug a device you can't physically reach.

ESP-IDF project standards and conventions

tedder — Sun, 07 Jun 2026 17:23:00 GMT

I've built a bunch of ESP32 projects using Espressif's ESP-IDF framework, and every time I start a new one, I second-guess myself on basic structure: where does the config go? How do I version the firmware? What partition layout should I use? This post is documentation-to-self, but hopefully it saves someone else from the iterative debugging I've done.

Project structure and build configuration

The ESP-IDF follows a pretty strict layout. At the top level you'll have:

main/: your application code (main.c is the entry point)
CMakeLists.txt: project-level build configuration
sdkconfig.defaults: sensible defaults for menuconfig options
partitions.csv: partition table (more on that below)
.clangd: language server config to fix false positives on macOS

The main/CMakeLists.txt is minimal:

idf_component_register(SRCS "main.c" 
                       INCLUDE_DIRS ".")

Keep components in main/ until you have more than a few files, then break them into components/mycomponent/.

OTA update patterns: ota_0 and ota_1

Never use the factory+ota_0 partition layout. I learned this the hard way when I bricked a device. Use ota_0 and ota_1 instead, so you can always fall back to the previous firmware if an update goes wrong.

Your partitions.csv should look like:

nvs,      data, nvs,     0x9000, 0x6000,
otadata,  data, ota,     0xf000, 0x2000,
ota_0,    app,  ota_0,   0x20000, 0x1C0000,
ota_1,    app,  ota_1,   0x1E0000, 0x1C0000,
spiffs,   data, spiffs,  0x3A0000, 0x60000,

The sizes here depend on your flash size (I'm assuming 4MB). The key is that otadata tracks which OTA partition is active, and the bootloader automatically switches between them. On a bad update, it rolls back to the previous partition on next boot.

I allocate 1.75 MB (0x1C0000) per OTA partition because my firmware is usually 600-800 KB, and that leaves plenty of headroom for growth without requiring a reflash to resize.

Firmware versioning

Every binary that gets flashed or pushed to S3 needs a version bump. Use a simple semver string:

// main/main.c
#define FIRMWARE_VERSION "1.2.3"
#define FIRMWARE_VERSION_MAJOR 1
#define FIRMWARE_VERSION_MINOR 2
#define FIRMWARE_VERSION_PATCH 3

Then in your OTA or setup code, log it:

ESP_LOGI(TAG, "Firmware version: %s", FIRMWARE_VERSION);

This sounds obvious, but I've spent frustrating hours trying to reproduce a bug and realizing the device is running an older build than I thought. A simple version string at startup saves that pain.

If you're pushing to S3 for OTA (see below), include the version in the build artifact name: myproject-v1.2.3.bin.

FIRMWARE_VERSION and esp_app_desc_t.version are not the same thing — you must keep both in sync. The #define you log at startup is just a C string. The version embedded in the binary's esp_app_desc_t struct — which is what esp_app_get_description()->version returns, and what OTA same-version skip logic compares — comes from CMake's PROJECT_VER at build time, not your #define.

If you use same-version skip in your OTA code (and you should), this distinction will bite you: if PROJECT_VER is not set, ESP-IDF embeds a fallback string like "1" or "1.2" in every single build. Your OTA check compares "1.2" on the device against "1.2" in the downloaded image, sees a match, and skips — every time, forever, no matter what you change.

The fix: enable CONFIG_APP_PROJECT_VER_FROM_CONFIG and set CONFIG_APP_PROJECT_VER to match your FIRMWARE_VERSION. You must update this in both sdkconfig.defaults AND sdkconfig directly — defaults only apply to keys not already present in sdkconfig, so if the key exists (even disabled), idf.py reconfigure will not override it.

# sdkconfig.defaults
CONFIG_APP_PROJECT_VER_FROM_CONFIG=y
CONFIG_APP_PROJECT_VER="1.2.3"

And in sdkconfig, find the existing # CONFIG_APP_PROJECT_VER_FROM_CONFIG is not set line and replace it:

CONFIG_APP_PROJECT_VER_FROM_CONFIG=y
CONFIG_APP_PROJECT_VER="1.2.3"

Every firmware bump requires updating three places: FIRMWARE_VERSION in main.c, CONFIG_APP_PROJECT_VER in sdkconfig.defaults, and CONFIG_APP_PROJECT_VER in sdkconfig. Missing any one of them means either your logs lie, or your OTA never runs.

WiFi authentication: stick with WPA2_PSK

Always use WIFI_AUTH_WPA2_PSK, not WIFI_AUTH_WPA2_WPA3_PSK. The mixed mode causes mysterious auth failures on home routers, especially older ones or devices that don't advertise both protocols equally.

wifi_config_t wifi_config = {
    .sta = {
        .ssid = (uint8_t *)CONFIG_WIFI_SSID,
        .password = (uint8_t *)CONFIG_WIFI_PASSWORD,
        .threshold.authmode = WIFI_AUTH_WPA2_PSK,
    },
};

Even if your home network supports WPA3, the mixed mode is a footgun. If you're deploying to multiple locations or devices with varying WiFi hardware, stick with WPA2_PSK. It's universally supported and won't surprise you.

Partition sizing and resizing pain

Leave expansion room in your OTA partitions. I allocate 1.75 MB when my current firmware is 600 KB because resizing partitions later requires a USB cable and esptool.py flash_id followed by a full reflash. It's doable but annoying.

If your project grows and you need more space, you have two options: 1. Resize partitions (USB reflash required) 2. Move to a larger flash chip (harder)

Allocate generously at the start. A few MB of unused space costs nothing.

Clangd/LSP false positives on macOS

If you're using clangd or VSCode's C/C++ extension on macOS, you'll see tons of red squiggles for ESP-IDF types and macros that actually compile fine. This is because clangd doesn't know about the build configuration.

Create a .clangd file at your project root:

CompileFlags:
  CompilationDatabase: build/compile_commands.json

Build once (so compile_commands.json exists), then clangd will use it and the false positives disappear. This is especially important for understanding macro expansions and avoiding frustration while editing.

S3 OTA uploads: the cache-control trap

If you're pushing built binaries to S3 for OTA updates, always include --cache-control "max-age=30" when uploading:

AWS_PROFILE=yourprofile aws s3 cp build/myproject.bin s3://your-bucket/firmware/ \
  --cache-control "max-age=30"

Without this, CloudFront (or any other CDN in front of S3) will serve stale versions for up to 1 hour, and your devices will pull the old firmware. I've spent hours debugging "why is the new firmware not deployed?" only to realize the CDN was serving the previous build.

Set max-age to something short (30 seconds works) so you get fresh binaries on each request, not "fresh every hour."

Battery-powered sensors: ESP-NOW vs WiFi

If you're building a battery-powered sensor, ESP-NOW uses far less power than WiFi, but requires a receiver that's always on (or close to it). Tradeoffs:

WiFi: - Higher latency and power draw (radio startup is expensive) - Works everywhere without custom infrastructure - Built-in over-the-air update support

ESP-NOW: - One-way or paired communication only; you need a gateway/receiver - Extremely low power if you can wake, transmit, and sleep in milliseconds - Mesh capability with enough peers

For a humidity sensor that transmits once per hour, ESP-NOW + a gateway is overkill. For a door sensor that needs to wake and transmit immediately, ESP-NOW is worth the complexity. For anything in between, build WiFi first and optimize to ESP-NOW later if battery life becomes critical.

That's the foundation. Every new project I start, I copy the partition table, firmware version pattern, and build wrapper from the last one, and I've saved myself countless hours of "wait, why is this partition 512K?" debugging. Hopefully this saves you from the same mistakes.

Fixing Elasticsearch/Logstash/ELK's DATESTAMP grok pattern

tedder — Fri, 15 May 2020 18:22:41 GMT

Elasticsearch, including Logstash and Pipeline Processors, love to use grok patterns. These are basically named regex patterns, allowing the complexity to be hidden behind easier-to-read labels (though they do require referring to the source). Regexes are great, with the "now you have two problems" caveat of any sufficiently advanced technology. (it could be worse, you could have 100 problems)

What's the problem? Timestamp support. Such a trivial issue has been a problem for a long time. I'll show a fix (rather, a workaround) below.

The problem: year-first timestamps

Many datestamps in logs are in a year-first format (e.g., 2020-01-01). That makes sense, as many operating systems and languages default to ISO 8601 for a human-readable datetime format. For instance, here's a recent example from my system's dpkg.log:

2020-05-21 06:01:01 upgrade tzdata:all 2019c-3ubuntu1 2020a-0ubuntu0.20.04

Or from a Mac's log:

2020-05-24 20:04:25-07 ted-macbook-pro softwareupdated[753]: Removing client SUUpdateServiceClient pid=5347, uid=0, installAuth=NO rights=(), transactions=0 (/usr/sbin/softwareupdate)

Or from Octoprint's python-based logs:

2020-05-25 03:26:36,643 - octoprint.server.heartbeat - INFO - Server heartbeat <3

There are other common variants, like the 'T' delimiting the date and time sections, time zones, or other delimiters- I will also be addressing this format, found in the grok comment:

2020/05/25-16:02:11.5533

I'm less concerned about time zones, since real computers run on UTC.

What's wrong with these? none of them are supported in vanilla grok in any version.

Digging into the problem

Let's look at logstash in 2015. Or logstash in 2020. Or elasticsearch in 2020.

They define UK-format and US-format dates:

DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}

And in a comment, they suggest that the datestamp will be accepted with slashes:

# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)

Which leads to an implication that the DATESTAMP would support it:

DATE %{DATE_US}|%{DATE_EU}
DATESTAMP %{DATE}[- ]%{TIME}

But.. look back to the US/EU formats. No year-first format. Sometimes you'll see a weird match, like "20 April 2001", but it's just seeing "2020/04/01", slicing off the first few digits, and parsing it as a date-first string. Weird, huh? This explains some of the weird indexes you might find in an ELK stack, where there's something like logstash-2001.04.01 and it's almost 20 years later.

Anyhow, on to...

The easy fix

If you have the luxury of redefining DATE, you can prepend a sane format to it:

DATE_YEARFIRST %{YEAR}[\/\-\s]%{MONTHNUM}[\/\-\s]%{MONTHDAY}
DATE %{DATE_YEARFIRST}|%{DATE_US}|%{DATE_EU}

Why prepend? That keeps it from accidentally matching 2020 as above.

The hard fix

But if you don't have the luxury of updating your grok patterns, you'll have to whip up a custom one:

(?<my_datetime_field>%{YEAR}/%{MONTHNUM}/%{MONTHDAY}[T\s\-]+%{TIME})

You'll notice I've simplified the delimiters from above. It's just less readable o accept them all, but you might need to do so:

(?<my_datetime_field>%{YEAR}[\/\-\s]%{MONTHNUM}[\/\-\s]%{MONTHDAY}[T\s\-]+%{TIME})

So, there you go. That can easily be stuffed into Logstash, or a processor. Hooray!

Caveats

Now, there is an 8601-style year-first pattern defined, so, great if it matches your format. It didn't match enough of my variants:

TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{ISO8601_HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

There was also a year-first version added to logstash in 2016, then the day and month were flipped, then it was removed or never made it to master. Hilariously, it was typoed as 8061 the whole time. It also didn't exist in elasticsearch, only logstash. It doesn't help that the Elasticsearch version of the file was moved in 2016, then also moved in 2018, which took away the easy-to-view commit history.

Why not fix it?

Here's an issue from 2015. Here's another one. Here's a PR from 2016. The issue isn't submitting a PR, obviously. Don't get me started on discuss.elastic.co, which auto-closes and locks discussions, meaning you're pretty much guaranteed to find an out-of-date, inferior, solution. If any at all.

Gavin Belson Signature Edition liquid cooled server

tedder — Sat, 04 Apr 2020 18:51:29 GMT

I've been slowly collecting bits to build a liquid-cooled server. I have no practical reason to do so, but I've been intrigued by the concept. Finally I bit the bullet and got the following:

I then ordered a bunch of parts from EKWB:

PowerColor Liquid Devil RX5700 XT
CPU/motherboard full block
X570 chipset block (the stock Aorus fan/block isn't compatible with the GPU)

Everything else was pretty normal to build a PC/pseudoserver. It took me quite a while to put this together- not only did I have to wait for two batches of parts from EKWB in Slovenia, plus deal with a customs delay, but I also needed to fabricate a bunch of parts.

The first thing was the radiator. I designed two "sliders" to hold it in, since the width of the radiator is close to the case width. It took a lot of iterations to deal with bolt holes, drain cap of the radiator, and so on.

I then designed a block to hold the pump. It only took a couple of tries on it. I was happy to put a bit of a "lean" on the side bolt holes, which leaves room for a rounded corner on the inside of the case.

I installed the motherboard and then tried adding the GPU and the water cooling. They both required disassembling the motherboard from its mounting plate. So many screws! The GPU was even more work when I realized the X570 chipset heatsink/fan were in the way. I first took it off and thought about cutting it, but it's a big chonky piece that would have been really difficult.

The next GPU problem is the Rosewill case doesn't have expansion slots in the back. So weird! Obviously it's designed for a motherboard to be placed there, but I guess they assume nobody will need to add a card to the motherboard. So I Dremeled out the back of the case. I might design and print something to secure the card in place.

To add to my list of GPU struggles, the liquid cooling makes the GPU taller than a 4U case. I took some measurements, carefully dremeled a hole, then designed a plexiglass popemobile window. I might add a second window or enlarge that one to show the cooling and RGBs, but that is optional and would come later.

I'm unhappy with a couple of things:

the motherboard only seems to have two fan headers that are usable from Linux. One has the pump, the other has puller fans. I have a few other rows of fans, so I either need to daisy them all together or control them from off the motherboard.
the RGB LED headers aren't controllable.
there are no ports for off-motherboard thermistors. I put four inline thermistors on the cooling loop but I have nowhere to plug them in.
It's hard to know if the pump is running. It sends RPM back, but there's no other real verification. I haven't solved this yet, but I will probably stick a flow indicator in the loop, then 3D print an optical sensor to look through it.
The cabling mess. I tried heat-shrinking the fan cables, but that made things worse, partly because it's thick heatshrink that glued everything together, so I can't remove a fan from the shrink. There's a lot of cabling coming from the PSU too. I'm delaying solving this until everything is done.

I have a lead on solving the first three things- I've designed a board around the ESP32 with three fan headers and two DRGB headers. If that works I'll add at least five inputs for the thermistors and flow control. That way I can use the DRGBs to display temperature at a glance, plus control the fans and gather/transmit all the important data back to my network.

What am I using it for? I have a couple things in mind. First, it sits in my Kubernetes cluster, and the GPU will let me transcode my security videos better. I thought I'd dabble with some other GPU computing, but there aren't great drivers or support for non-Nvidia cards :/. As a final use, if I can figure out how, it'd make for a nice Steam Remote gaming system, if I can figure out how to do it with a non-Windows and container-based system.

The crowning touch was to add the 3D-printed replica of the Gavin Belson signature to the case. It looks perfect!

Deleting (almost) all of my tweets

tedder — Mon, 03 Feb 2020 01:18:56 GMT

I decided a few months ago that my 10+ years of Twitter history wasn't helpful. It had more chance to be a liability than anything. So I finally wrote a script to delete it.

I deploy it locally as a Kubernetes cron, so all of the secrets are stored elsewhere. That means I've shared the repo:

https://github.com/tedder/tweet-cleanup

It isn't perfect (for instance, dealing with pinned tweets), it's still hardcoded to my username, but I'd love PRs.

Far Side comic newsfeed (RSS feed)

tedder — Mon, 03 Feb 2020 00:42:27 GMT

In late 2019, the archives of Gary Larson's awesome The Far Side comic officially came back to the web. It's fun to see them, not just the 10 that needs more JPEG. Two 'daily' comics are posted per day, which is cool.

Unfortunately, there's no newsfeed. So, I fixed that. It uses the modern JSON feed spec. The feed URL is: https://dyn.tedder.me/rss/farside/daily.json

If you need an older style feed, here's an Atom XML feed: https://dyn.tedder.me/rss/farside/daily.xml

You can see a sample of it being used on Newsblur (even if you don't have a login).

Robocalling in 12 minutes*

tedder — Thu, 14 Mar 2019 15:07:53 GMT

On John Oliver's Last Week Tonight, he showed how they were robocalling the FCC commissioners. As an offhanded comment he said their tech person literally got it running in less than 15 minutes. I took that as a challenge, and based on a comment from Kirsten, my goal was to get audio of our Audreycat growling when she answered the phone.

I was confident because I knew I had an account ready to go at Twilio. They have developer-friendly APIs and I had previous experience of managing over 10k phone numbers on their platform, though I hadn't done it recently.

So, I bought a phone number from an area code she'd recognize and cobbled together the code from their sample:

from twilio.rest import Client
k = "+15035551212"
phone="+13095551212"
acct_sid="xx"
acct_tok="yy"

client = Client(acct_sid,acct_tok)
call = client.calls.create(
                        url='https://tedder.me/twilio/hi3.xml',
                        to=k,
                        from_=phone
                    )

print(call.sid)

And here's the XML, which tells Twilio what to do after the phone is picked up, which I put on S3:

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Say voice="alice">hi kirsten</Say>
    <Play>https://tedder.me/twilio/audrey.mp3</Play>
</Response>

Sure enough, I had her phone ringing 12 minutes later. She picked up, and a robot voice told her there was an error. What?!

So, I did it in 12 minutes, but it took another 13 minutes to figure out what was wrong. Twilio's error console said it was returning a 403 error, which didn't make sense, as I could access the content just fine.

With some further digging I found that the XML URL is hit with POST, so I knew the problem- the content was being served from S3, which only expects a GET. So I had to dig further to find all of the parameters to the Twilio code. It was actually really hard to find good documentation for the python library, as everything pointed to the basics of using the library. Finally I found the entrypoint to the documentation, and after a lot of work I got to the calls.create documentation. It was difficult to find but the answer was easy- I needed to add a parameter called method:

call = client.calls.create(
                        method='GET',
                        url='https://tedder.me/twilio/hi3.xml',
                        to=k,
                        from_=phone
                    )

After doing that it was easy. I can't say it was done in 15 minutes like John Oliver's tech dude, but I still suspect his wasn't fully working after 15 minutes either.