It’s got rhythm…

After a lot of debugging (and swearing) and very hairy math, I have my version of the NTP libraries mostly keeping time consistently. It does the full offset and delay calculations. There are still a few quirks that I need to track down, but the skew is far more predictable now–though still larger than I would like. In 10 seconds, the clock drifts between -4000 and -14000 microseconds (about 1000 ppm, which is pretty high). With a little adaptive frequency shifting I think I can get that drift down quite a bit.

I have starting thinking about submitting a pull request to merge my NTPClient changes back into the main library, but I committed a cardinal sin. I modified an older version of the code rather than starting with the latest. My attempts to reconcile their changes (mostly making all the getters idempotent) and mine (changing the fundamental time unit from seconds to microseconds while not breaking the API) have proven very clunky, and I’ve given up for the moment. I’m thinking of heading in the opposite direction and just forking my effort into a whole new refactored library with a more elegant way to handle complex time calculations, similar to the DateTime class in the RTClib, but more flexible.

What time is it, really?

A big part of the success of this project will hinge on getting all the microphones to agree on what time it is. I don’t really care if that time is particularly accurate, as long as all the sensors agree on it. Of course, they’ll never agree exactly, but if I can get them in agreement to within 50-100 microseconds, that should be good enough.

I’ve starting experimenting with the Network Time Protocol (NTP). In theory, this should be satisfactory for what I’m doing. However, the implementation I’ve found for Arduino based systems is pretty basic, and doesn’t implement the full protocol. First, it only exposes the whole number of seconds, and doesn’t expose the fractional part of the seconds. I’ve modified it to expose down to microseconds. Second, it simply takes t2 of the clock synchronization algorithm. It doesn’t do the full offset and round trip delay calculation. This is good enough if all you care about is rough accuracy (+/- 1 second), but apparently isn’t sufficient for what I need. But even if I extend the library further I may not be able to achieve the accuracy I need.

I have my devices syncing to the NTP server on my home box every 10 seconds or so, and I’ve had them log the amount of skew (change in current time) they are seeing every refresh. Over long stretches, it averages about 5,000 microseconds fast per 10 seconds of clock time–which is pretty good and easily compensated for, but individual skews are all over the place. It jumps back and forth by 5,000-120,000 microseconds (5-120 miliseconds) at a time. Keep in mind that with a 20 meter baseline, the delta time is at most 52ms, and usually much less.

What I’m seeing in the logs

This might partly be due to the differing transmission delays that the algorithm currently ignores, (that’s one reason I’m syncing to a local server, to minimize the transmission delays), but 120ms is a pretty big transmission delay for a local network, and is closer to what I would expect for US to Europe round trip.

So what are the sources of error?

  1. The shortcomings in the current library.
  2. The stability of the clock on the microcontroller.
  3. The stability of the clock on the server.
  4. The inevitable jitter caused by the vagaries of WiFi.

Of those, I can only address the first three sources of error. Time will tell if that’s good enough.

Initial Mic Prototypes

So I’ve gotten initial prototypes of the microphones done and am ready to start playing more seriously with the software.

First the Feather M0 prototype:

Feather M0 WiFi mic prototype
Sorry about the messy workspace

It’s got nice clean lines because most of the magic happens on the existing boards. I had to jumper the 8kHz square wave output of the RTC board to pin 10 so that it could generate timing interrupts, and I connected the microphone directly to power and the A2 analog input.

The Feather Huzzah ESP8266 microphone prototype is a little messier:

Feather ESP8266 mic prototype
The need for that voltage divider really tripped up the elegance of the design.

Since the microphone generates voltages from 0-3.3V, and the ADC on the 8266 can only handle a maximum input of 1v, I needed a voltage divider to bring down the mic voltage before feeding it into the analog pin. The proto board is overkill, but I didn’t want to wire it up freeform. I’ll put the DHT11 temperature sensor on this board as well, once it arrives.

So far, the M0 board has behaved well. at least once I managed to update it’s WiFi firmware. The ESP8266 has been more problematic. As long as I keep the sampling rate at 500 Hz or less, it behaves pretty well, but as soon as I try to push it past that, it starts dropping packets (sent and received) on the floor and returning errors. As near as I can tell, this is because more of the WiFi handling is done by the main processor (as opposed to being passed off to the dedicated subprocessor in the M0) and when I’m pushing the read rate up, it starves the main processor of the necessary time to do the WiFi stuff. 500 Hz is 20x less than my target rate of 10kHz though, so unless I can solve this problem by changing strategies, I’m going to have to switch to using the M0, and as I said, that will strain the budget. (Ideally I’d like to keep this under $500, but if I have to spend $55 on each microphone, that will blow almost half the budget there.) Maybe I don’t actually need the RTC chip though. There are cheaper oscillators and I can compensate for temperature using NTP.

Trials and Tribulations of debugging

I’ve gotten both the Huzzah ESP8266 and the M0 WiFi set up as monitors, though the huzzah is missing both the mic and temperature sensor for now. The code compiles cleanly on both, but is not completely working on either so far.

The ESP8266 seems to do most of it’s i/o in software rather than firmware or hardware, and that means resource contention can cause it to misbehave. If I have the timer interrupt library set for more than a few hundred interrupts/second (and I need 8K to 10K for the project), it doesn’t seem to be able to send or receive UDP packets, and eventually stops accepting new packets to be queued (probably because some buffer is full). I’ve tried adding a yield() call in the sending routine, and that doesn’t appear to have helped.

For the M0, it seems like it’s caching old library code. I had forked the NTPClient library to add the ability to get fractional parts of the second, since the existing library throws that data away. In theory, the new measurements are precise to the microsecond (not necessarily accurate, but precise, and hopefully accurate to within about 50-100 microseconds if synced to the same server). However, in my first pass I introduced a bug that caused it to jump about 5 years into the future (I’ve invented time travel!). and while I’ve squashed it in all my test cases, it continues to exhibit in the microphone code, though only when compiled for the M0. My best guess is that the arduino IDE is caching the compiled library object files somewhere in the project and and I can’t convince it to recompile. It’s also possible that I haven’t actually completely squashed the bug, but that seems less likely given that my test cases pass.

They both are dumping data to the simplistic server I have written up in python though–at least when the Huzzah can transmit UDP at all. So that’s gratifying.

[Update]. It was a real bug, but not in the NTPClient library. I was passing an IPAddress object in place of the hostname, which was expecting a const char *. The compiler helpfully converted the IPAddress object to a const char *, but in so doing smashed the stack of the NTPClient object so that the time_offset parameter to about 159 million seconds, which is roughly 5 years.

Huzzah ESP8266 arrived

The Feather Huzzah ESP8266 arrived from Adafruit today:

Featjer Huzzah ESP8266 unboxing

But I forgot to order the DHT11 temperature sensor to go with it, so I’ll have to prototype without it. I’ve been trying out my code on it, and it isn’t behaving as well as the M0. It’s not sending the UDP packets to the server, and after 10 seconds or so the endPacket() method on the UDP class starts returning errors. Time to get to reading docs and such.

The M0 has arrived

The parts for the Feather M0 microphone have arrived from DigiKey. I usually buy direct from Adafruit, but:

  • Adafuit was out of the loose header Feather M0 WiFi
  • DigiKey could get it to me quicker.

Here they are after I unboxed them:

Parts for the M0 Microphone prototype
Clockwise from the top: Feather M0 Wifi, Electret Microphone with amp, DS3231 Real Time clock FeatherWing.

The build should be pretty simple. It’ll look something like:

Fritzing diagram of Feather M0 microphone assembly.
Pretty simple build. Some headers and 4 wires is all it takes.

I already have a first pass at the code done too. We’ll see how it turns out.