Commissioning update!
March 23, 2020
A lot has been going on and I wanted to tell you about it personally! In case you’ve forgotten, I’m Eliot Gillum and the principal investigator for LaserSETI.
Our last update was the exciting announcement that LaserSETI was installed at the Robert Ferguson Observatory in Sonoma County in Northern California. It was mostly pictures to help share the feeling of it all, but this update will be a lot more text, as I want to share some of the “hands on” behind the scenes action of commissioning a new instrument. In case that’s not your thing, I won’t be offended if you skip to the end for the exciting announcements.
Before getting into the specifics, I should say that, while none of the specific issues were anticipated (or else we would’ve addressed it ahead of time!), the general types and quantity were as expected. A sensitive instrument in an outdoor environment is always going to require tweaks. I also think it’s worth noting that—knock on wood—no “big scary” types of problems happened; nothing that threatened the design or instrument health.
The first issue encountered was the interior temperature within the enclosure rising too high during the day, such that the cameras’ coolers were unable to maintain the CCDs at 0°C. This wasn’t an immediate problem because we can’t observe during the day, but the thermal cycling and extremes weren’t good for the equipment and, since we’ll be deploying to other parts of the globe that might be hotter than Northern California, it was important to get ahead of this. The problem wasn’t so much internally generated heat or ambient external, but insolation—direct sunlight. Our first course of action was to paint the cover with a special white thermal paint (see below) to dramatically increase the albedo (amount of light reflected) as well as aid radiative cooling from inside (see Figure 1). We also subsequently switched the color of the 3D-printed components to white, embedded mirrors in the sunshade, and improved the airflow from the instrument bay down to the wiring “bunker” where the exhaust fan and ports are located.
Figure 1 – LaserSETI covers after being painted white with specialized thermal paint
As some of you are likely aware, when 3D printing, a wide variety of different plastics can be used, each with their own unique properties: hardness, texture, impact strength, thermal, etc. At the high end, some very expensive materials are possible, as well as outsourcing to specialized shops with high end machines. But since one of LaserSETI’s fundamental principles is “low cost to achieve global coverage,” we have focused on “in-sourcing” the printing to keep costs low. We had started with ABS plastic, which is used for everything from LEGOs to hand tools, but our ABS printer started being unable to print large pieces and not even the manufacturer could figure out how. We decided to back off to PLA, which is much easier to work with and hope that the parts we used it for wouldn’t experience high enough thermal and mechanical loads. Unfortunately, its lower softening point conspired with the initially high internal temperatures, and we’ve had to replace all the PLA parts. The plastic we’re now using is polycarbonate, which has ideal thermal properties but required a new printer. The parts take longer to print but come out very high quality, although the printer has also proven itself a lot of work to keep operating smoothly.
To keep peeling the onion, while the polycarbonate plastic is a great improvement overall, switching to it exposed a flaw in the gear tooth shape we’d designed into the sunshade mechanism. Since polycarbonate is more rigid, the sunshade started binding when approximately 75% open. Fortunately, we were at least still able to observe in that condition, while we investigated the problem. Once diagnosed, it was easy enough to design a better tooth, re-print the gears and sunshades, and re-install them on our next visit. In Figure 2, below, you can see the new trapezoidal gear tooth, as well as the mirrors added to reject additional heat from the sun.
Figure 2 – Interior view from pi camera: sunshade with embedded mirror, stepper motor, outside view
Besides the heat, the weather has thrown some other curve balls. Now that it’s winter, we expected more cloudy nights, which is partially compensated for by the longer nights to observe—and we’ve had some really great ones! Additionally, high winds have been an unexpected issue, since the power company aggressively shuts down power over large areas, in order to avoid more tragic firestorms like the Tubbs Fire of 2017. At one point, we were down for two weeks as the power went out and came back. Then, to add injury to insult, the power fluctuations were enough to damage the Ferguson VPN server, which required more time to replace. The silver lining has been a thorough test of the LaserSETI electrical systems, which passed with flying colors!
Figure 3 – Raspberry Pi at top center, humidity sensor, camera, motor, and GPS antenna cables visible
One piece of hardware which hasn’t been robust enough, however, is the Raspberry Pi single-board computer used in each instrument as its the central nervous system. It’s responsible for wide range of tasks, from monitoring environmental sensors, to controlling the sunshade, to the interior camera, to super accurate time using a specialized GPS. We build and solder all these various components ourselves and had initially optimized the design for a low profile, not knowing exactly how much clearance we had with the cover. Faced with transient electrical failures that were clearly related to thermal expansion and observing that we had enough spacing from the cover, we’ve greatly simplified the electronics design with the double benefits of making it more robust and easier to build!
The specialized GPS setup is interesting area for more detail. Most people know that GPS determines your location, but not everyone realizes it does that by determining the time very accurately: usually within hundreds of nanoseconds. LaserSETI uses this to synchronize observations across observatories, so we can match observations with confidence and even measure the arrival time delay based on the source direction! We’re using a commercial-off-the-shelf (COTS) GPS but with an additional wire called PPS, meaning “pulse per second.” This little wire has a “positive edge” at the precise beginning of each second and greatly improves the timing accuracy over the simple position reports via the serial port. At least, in theory, that is. It took months of reading, experimenting, and tuning to optimize the configuration so that the raspberry pi is accurate to within a dozen microseconds, and the PCs are usually within 100 microseconds!
Figure 4 – Screenshot of real-time monitoring of clock delays (in microseconds)
Now, the system is so accurate, the biggest daily disturbance is when the sunlight suddenly hits the GPS antenna or goes away, because of the rapid change in temperature! In Figured 4 above, sunny days are yellow, cloudy are circled in white, and partly cloudy days in gray. And to try to address even this variability, we’ve designed a new GPS antenna mount with a sunshade, which we’re told looks like one of the bad guys from Super Mario (see Figure 5 below).
Figure 5 – GPS mount with new “Mario” sunshade, designed to reduce thermal shock causing clock drift
Another electronics issue has been a few random computer lockups. LaserSETI was designed to handle flaky cameras with relays so they could be power cycled, but not the PC or Raspberry Pi. We’ve now added a relay so the pi can reboot the PC, and are rewiring the PC-controlled relays to move both cameras onto a single relay, freeing up the other to allow it to reboot the Pi. The Ferguson staff have been very accommodating and helpful in so many ways, but we don’t want to ask more than necessary of them, plus as we go to Hawaii and beyond it’s critical the instrument is able to be self-healing.
Having taken data for months, we wanted to be able to examine it in detail, but we couldn’t exactly upload 13 TB (13 million megabytes) over the satellite link that connects Ferguson to the outside world. Forced to resort to “SneakerNet”, we built a 30 TB disk array, which has worked great—ever since a certain online retailer sent the right type of disks. We’ll continue to use this array to keep as much data as we can in the long term, for analysis and sharing too!
Figure 6 – Snapshot of controller software while observing recently
Because of all the work above, I’m pleased to share that we’ve begun building the second round of two instruments: the ones going to Hawaii. Now because our 33-page site permit application isn’t yet approved, we’re still not able to announce the exact location, but I think that’ll come in our next update. What I can say is that the cameras, the most expensive part of the instrument, have been built and shipped, the instrument design improved from electronics to steel, the fabricator has begun cutting and welding, and the 3D printer is running overtime.
While the engineering behind LaserSETI is fun, I know we’re all in it for the science. But since the plan for publishing the full instrument results involves having both observatories taking data, I thought we could at least offer a preview of what we’ve seen thus far. Bear in mind that, because we’re still working out the kinks in the commissioning process and don’t have Hawaii online yet, this is based on less than quarter of the system’s full observational capability, as funded by you. What I can say is that we haven’t yet identified any events that met our detection criteria and were seen simultaneously by both cameras within the instrument. We will, of course, keep taking more data every night it’s possible. And we’re reviewing the raw data, as well as detector thresholds and algorithms—now that we’ve got a lot more data to work with—to check for potential bugs or improvements.
I’d like to offer a brief preview for future updates. Our next one will probably be just before or after installing the instruments in Hawaii—after they’ve been built, tested, and shipped. Beyond that, we’ve got some exciting secondary (non-SETI) science to talk about, as well where LaserSETI will go after the culmination of this campaign.
Finally, I want to thank every single one of you for your support. While we certainly can’t buy parts and pay contractors without funding, knowing that there’s hundreds of people around the world who were just as excited about LaserSETI has meant the world to me and definitely provided extra motivation to make the project the most it can be!
