Every year, now for the past 3 years, there’s a giant wind/ice storm that knocks power and connectivity out at the summit of Haleakala. This year, it started over a week ago, and we’ve just regained connectivity to the instruments. Hooray!
Fingers crossed we’re done for this year and that the infrastructure improvements they’re doing make next year even shorter or a non-event.
Screenshot of IFA2 control console
We were actually observing when the power went out, but our multiple levels of safeguards obviously did their job. As you can see, the cameras are in fine working order.
It sure would be nice if the moon weren’t so bright, though! Even outside of the field of view, the two bright semi-circles in the bottom half of the images are scattered light from it.
Making our next 10 instruments efficiently, as well as ensuring their reliability and ease of maintenance, is a top priority. Some of you might remember a previous iteration of a power board, an attempt to reduce wiring and build effort that encompassed all power wiring in the instrument and attempted to eliminate a COTS (commercial off the shelf) component. That board, however, was overpriced and unnecessarily aggressive, so we switched vendors, changed how we did the cabling to avoid joints, and reduced the scope of the board to handling the two remaining wiring challenges: the fans and FLI cameras.
The result is above and below, and is 40x cheaper than before, while still eliminating all of the manual labor, difficulty when swapping out components, and long-term reliability risk! Fans plug into 4-pin headers, and FLI cables will screw into block terminals (not rendered below) in the bottom right. No soldering, everything labelled, quicker and better than could be done by hand.
3D rendering of new LaserSETI power breakout board
We hope you enjoy these peeks into the details behind LaserSETI, but stay tuned for some bigger updates and announcements soon!
Since we’re building 10 new instruments, we’ve made a number of small investments to make building them easier and faster. This was a fun one to share:
Two LaserSETI instrument bases, with different types of dollies to make building them easier
On the left, you’ll see an off-the-shelf appliance dolly. Cheap and easy to obtain, but it only allows access to the wiring area in the instrument base with the panels removed and with the dolly somewhat awkwardly interfering with said access. On the right, we’ve machined custom aluminum “legs” with casters, so the bottom panels can stay on and even be opened or closed as necessary. The machining took a couple of hours, however, so we’ll be testing both side-by-side, to see which fits the need better–or maybe both are good for slightly different purposes or phases of construction!
For reference on our development process, we’re including XKCD’s study of tool making below (“automation” in programming-speak). 😉
We didn’t get as much time on the sky this week than normal, as we had work going on at both observatories, RFO and IfA. One was planned, the other not so much.
At IfA, our maintenance was unplanned. IFA2’s sunshade stopped working, which is scary because the sunshade protects from the Sun potentially damaging the shutter on the FLI science cameras. You can see the problem below, using our internal “PiCam”: the polycarbonate gear started to slip around the motor shaft.
We recent started a project to detect the the sunshade position using computer vision to catch exactly this sort of problem automatically but, since we currently check every morning, we caught it manually. Fortunately, Team LaserSETI member Doug Hagan who’s local on Maui, was able to make the long drive up Haleakala and close it by hand.
Speaking of cameras, the horz-cam on IFA1 had started misbehaving transiently. We tried remote diagnostics and troubleshooting but eventually we had to admit it wasn’t succeeding. So we’ve sent a new camera (and a new gear) to Doug, and he’ll be replacing both shortly.
Meanwhile, at RFO in California, they’re upgrading the roof surface of the observatory which our instruments sit directly on top of. We worked with observatory team to come up with a plan for them to coat the the whole roof without disturbing the position of the instruments.
If you’ve been reading the news recently, it won’t surprise you to hear that our instruments on the summit of Haleakala, Maui, have been offline for the past couple days.
The good news is they came back online this afternoon, and seem to be in great shape. In fact, I would assess-timate the high winds even did a good job cleaning the windows.
What LaserSETI’s IFA2 instrument saw when it woke up
Of course, that’s the easy part of this post. Our knowledge of the enormity of the tragedy from the fires on Maui grows day over day, and wrenches our hearts ever harder. People, businesses, history, nature, all gone and without warning.
Personally, I haven’t heard from everyone I know there, but each one I do hear from is a gift. It reminds me all too much of the firestorms that struck California in recent years–not “just” a fire, but a fast-moving “natural” blowtorch that creates its own weather as it decimates everything in its path. I keep thinking about a sign that hangs in my mother-in-law’s house, and she was kind enough to take a picture of it for me.
If there’s anything you can do to help the wonderful people of Maui recover, now or in the future, I hope you’ll act on that opportunity. The best resource I could find on ways to help was here.
All the life we know of clings to the thin skin of a giant rock flying through space. I think we should take care of each other.
The hard drive in RFO2 has given us trouble before. A year ago, we swapped out the disk, the enclosure, and even the computer itself. For a while, it was smooth sailing, but then started acting up again.
So, after replacing the disk and enclosure again, we tried the power and data cables. That was an improvement, but still not 100%. And since we can’t observe without the hard drive, anything less than perfect isn’t good enough.
We’ll be headed up to Ferguson again soon, this time with a new idea in hand, to try to put this trouble maker to bed once and for all: a solid state disk. Smaller capacity, but completely different performance, electrical, and other characteristics.
Since RFO2 wants to be an attention-hogging snowflake, here’s the work from its perspective. LaserSITTIE is, by far, my favorite piece of maintenance equipment. There’s nothing like sitting when we previously had to bend over the instrument, or kneeling on it instead of the often-sweltering roof.
Many of LaserSETI’s parts are challenging to print, simply due to the required function of the part and the limitations of Fused Deposition Manufacturing (FDM), aka 3D printing. We’d worked out most of the kinks in the original design of the parts, but one has been giving us trouble.
Failed print used to fabricate new brace which is holding broken column in place
This is the bracket that holds the “horz-cam” but also, underneath in the bay visible above, holds the router that connects all the various computers that make LaserSETI operate and analyze.
The near side is where all the ethernet cables plug in, so obviously can’t be a solid wall. The highlighted support pillar is large enough to grow into the camera bracket higher up, but small enough to not cause a problem with the cables below. But sometimes, if everything below didn’t print 1000% perfectly, it became a long lever arm that got pushed around by the print head and eventually broke off.
It’s a large part, taking 75 hours of machine time and about $15 in raw plastic. So a print failure more than halfway through is a frustrating waste of time, money, and effort. And it’d happened one time too many times. There are a plethora of factors, mainly the printer and type of thermoplastic, but many variables aren’t realistic to control precisely, such condition of the machine, ambient temperature and humidity, feedstock humidity and other imperfections.
We’d tried fixing it “the right way” by printing a support pillar on the side, meant to be broken off once the print job finishes. But it wasn’t getting a solid enough connection to the model and too often failed in its one job.
Printing again with the brace in place
As the saying goes, “if you want something done right, do it yourself.” So we carefully crafted a brace from scrap wood, it fits tightly, and we’re expecting this will get the job success rate up, from maybe 65% to more like 100%!
It seems things are slowly returning to normal on Haleakala. The power has been more stable <knocks on wood> and this is the second night in a row of those gorgeous clear skies. We’ve got a lot of observing time to make up!
Live view from IFA1
We’ve also got an exciting announcement coming up about data from Maui. More on that soon!
No less than 75 days after a large storm hit Maui, we heard from the instruments on Haleakala today!
The storm knocked out both power and fiber to the summit and it’s been a long and bumpy road getting power and connectivity back. The turmoil caused all sorts of secondary damage: breakers, generators, routers–all had to be replaced as problems were discovered sequentially. Sometimes temporary repairs had to be undone and redone. Sometimes replaced components had to be replaced again. We’re very grateful to the tenacity of the IfA team, as well as the workers at MECO and Hawaiian Telecom.
Screen capture of telemetry from IFA1
As you can perhaps read from the graphs, tonight the humidity was 100%, a strong indication you won’t be seeing any stars. But, as a system test, we observed anyways for a short period and–happily–everything performed as expected. Unfortunately, utility power was lost again, near 10pm local time.
We were only online for a handful of hours, but it’s so nice to know that, after a 2.5 month sabbatical and torrential ice storm, everything is still in great shape! Fingers crossed for more stable power from here on out, and getting back to the science!
We lost power at RFO from about 8am to 5pm today. Everything came back online as expected, however, and we’re observing now.
Not a packet yet from IfA. They found the problem, though: the network was damaged in our building. Physical repair completed, and (eagerly) awaiting reconfiguration to bring us back online.
