It’s the best night we’ve had at RFO in at least a month. Lots of stars, clear, not too much moisture, few to no clouds. 🔭 Hooray!
At IfA, however, the saga continues. Power had been “restored” via generator, then the main network cable was fixed… and re-fixed. Then we lost power from the generator, had to replace a major breaker after ordering a replacement from off the island, and now mains power and generator are both up.
200A Circuit breaker which needed replacing
As you might have concluded by now, there was a lot of damage to the equipment on the mountain from the storm in December. It’s not clear what was from direct storm damage, power fluctuations, or other secondary causes, but clearly there’s still more to repair because… we still haven’t been able to reach the instruments, despite general connectivity being restored for a third time this month.
My hope is and endeavor will be that this is the last post about what’s not working, and the next one has only good news. Next week? Fingers crossed. Stay tuned and thanks for thinking good thoughts for us!
In the years that LaserSETI’s been operational across two sites, this is certainly the longest weather delay we’ve ever had simultaneously affecting both sites.
The good news is that Hawaiian Telecom made good progress repairing the fiber on the slopes of Haleakala today, and should get us back online by Friday at the latest–assuming they can secure a helicopter to assist. Apparently, helicopters don’t grow on trees, even in Hawaii.
The probably-not-good news is that, here in CA, the weather is still bad and who knows when it will clear up. Forecast says rain for another week, but maybe some of that could be during the day and the clouds could push out at night? 🙂
Current weather radar for Ferguson Observatory (located within Sugarloaf Ridge State Park)
Just because you’re in Hawaii, doesn’t mean it’s warm! We got our first frost of the season, after being socked in and unable to observe last night.
Frost on the windows of IFA1
We can see effects of cooler temperatures in California, too. One of the camera’s in RFO2 is especially prone to being sticky when cold, but fortunately we now have software to handle it automatically.
Ideally, and with most of the other cameras, it takes only 3 attempts to open the shutter all the way 3 times in a row, but last night it took 9.
For the first time in years, I took a trip to Maui where the primary reason was actually to have fun. But of course I wasn’t going to miss the opportunity to visit LaserSETI! And my father-in-law was kind enough to come along for the ride and even give me a hand–literally and figuratively.
It’s always a pleasure, after long drive up the mountain as the last step of the journey, to climb the stairs of the Airglow Building and see the instruments, peacefully sitting amongst the incredible scenery.
The main bit of work we did, besides inspection and maintenance check, was to install a 19V step-up transformer, because we’ve found that the USB bus on this newer generation of NUC computers can be a little finnicky when supplied with 12V like everything else in the instrument. Since the science cameras and disk are connected via USB, any issues in that department are big issues.
Checking in on the air filters, it was clearly time to replace them. Apparently summer is a dusty time up there on the summit. These MERV 13 filters aren’t cheap, but they do a their job very well and have demonstrated a very reasonable operational life expectancy.
In every LaserSETI instrument, there’s a large-capacity USB hard drive that we use to capture all the data coming from the two science cameras. Even with the biggest drives being made today, we can only store a month or two worth of data, but it’s an important staging area. The one in IFA2, however, had been dropping offline whenever we tried to read or write too much data to it.
So we shipped out a new drive to Doug, who’s local on Maui, and yesterday he drove the 3-hour round trip to the summit and replaced the drive. The weather had been uncooperative and we prefer not to operate in the rain, but he saw an opening and made a dash for it.
New hard drive in IFA2, underneath the science computer, which is below the “vert-cam” science camera
I don’t want to say “it’s fixed” too early or jinx it, but last night we made it through a full night of observing, streaming data at over 100 Mbps, while also backhauling the data to our datacenter in California… stressing the hard drive to the max, and everything went smoothly! Fingers crossed it stays happy for a good long while.
Turns out I did save a “before” movie of the RFO1 sunshade that wasn’t opening fully! It had been working great for a long time, but the real world is messy like that apparently. Clearly it was the humidity, cold temperatures, phase of the moon, and local unicorn population that was causing the friction.
RFO1 sunshade before fancy lubricant
Speaking of locals, Dan, did a phenomenal job this weekend and got the main computer within RFO2 back online! It’s been a long string of wacky issues, from the exact DC voltage it was receiving, to the firewall, to the unusual Vadj potentiometer on our DC transformer. The science cameras are still offline, but that’s probably a minor issue and we’ll hope to be observing again this week.
In related news, the hard drive on IFA2 has been flaky, so we’re also hoping to get that replaced on a similar timeline.
The road to the summit of Maui’s Haleakala Mountain has 32 switchbacks.
When Eliot Gillum, Director of LaserSETI, picked me up at the Maui airport, he jokingly asked me if I got car sick, and I laughed him off. But the ride up Maui’s tallest mountain is not to be taken lightly. It’s a gorgeous meander filled with blind turns and steep drop-offs. Gaggles of bikers laugh as they stream down the road, unbothered by the cars alongside them, and a few of the more serious athletes huff and puff their way up the 22-mile incline.
Eliot and I are making this trek from sea level to the 10,000-foot summit in a car with LaserSETI cameras in the trunk. LaserSETI is the first all-sky, all-the-time optical SETI project, which focuses on detecting laser pulses from outside the solar system. Distant advanced civilizations could use lasers for communication and spacecraft propulsion.
Last August, Eliot installed two LaserSETI instruments on the top of Haleakala. But by December, the science cameras had started going offline one by one. Each LaserSETI instrument consists of two wide-field, high-sensitivity cameras to capture images of the night sky, and when one of them goes out, observing is severely compromised. If you lose both, you’re out of business entirely. Fortunately, we’d come equipped with four replacements—two for each instrument—and swapping the malfunctioning cameras out for new ones is a quick process.
When we reach the top of the mountain, the visitor’s center is overflowing with people who have made the hike to watch the sunset. The sun is already dipping close to the horizon and shining orange against the clouds several thousand feet below. It’s beautiful, but we are running out of daylight for the camera swap.
Once we park the car, Eliot and I carry the equipment a few hundred more feet to the LaserSETI instruments, which rest on top of the “Air Glow” building. It’s a small structure made of cement and volcanic stone that holds equipment for several other instruments that share the roof with LaserSETI.
Since we’re so high up in the atmosphere, we put our coats on over our Aloha shirts before we get to work. Well, Eliot does most of the work. He opens the protective steel cover on top of the first instrument and disconnects the current cameras’ power supplies. Inside, we extract the components over each camera then delicately exchange the old ones for the new ones Eliot tested in his workshop back in California.
The sun has dipped below the horizon by the time we finish, and my fingers are pretty blue. Still, it will be another half an hour or so before it’s dark enough to test the cameras.
Turns out we don’t need that long.
Retreating inside the building into a spartan “Blast Room,” Eliot connects to the instrument’s computer. And it’s not detecting the cameras. Nothing has changed after the camera swap. And the two new cameras we just installed worked perfectly in California.
Eliot stares at his computer. I stare at the wall. Could something have happened to the cameras in transit? The box looked fine when we picked it up. Eliot heads back to the roof with his flashlight but finds nothing that could be causing the problem.
It wasn’t the cameras causing the instrument malfunctions after all.
The Raspberry Pi and PC are working, so power is flowing within the instrument. Our best guess is that there’s something wrong with the relays—electrically controlled switches that stream power to individual components within the instrument.
Whatever it is, we can’t figure it out right now. It’s dark and cold, making it hard to investigate and risks accidentally causing a new problem while fumbling around within the tight space. So, we close shop to get a good night’s sleep.
We head back to the car and down the perilous mountain road to the farmhouse, a historical building and base for many scientists at the University of Hawaii’s Institute for Astronomy. Our colleagues at IfA graciously allowed us to bunk at the house, where the Institute used to conduct classes and research.
Early the following day, fueled by iced coffee and the intrigue of our scientific mystery, we head to the Home Depot in Maui’s biggest city, Kahului, for a multimeter. This tool will measure the electricity in different instrument parts to find the issue.
We hit the road with a multimeter and our second round of coffee. It’s a gorgeous day on Maui, so the road is filled with bicyclists and runners. We even spot a few Nene, the Hawaiian goose, as we make our ascent.
We return to the Blast Room to drop off our gear and put on sunscreen before heading to the roof. Eliot gets to work testing the relays. These relays, a sort middleman, were added to enable the team to restart individual components remotely without shutting the entire instrument down.
Eliot tests the first connection, which sends power to the Raspberry Pi. There’s enough voltage going into the relay and enough coming out. So that’s not the issue.
The second connection, which supplies power to the hard drive, is also good.
Then we get to the connection to the camera. Power goes into the relay, but the terminal leading out and into the camera shows virtually no voltage.
We’ve found our problem. We still don’t know why, but we now have proof that the relays have died, and that’s why the cameras turned off—they don’t have any power. Eliot is surprised since he saw some errors from the cameras that he’s never seen from their siblings in California. But the evidence is undeniable that the issue rests with the relays.
Eliot brought one spare relay board, but since this problem also occurred on the other instrument a few feet away, there was probably something wrong with this batch of relays from the manufacturer. At this point, it’s best to remove the relays from both instruments.
Armed with a screwdriver and pliers, Eliot disconnects the appropriate wire from the relay and connects it directly to the camera. He reinstalls the fuse, and we hear them power up. When we return to his computer, the cameras are ready to snap pictures. Voila!
It’s now late afternoon, but there’s plenty of light left in the day to bypass the relay on the other instrument. Once that’s complete and the cameras in that setup are online, we retreat to the Blast Room to wait for astronomical twilight. Astronomical Twilight is when the sun is 12 degrees below the horizon, meaning there’s not much daylight bouncing around in the atmosphere, and scientists can begin observing faint, faraway space objects.
We’re waiting for this moment because we need to ensure the cameras in the first instrument, which we changed out last night, are focused. We aren’t concerned about the other instrument since we disconnected the relay and didn’t touch the cameras.
At 7:33 p.m., it’s go time. Eliot connects to the LaserSETI console and snaps a picture with the instrument with the new cameras. It takes about a minute for the images to load. Both cameras captured an image of a grey background with a speckling of white dots. The cameras happen to be perfectly focused already—a stroke of luck.
Since both instruments are in focus and actively recording the night sky, job well done! We pack up, take a moment to enjoy all the stars twinkling in the darkness, then head back to the car.
The LaserSETI team uses a design Eliot created in collaboration with many SETI Institute scientists to construct each LaserSETI instrument. But every time a problem crops up, it’s a brand-new challenge. But each bump in the road is a lesson the team will apply to new instruments as they’re built and deployed.
There are currently LaserSETI instruments on Haleakala in Maui and at the Robert Ferguson Observatory near Kenwood, California. The team has already begun work on the next ten instruments to place in other locations around the world. When these are up and running, LaserSETI will monitor the nighttime sky in roughly half of the western hemisphere and bring us closer to understanding whether another intelligent life in the universe exists.
Yesterday, Dan and I spent all day at Ferguson Observatory (RFO). We had a spot of rain to work around, but a number of things to get done and the trip was largely successful.
RFO1 (foreground) and RFO2 (covered) pose in front of RFO’s 8″ refractor dome
We dropped off the Drobo “SneakerNet” disk array that we use to exfiltrate data from RFO because the satellite link is too slow and expensive for that. LaserSETI analyzes data in real-time, but since we’re still in the early phases of our dual observation strategy, we’ve kept pretty much every pixel of observed photons thus far. This time was special because we had a failed disk (it’s a 5-disk array), but even after replacing that disk with two different spares, we concluded that one of the ports in the Drobo itself has failed.
We soldered the vert-cam’s power lead after having previously bypassed the relay in precariously fragile but 100% reliable manner *cough*cough*zip*tie*cough*.
Eliot takes a moment to appreciate Sonoma County’s beauty while working on RFO1
If you’ve been watching the live pictures from the instruments recently, you would’ve seen evidence of RFO1’s sunshade not opening fully. I think it might be related to the humidity but, in any case, we carefully applied our special Teflon lubricant meant for polycarbonate, allowing it to out-gas as much as possible before reinstalling it, to prevent any potential chemical reactions on the optical surfaces. It’s now performing flawlessly! I wish I’d remembered to save a before-and-after movie since it records a movie of every open or close, but it currently overwrites the previous one. It’s on our To Do list to save a sample, for the curious as well as to document the daily operation of this critical component.
Snapshot of live view from RFO1 on March 20th
Since it was wet, it was a good chance to observe the seals, louvers, and other ingress protection mechanisms in action. I’m pleased to report that not a drop of water could be found inside the instrument!
Clean and dry “bill of health” for the enclosures
On RFO2, the problem had been with its external hard drive, mostly working whenever tested but failing fairly quickly under the heavy load of science observing. Having replaced the drive, enclosure, and cord, we replaced the PC itself. Long story short, Dan did a huge amount of work, figuring out that adding an internal drive didn’t work while on 12 volt DC power (but did work on 13 volt DC), that modern browsers refuse to speak the old version of SSL that our router uses, and that Windows Firewall is unreasonably insistent in blocking remote access. All of these issues took an inordinate amount of time to diagnose, particularly because they all happened simultaneously and hence had to peel the layers of failure and obstacle like an evil onion. Now that we’re pretty sure we have a complete diagnosis, we expect to have RFO2 back online within a week or so, since it will require a another trip–the 5th actually–on this issue.
RFO2 in the middle of one of its (many) removals of its PC science computer
Jumping to Hawaii for a minute, after a surprising but successful visit last month to Haleakala, IFA1 has performed well and taken a lot of great data. Unfortunately, IFA2 has started exhibiting similar external hard drive issues to RFO2, so we’re making arrangements to try swapping out the disk and enclosure, as well as create some software backup procedures to allow some observing when this symptom appears, based on our learnings from trying to do so with RFO2.
For those of you who followed the Mars Exploration Rovers, I think you’ll understand why I feel like both RFO1 and IFA1 are like Opportunity, and RFO2 and IFA2 are like Spirit. After operating on Mars for more than a year, the NASA team noted that everything came easily to Opportunity, while Spirit had difficulty after difficulty. It’s especially eerie that Opportunity landed first, and the LaserSETI analogs are RFO1 and IFA1.
It’s been longer than I wanted since the last update, but I promised myself the next update would have real news to share, instead of trials, tribulations, and delays—and that day has finally come!
The LaserSETI instruments bound for the summit of Haleakala are on a boat and have land in sight! We have plane tickets, a rental car (!!!), and a date to begin installation. Our second observatory is coming together, and we’ll be able to start co-observing our two fields-of-view.
I was a little excited that the instruments were finally shipping out. I do own other shirts, for the record.
And here’s what’s inside each case: one instrument, half of the riser for the instrument that needs sit above ground level, non-instrument components like conduit, cable, and the DC transformer, as well as spare parts just in case. The little blue box is a “transport data logger” which records the forces experienced during shipment, for learning and insurance.
The cases were just a few inches too big for the truck bed so we built a platform on top. I can confidently say it was the most comfortable and secure ride of the instruments’ lives.
I don’t want to dwell on the negative but, since the instruments have been built and waiting for 10 months now, I think it’s important to account for the time. Unsurprisingly, the majority was due to COVID-related travel restrictions, both in Hawaii and then California. The remaining third, however, was difficulty in obtaining shipping cases. One surprise was that reusable cases were less than 50% more than one-time use crates which, since we have many more instruments to ship out around the world, will save thousands of dollars. It also means that, in the future, there’s one less external dependency to keep the project on track. The major cause here, however, was a vendor that was bad at communicating and worse at delivering. It’s no astronomy joke to say it was “night and day” once we were finally able to identify an alternative vendor and case product that suited our needs on a reasonable timeline. The value of people and companies who keep their word and deliver on-time just cannot be overstated!
The next update will certainly be soon, to share pictures and stories on the installation, and I’m confident we’ll get some stunning pictures from on top of that mountain! And stay tuned in case we have any surprises to announce!
