Doubling Down

We lost power at RFO from about 8am to 5pm today. Everything came back online as expected, however, and we’re observing now.

Not a packet yet from IfA. They found the problem, though: the network was damaged in our building. Physical repair completed, and (eagerly) awaiting reconfiguration to bring us back online.

Almost there?

It’s the best night we’ve had at RFO in at least a month. Lots of stars, clear, not too much moisture, few to no clouds. 🔭 Hooray!

At IfA, however, the saga continues. Power had been “restored” via generator, then the main network cable was fixed… and re-fixed. Then we lost power from the generator, had to replace a major breaker after ordering a replacement from off the island, and now mains power and generator are both up.

200A Circuit breaker which needed replacing

As you might have concluded by now, there was a lot of damage to the equipment on the mountain from the storm in December. It’s not clear what was from direct storm damage, power fluctuations, or other secondary causes, but clearly there’s still more to repair because… we still haven’t been able to reach the instruments, despite general connectivity being restored for a third time this month.

My hope is and endeavor will be that this is the last post about what’s not working, and the next one has only good news. Next week? Fingers crossed. Stay tuned and thanks for thinking good thoughts for us!

It’s Cold Outside!

Just because you’re in Hawaii, doesn’t mean it’s warm! We got our first frost of the season, after being socked in and unable to observe last night.

Frost on the windows of IFA1

We can see effects of cooler temperatures in California, too. One of the camera’s in RFO2 is especially prone to being sticky when cold, but fortunately we now have software to handle it automatically.

Ideally, and with most of the other cameras, it takes only 3 attempts to open the shutter all the way 3 times in a row, but last night it took 9.

Hot hot hot!

LaserSETI would like the sympathize with all of the humans experiencing a heat wave today.

We spent a long time refining the thermal protection system for the instruments. Internal circulation, special heat-shedding paint, multiple exhaust fans, sunshades with embedded mirrors to reject even more heat. Days like today prove the value of that investment.

It’s 44 C (112 F) right now at Ferguson Observatory (RFO) and the cameras are the hottest I’ve ever seen them: 61 C (142 F)!

For comparison, the cameras on top of Haleakala, Maui (IFA) are a cool 16 C (61 F) right now.

LaserSETI @ RFO Speaker Series

Tomorrow night, May 26th, I’ll be kicking off the annual Ferguson Observatory Speaker Series. We’ll start from SETI basics, explain how LaserSETI is designed, and take questions. It’s free, it’s virtual, and they’ll give you your money back if you don’t learn anything!

Same Stars, Different Weather

As predicted by the forecast below, RFO is socked in tonight. Tomorrow night should be really good, however.

Astronomer’s forecast, credit A. Rahill

The first three rows mostly characterize moisture in the air (a lot tonight, very little tomorrow night). Seeing represents turbulence, but that doesn’t affect us much because our pixels are big, which is because our field of view (FOV) is enormous (75 degrees!). Darkness matters a lot however for the same FOV reason. And this week, the Moon rises and sets with the Sun, which makes the nights nice and dark because the Moon isn’t lighting up every tiny water droplet and mote of dust that happens to be floating around the sky.

And so this is what the sky currently looks like at Ferguson:

Live view screen capture from RFO1

However, at the same time, looking at the exact same stars–but from underneath a different part of the atmosphere–IFA2 is having a great night:

Live view screen capture from IFA2

Hopefully, tomorrow night it’ll stay this way in Hawaii but also look like this in California!

Follow-Up from Last Week

Turns out I did save a “before” movie of the RFO1 sunshade that wasn’t opening fully! It had been working great for a long time, but the real world is messy like that apparently. Clearly it was the humidity, cold temperatures, phase of the moon, and local unicorn population that was causing the friction.

RFO1 sunshade before fancy lubricant

Speaking of locals, Dan, did a phenomenal job this weekend and got the main computer within RFO2 back online! It’s been a long string of wacky issues, from the exact DC voltage it was receiving, to the firewall, to the unusual Vadj potentiometer on our DC transformer. The science cameras are still offline, but that’s probably a minor issue and we’ll hope to be observing again this week.

In related news, the hard drive on IFA2 has been flaky, so we’re also hoping to get that replaced on a similar timeline.

RFO Site Visit

Yesterday, Dan and I spent all day at Ferguson Observatory (RFO). We had a spot of rain to work around, but a number of things to get done and the trip was largely successful.

RFO1 (foreground) and RFO2 (covered) pose in front of RFO’s 8″ refractor dome

We dropped off the Drobo “SneakerNet” disk array that we use to exfiltrate data from RFO because the satellite link is too slow and expensive for that. LaserSETI analyzes data in real-time, but since we’re still in the early phases of our dual observation strategy, we’ve kept pretty much every pixel of observed photons thus far. This time was special because we had a failed disk (it’s a 5-disk array), but even after replacing that disk with two different spares, we concluded that one of the ports in the Drobo itself has failed.

We soldered the vert-cam’s power lead after having previously bypassed the relay in precariously fragile but 100% reliable manner *cough*cough*zip*tie*cough*.

Eliot takes a moment to appreciate Sonoma County's beauty while working on RFO1
Eliot takes a moment to appreciate Sonoma County’s beauty while working on RFO1

If you’ve been watching the live pictures from the instruments recently, you would’ve seen evidence of RFO1’s sunshade not opening fully. I think it might be related to the humidity but, in any case, we carefully applied our special Teflon lubricant meant for polycarbonate, allowing it to out-gas as much as possible before reinstalling it, to prevent any potential chemical reactions on the optical surfaces. It’s now performing flawlessly! I wish I’d remembered to save a before-and-after movie since it records a movie of every open or close, but it currently overwrites the previous one. It’s on our To Do list to save a sample, for the curious as well as to document the daily operation of this critical component.

Snapshot of live view from RFO1 on March 20th

Since it was wet, it was a good chance to observe the seals, louvers, and other ingress protection mechanisms in action. I’m pleased to report that not a drop of water could be found inside the instrument!

Clean and dry “bill of health” for the enclosures

On RFO2, the problem had been with its external hard drive, mostly working whenever tested but failing fairly quickly under the heavy load of science observing. Having replaced the drive, enclosure, and cord, we replaced the PC itself. Long story short, Dan did a huge amount of work, figuring out that adding an internal drive didn’t work while on 12 volt DC power (but did work on 13 volt DC), that modern browsers refuse to speak the old version of SSL that our router uses, and that Windows Firewall is unreasonably insistent in blocking remote access. All of these issues took an inordinate amount of time to diagnose, particularly because they all happened simultaneously and hence had to peel the layers of failure and obstacle like an evil onion. Now that we’re pretty sure we have a complete diagnosis, we expect to have RFO2 back online within a week or so, since it will require a another trip–the 5th actually–on this issue.

RFO2 in the middle of one of its (many) removals of its PC science computer

Jumping to Hawaii for a minute, after a surprising but successful visit last month to Haleakala, IFA1 has performed well and taken a lot of great data. Unfortunately, IFA2 has started exhibiting similar external hard drive issues to RFO2, so we’re making arrangements to try swapping out the disk and enclosure, as well as create some software backup procedures to allow some observing when this symptom appears, based on our learnings from trying to do so with RFO2.

For those of you who followed the Mars Exploration Rovers, I think you’ll understand why I feel like both RFO1 and IFA1 are like Opportunity, and RFO2 and IFA2 are like Spirit. After operating on Mars for more than a year, the NASA team noted that everything came easily to Opportunity, while Spirit had difficulty after difficulty. It’s especially eerie that Opportunity landed first, and the LaserSETI analogs are RFO1 and IFA1.

See you again soon!

Going to the Mountain

October 22, 2020

It’s exciting times for LaserSETI!  In this update we’ll talk about the state of the instruments, both in California and Hawaii, and deployment plans for installing the latter.

As probably many of you heard, the fire season this year in California was unprecedented.  Those of us in the Bay Area will likely always remember the Day It Looked Like Mars, which was a little bit neat for those of us who’ve dreamt of living on the Red Planet, but mostly scary –  a pretty good simulation of a nuclear winter, perhaps.  The most recent set of fires, generally referred to as the Glass Fires, did actually enter Sugarloaf Ridge Park where Ferguson Observatory and our LaserSETI CA instruments reside.  Fortunately, CalFire proactively protected the observatory and worked hard to keep the fires from pushing any further south into the park.  We’ll continue to work closely with the great team at Ferguson to do everything we can to keep the observatory safe, and our hearts go out to all those affected by these catastrophes.  It really hits home for us at the SETI Institute when one of our colleagues lost his home in the Lightning Complex fires in the Santa Cruz mountains.

(You can follow Ferguson Observatory here)

The evacuation orders have lifted but Sugarloaf Ridge Park is still “closed for infrastructure repairs.”  So, unfortunately, after a week of not being able to observe due to the smoke in the sky, we were then without power for over a week, and just got back on the sky two nights ago.  But just last night, as a precautionary measure, I stopped the observing process and secured the instruments, because Pacific Gas & Electric started forecasting a Public Safety Power Shutoff (PSPS) in our area.  When weather deemed hazardous to their infrastructure is forecasted, they shut power off to avoid starting more fires.  This isn’t the first time in the past year that we’ve had extended observing interference, but I think it’s now the longest, most complicated, and dramatic!

As soon as possible, I’ll be heading up to check on the instruments and do some upgrades. I never saw any significant soot accumulate on the enclosure windows, but it’s a good time to double check and replace the air filters. I’ve got new light baffles which should have greater longevity and will reduce the pressure on the sunshade opening. And most importantly, I’ve got new LaserSETI decals to put on the covers, since we had to remove the old ones when we applied the white thermal paint.

Speaking of which, I thought everyone would enjoy seeing the newly built and tested instruments. It’s all white but the covers are white thermal paint to help keep the instruments cool, while the rest is simply powder coated so the instrument will blend in in situ. You can see the piano hinge of the new wiring box design which not only saves money, but makes it much easier to access that space. My favorite improvement, though, are the handles on the cover. That component weighs about 60 lbs. so now it’s much easier to remove and put back on! You can also see the new relays underneath the power distro box in the front, which now allow both computers to be rebooted by the other, in addition to the original capability of rebooting the cameras.  If you want to learn more about the components and design, let us know and we can have a Facebook Live or dedicated update.

Long story short, the instruments here in California have taught us a lot on how the designs could be improved.  The differences may not be immediately apparent to the untrained eye—which I suppose is a positive sign that our designs were pretty good to start with—but there are over two dozen separate improvements to the enclosure and supporting electronics.  All of these have been incorporated into the two new instruments soon heading for Hawaii, and as many as possible retrofitted on the instruments installed at Ferguson.  There’s not a lot we can do about the wildfires, high winds, power outages, and pandemics that slowed progress and interfere with observing, but we’ll keep doing whatever we can to be prepared and adjusting as necessary!

And finally, the big news is we’re finally able to announce our location in Hawaii. The University of Hawaii’s Institute for Astronomy has agreed to host the two instruments shown above on the summit of Haleakala on the island of Maui!  The site is at 10,000 feet above sea level (3000m) and is absolutely outstanding. 

As you can see on this rendering, we created for the state Site Plan Approval application, our neighbors will be PANSTARRS 1 and 2, another awesome project focused on transients!  And if you were wondering why that truss was underneath one of the instruments in the previous picture, now you know: so we can see over the railing.  It was no simple matter, however, for it to be able to tolerate winds over 130 mph!  The other instrument doesn’t need that, as you can see from the rendering below. 

We’re working on the shipping and final installation details now, but the good news is at least Hawaii’s mandatory 2-week COVID quarantine can now be bypassed with a negative rest result within 72 hours of flight departure.  As it stands, we’re shooting for First Light to happen before the end of this year! It is 2020, though, so I would hold off ‘cashing that check’ until installation actually happens!

As usual, we’ll have a lot more to talk about in our next installment, probably 4 to 6 weeks from now, and I think you’ll find it worth the wait. As always, thanks so very much for all your support!