Category RFO

We didn’t get as much time on the sky this week than normal, as we had work going on at both observatories, RFO and IfA. One was planned, the other not so much.

At IfA, our maintenance was unplanned. IFA2’s sunshade stopped working, which is scary because the sunshade protects from the Sun potentially damaging the shutter on the FLI science cameras. You can see the problem below, using our internal “PiCam”: the polycarbonate gear started to slip around the motor shaft.

We recent started a project to detect the the sunshade position using computer vision to catch exactly this sort of problem automatically but, since we currently check every morning, we caught it manually. Fortunately, Team LaserSETI member Doug Hagan who’s local on Maui, was able to make the long drive up Haleakala and close it by hand.

Speaking of cameras, the horz-cam on IFA1 had started misbehaving transiently. We tried remote diagnostics and troubleshooting but eventually we had to admit it wasn’t succeeding. So we’ve sent a new camera (and a new gear) to Doug, and he’ll be replacing both shortly.

Meanwhile, at RFO in California, they’re upgrading the roof surface of the observatory which our instruments sit directly on top of. We worked with observatory team to come up with a plan for them to coat the the whole roof without disturbing the position of the instruments.

The hard drive in RFO2 has given us trouble before. A year ago, we swapped out the disk, the enclosure, and even the computer itself. For a while, it was smooth sailing, but then started acting up again.

So, after replacing the disk and enclosure again, we tried the power and data cables. That was an improvement, but still not 100%. And since we can’t observe without the hard drive, anything less than perfect isn’t good enough.

We’ll be headed up to Ferguson again soon, this time with a new idea in hand, to try to put this trouble maker to bed once and for all: a solid state disk. Smaller capacity, but completely different performance, electrical, and other characteristics.

Since RFO2 wants to be an attention-hogging snowflake, here’s the work from its perspective. LaserSITTIE is, by far, my favorite piece of maintenance equipment. There’s nothing like sitting when we previously had to bend over the instrument, or kneeling on it instead of the often-sweltering roof.

We lost power at RFO from about 8am to 5pm today. Everything came back online as expected, however, and we’re observing now.

Not a packet yet from IfA. They found the problem, though: the network was damaged in our building. Physical repair completed, and (eagerly) awaiting reconfiguration to bring us back online.

It’s the best night we’ve had at RFO in at least a month. Lots of stars, clear, not too much moisture, few to no clouds. 🔭 Hooray!

At IfA, however, the saga continues. Power had been “restored” via generator, then the main network cable was fixed… and re-fixed. Then we lost power from the generator, had to replace a major breaker after ordering a replacement from off the island, and now mains power and generator are both up.

As you might have concluded by now, there was a lot of damage to the equipment on the mountain from the storm in December. It’s not clear what was from direct storm damage, power fluctuations, or other secondary causes, but clearly there’s still more to repair because… we still haven’t been able to reach the instruments, despite general connectivity being restored for a third time this month.

My hope is and endeavor will be that this is the last post about what’s not working, and the next one has only good news. Next week? Fingers crossed. Stay tuned and thanks for thinking good thoughts for us!

Just because you’re in Hawaii, doesn’t mean it’s warm! We got our first frost of the season, after being socked in and unable to observe last night.

We can see effects of cooler temperatures in California, too. One of the camera’s in RFO2 is especially prone to being sticky when cold, but fortunately we now have software to handle it automatically.

Ideally, and with most of the other cameras, it takes only 3 attempts to open the shutter all the way 3 times in a row, but last night it took 9.

LaserSETI would like the sympathize with all of the humans experiencing a heat wave today.

We spent a long time refining the thermal protection system for the instruments. Internal circulation, special heat-shedding paint, multiple exhaust fans, sunshades with embedded mirrors to reject even more heat. Days like today prove the value of that investment.

It’s 44 C (112 F) right now at Ferguson Observatory (RFO) and the cameras are the hottest I’ve ever seen them: 61 C (142 F)!

For comparison, the cameras on top of Haleakala, Maui (IFA) are a cool 16 C (61 F) right now.

Tomorrow night, May 26th, I’ll be kicking off the annual Ferguson Observatory Speaker Series. We’ll start from SETI basics, explain how LaserSETI is designed, and take questions. It’s free, it’s virtual, and they’ll give you your money back if you don’t learn anything!

RFO Speaker Series: The LaserSETI project with Eliot Gillum

Join RFO and Eliot Gillum for the SETI Institute for an informative conversation on the LaserSETI project!LaserSETI is an ambitious project to continuously scan the whole sky for laser flashes from beyond the Earth-Moon system, ostensibly from another civilization. The first two instruments in this…

As predicted by the forecast below, RFO is socked in tonight. Tomorrow night should be really good, however.

The first three rows mostly characterize moisture in the air (a lot tonight, very little tomorrow night). Seeing represents turbulence, but that doesn’t affect us much because our pixels are big, which is because our field of view (FOV) is enormous (75 degrees!). Darkness matters a lot however for the same FOV reason. And this week, the Moon rises and sets with the Sun, which makes the nights nice and dark because the Moon isn’t lighting up every tiny water droplet and mote of dust that happens to be floating around the sky.

And so this is what the sky currently looks like at Ferguson:

However, at the same time, looking at the exact same stars–but from underneath a different part of the atmosphere–IFA2 is having a great night:

Hopefully, tomorrow night it’ll stay this way in Hawaii but also look like this in California!

Turns out I did save a “before” movie of the RFO1 sunshade that wasn’t opening fully! It had been working great for a long time, but the real world is messy like that apparently. Clearly it was the humidity, cold temperatures, phase of the moon, and local unicorn population that was causing the friction.

Speaking of locals, Dan, did a phenomenal job this weekend and got the main computer within RFO2 back online! It’s been a long string of wacky issues, from the exact DC voltage it was receiving, to the firewall, to the unusual Vadj potentiometer on our DC transformer. The science cameras are still offline, but that’s probably a minor issue and we’ll hope to be observing again this week.

In related news, the hard drive on IFA2 has been flaky, so we’re also hoping to get that replaced on a similar timeline.

Yesterday, Dan and I spent all day at Ferguson Observatory (RFO). We had a spot of rain to work around, but a number of things to get done and the trip was largely successful.

We dropped off the Drobo “SneakerNet” disk array that we use to exfiltrate data from RFO because the satellite link is too slow and expensive for that. LaserSETI analyzes data in real-time, but since we’re still in the early phases of our dual observation strategy, we’ve kept pretty much every pixel of observed photons thus far. This time was special because we had a failed disk (it’s a 5-disk array), but even after replacing that disk with two different spares, we concluded that one of the ports in the Drobo itself has failed.

We soldered the vert-cam’s power lead after having previously bypassed the relay in precariously fragile but 100% reliable manner *cough*cough*zip*tie*cough*.

If you’ve been watching the live pictures from the instruments recently, you would’ve seen evidence of RFO1’s sunshade not opening fully. I think it might be related to the humidity but, in any case, we carefully applied our special Teflon lubricant meant for polycarbonate, allowing it to out-gas as much as possible before reinstalling it, to prevent any potential chemical reactions on the optical surfaces. It’s now performing flawlessly! I wish I’d remembered to save a before-and-after movie since it records a movie of every open or close, but it currently overwrites the previous one. It’s on our To Do list to save a sample, for the curious as well as to document the daily operation of this critical component.

Since it was wet, it was a good chance to observe the seals, louvers, and other ingress protection mechanisms in action. I’m pleased to report that not a drop of water could be found inside the instrument!

On RFO2, the problem had been with its external hard drive, mostly working whenever tested but failing fairly quickly under the heavy load of science observing. Having replaced the drive, enclosure, and cord, we replaced the PC itself. Long story short, Dan did a huge amount of work, figuring out that adding an internal drive didn’t work while on 12 volt DC power (but did work on 13 volt DC), that modern browsers refuse to speak the old version of SSL that our router uses, and that Windows Firewall is unreasonably insistent in blocking remote access. All of these issues took an inordinate amount of time to diagnose, particularly because they all happened simultaneously and hence had to peel the layers of failure and obstacle like an evil onion. Now that we’re pretty sure we have a complete diagnosis, we expect to have RFO2 back online within a week or so, since it will require a another trip–the 5th actually–on this issue.

Jumping to Hawaii for a minute, after a surprising but successful visit last month to Haleakala, IFA1 has performed well and taken a lot of great data. Unfortunately, IFA2 has started exhibiting similar external hard drive issues to RFO2, so we’re making arrangements to try swapping out the disk and enclosure, as well as create some software backup procedures to allow some observing when this symptom appears, based on our learnings from trying to do so with RFO2.

For those of you who followed the Mars Exploration Rovers, I think you’ll understand why I feel like both RFO1 and IFA1 are like Opportunity, and RFO2 and IFA2 are like Spirit. After operating on Mars for more than a year, the NASA team noted that everything came easily to Opportunity, while Spirit had difficulty after difficulty. It’s especially eerie that Opportunity landed first, and the LaserSETI analogs are RFO1 and IFA1.

See you again soon!