Back

Failure analysis of the Arecibo 305 meter telescope collapse

235 points1 yearnap.nationalacademies.org
magicalhippo1 year ago

In the summary they state:

All the reported experimental zinc electroplasticity (EP) data were developed at current densities orders of magnitude higher than those possibly present in the Arecibo Telescope but measured in laboratory experimental periods that were orders of magnitude shorter than the telescope's socket zinc service.

There are no reported experimental data concerning low-current, long-term EP, which the committee has lumped together under the term "LEP", affecting zinc's creep mechanisms over decades.

The timing and patterns of the Arecibo Telescope's socket failures make the LEP hypothesis the only one that the committee could find that could potentially explain the failure patterns observed.

Accelerated aging is, as far as I know, pretty much standard in the industry. Nobody can wait 20 years to find out if a certain material is good enough or not.

However, the real failure seems to be the lack of urgency when they signs started to show up:

Upon reflection, the unusually large and progressive cable pullouts of key structural cables that could be seen during visual inspection several months and years before the M4N failure should have raised the highest alarm level, requiring urgent action. The lack of documented concern from the contracted engineers about the inconsequentiality of the cable pullouts or the safety factors between Hurricane Maria in 2017 and the failure is alarming.

deskr1 year ago

> The lack of documented concern from the contracted engineers about the inconsequentiality of the cable pullouts or the safety factors between Hurricane Maria in 2017 and the failure is alarming.

This is crazy. Basically cables were pulling out for months and years and no one raised the alarm? In many industries that could be a criminal or career ending malpractice. Are the "contracted engineers" liable?

8bitsrule1 year ago

Hmmm. I was just looking this over a couple of days ago.

On Nov 19 2020, NPR was already reporting that "Sean Jones, Director for the Mathematical and Physical Sciences Directorate at the NSF, said the telescope will be dismantled. ... after receiving ... the engineering assessments, we have found no path forward that would allow us to do so safely..." https://www.npr.org/2020/11/19/936677582/world-renowned-arec...

In retrospect, it seems that, -at its age- the design got in the way of repairs. The scope made possible a lot of great finds in its day, but it'd become too weak to save.

kreims1 year ago

You go to school and learn that 2+2=4. You get a consulting job and learn 2+2= whatever the client says it is.

I wouldn’t jump to the conclusion that the consulting engineer was incompetent. Sometimes a bureaucrat tells you to sharpen your pencils and come back with the answer That fits the budget they have if you want to keep your professional services contract going.

foxglacier1 year ago

Yea and that's a violation of professional ethics. Being corrupt or working for a corrupt organization in no way justifies it. Professionals are trusted by society to act responsibly even when it's against the interests of their clients or employers.

+1
kreims1 year ago
rovr1381 year ago

Specially licensed professionals.

cssanchez1 year ago

They specifically stated a lack of documented concern by the engineers. First thing you learn as a contractor is CYA.

magicalhippo1 year ago

In chapter 5 they go into how the ownership was transferred to the University of Central Florida (UCF) at the start of 2018, after the hurricanes in 2017.

It seems unlikely that UCF had adequate time and resources to review and understand the Arecibo Telescope's original 1963 design, the 1974 upgrade, the structural inspection and maintenance records produced for nearly 50 years, [...] and the key factors, such as the wire breaks and cable pullout of the sockets and their significance on the strength and integrity of the structure.

The measured cable pullout may have appeared "normal" to [the UCF staff] and was not on their radar as signs of structural distress. The lack of concern may be because a small cable pullout was present from the beginning, and no one in authority had previously raised an alarm

But yeah, seems weird no-one of the contractors tried to raise an alarm.

hinkley1 year ago

I bet there’s a r/MaliciousCompliance in Spanish out there somewhere about how UCF cancelled existing maintenance contracts and hired new companies to keep an eye on the equipment, and did it so rudely that the old company didn’t hand over important warnings like this.

Discontinuity could explain problems like this readily.

londons_explore1 year ago

Since the design had a factor of safety of 2, and no other cables exhibited pull out at such high safety factors, checking such things might not even have been on the checklist.

bsder1 year ago

> Basically cables were pulling out for months and years and no one raised the alarm?

And what if they did? Then what? My guess is the conversation went something like this:

"Okay, cables are pulling out. Raising this issue will not magically make money appear to fix it as nobody wants to fund this. Tell me, how much do you like your job? If you flag this, you may lose your job directly and if the project gets shut down you may lose your job indirectly. So, how about we bury this as much as possible, cross our fingers and bank as much money as possible in the meantime, eh?"

foxglacier1 year ago

It creates a paper trail that transfers liability from the individual to their employer, which then incentivizes them to make it safe somehow, even if that means shutting down and demolishing the whole thing. Better that than somebody getting killed.

LorenPechtel1 year ago

But that would have required spending money. And why would the engineers be liable? They weren't being paid to fix it.

renewiltord1 year ago

[flagged]

feoren1 year ago

As someone who works in a company with 10,000+ professional engineers*: yes, yes they do make mistakes. Despite what you seem to think, they are actually human beings. Good processes understand this and have multiple layers to catch and correct mistakes. But it's hard to fight limited data. Plenty of engineering decisions are still made based on numbers in photocopied tables from two or three studies from 50 years ago, each having a dozen or two data points. It's surprising how limited some of the data is that these Professional Engineers are basing their decisions on.

*Edit: okay, I don't know exactly how many are officially PEs vs. junior engineers or something, but at least that many are "PE-track".

Brian_K_White1 year ago

They are not saying engineers don't make mistakes.

They are complaining that developers who can crank out garbage with no consequences and take home $200k+ aren't respected as much as civil engineers who have to get licences and suffer accountability and take home less.

The poor baby.

LaffertyDev1 year ago

> Professional Engineers do not make mistakes. They are real engineers.

My _literal_ engineering 101 case studies on "engineering failure":

https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge_(1940)

https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse

https://en.wikipedia.org/wiki/I-35W_Mississippi_River_bridge

Are all failures traced down to the underlying design and engineering. Granted, they are famous because they are rare. I almost linked the Columbia disaster but that is sufficiently murky between engineer and management.

Stop being elitist.

astrange1 year ago

This one wasn't long ago.

https://en.wikipedia.org/wiki/Florida_International_Universi...

Btw, the cult of the professional engineer is apparently much stronger in Canada, to the point I've actually impressed women by calling myself a software engineer despite the lack of iron ring. I wonder if this makes up for them being paid peanuts?

mden1 year ago

I think people are missing that you're being satirical (maybe there is a better word for it?). Unless I'm the one being wooshed.

+1
rootusrootus1 year ago
haccount1 year ago

Anyone in the field will likely say that they wouldn't trust half of their colleagues or themselves.

It's pretty much how professional prestigious positions work. Beneath the gilding there's duct tape repaired with super glue.

+2
renewiltord1 year ago
8bitsrule1 year ago

The Minneapolis I-35W bridge that collapsed in 2007 supposedly had bad gussets. Yet it had barely passed previous inspections; in 2005 it was deemed to have met "minimum tolerable limits to be left in place as it is".

In 2005 the US said it was in the same condition as 75,000 other bridges in the US: 'structurally deficient', which is also what the US called it in 1990. I'd guess that the same quality of decisions brought down Arecibo. Gotta pay the piper.

mNovak1 year ago

I wouldn't overinvest your confidence in these licenses and certifications. A "Professional Engineer" is just a Bachelors level engineer who took one extra test. It's extremely common for MechE and CivE. As far as certifications go it's not quite the same as, say, passing the bar exam or finishing med school.

+1
albrewer1 year ago
dylan6041 year ago

But it is different than that cert you receive after taking a 2 week bootcamp on Python.

tiahura1 year ago

I'm not sure by your tone if this is sarcasm, but it has been shown by formal proof that if something failed the root cause must be an MBA.

mkesper1 year ago

And further:

The reliance by the consultants (before and after the first cable failure in 2020) on a perceived allowable pullout of one-sixth of the cable diameter, which should only be seen at loading at 80 percent of ultimate cable strength, does not align with the AASHTO M 277 standard guidance. The committee, therefore, disagrees with the suggestion made in the Thornton Tomasetti, Inc. (TT) 2022 report, Arecibo Telescope Collapse: Forensic Investigation,13 to use the D/6 limit as a threshold for slip monitoring.

UniverseHacker1 year ago

As a sailor that uses similar fittings in sailboat rigging… the idea of not treating a cable pullout from a terminal as an emergency is absurd. If that happened on a sailboat rig I’d assume it had a remaining strength of zero and will fall in seconds or less- and take the most rapid possible action to keep the structure from coming down by deloading the shroud and rigging an emergency replacement.

Iolaum1 year ago

I would be curious if there are any accounts/reports of people working there - especially engineers - that could verify the claims of no alarm being raised when the first structural failures started showing up. It is often the case that people on the ground report such things but when "reports" are written for the higher ups, facts get "massaged" according to expectations.

photon_rancher1 year ago

I was on the suspended section about a year before collapse - everyone could tell it was about to fail. Visible stress in rusty beams, old cables, and failing concrete. I think it just came down to underfunding at the end of the day - there was no money available to ask for even if the alarms were louder.

And for most applications it was obsoleted by the larger new telescope in china, so there was no large push to turn that ship around from the wider scientific community.

77pt771 year ago

Bearers of bad news get shot.

Everyone working in large corporations knows this.

lazystar1 year ago

being a bearer of bad news is a sign that you are not disagreeing and committing.

nick34431 year ago

Sounds like it was a blind spot for the engineers due to the normally bulletproof service history of zinc spelter cable terminations in structures.

potato37328421 year ago

>normally bulletproof service history of zinc spelter cable terminations in structures.

They're relatively new, they're relatively expensive (which saves them from being used in the worst of the applications) and like most things inherent to steel cable and whatnot they tend to get employed in situations where safety factors are generous. Don't count your chickens before they hatch.

nick34431 year ago

I am not an expert, I read this in the report: The long-term zinc creep failure of the Arecibo Telescope sockets and the subsequent cable pullout has never been documented elsewhere despite a century of zinc-filled cable spelter sockets use. 17 The type, size, length, and fittings of the cables used in the Arecibo Telescope (whether the original cables constructed in the 1960s or the auxiliary cables installed in the 1990s) were catalog-selected items, not at all unusual, with decades of proven performance. Exten-sive structural modeling of the Arecibo Telescope, independently confirmed by laser cable sag surveys, validated that under all static and cyclic loading conditions, the cable loads barely exceed half the nominal cable strength. While PLC can occur at stresses well below yield, such a creep failure has never been reported in spelter socket zinc.

londons_explore1 year ago

> Exten-sive structural modeling of the Arecibo Telescope, independently confirmed by laser cable sag surveys, validated that under all static and cyclic loading conditions, the cable loads barely exceed half the nominal cable strength.

I have a feeling such measurement and modelling might have been wrong.

It's easy to miss some oscillation mode, particularly high frequency longitudinal wave's in the cables, which couldn't be picked up by a laser survey due to the kilometers per second these waves travel.

+1
hinkley1 year ago
thimkerbell1 year ago

[flagged]

magicalhippo1 year ago

I think absolutely not, for several reasons. First off, the report mentions this failure of the zinc filled sockets never been reported elsewhere before.

Secondly, even if it had, the report mentions that written documentation of inspections and so on was spotty and inconsistent.

That said, if the failure mode had been published before, and inspections and maintenance was written down in a consistent and detailed manner, then perhaps an LLM-based tool could have linked the two. I'm thinking of a RAG-like search thing that compared inspection reports with published documents. Alternatively just assisting a human in finding relevant published reports.

Wololooo1 year ago

Even if I am not a radio astronomer, this instrument was unique in nature and still in use for very specific observations and until the very end of its operation was able to deliver data. It is disheartening to learn that it was indeed due to gross maintenance negligence (as assumed originally by many) that the final fatal failure occurred, when potentially the structure could have been salvaged.

magicalhippo1 year ago

Losing Arecibo was, as I understand it, a big blow to the NANOGrav experiment, which is looking for very low frequency gravitational waves by measuring pulsar timing variations[1].

Albeit having limited field of view the Arecibo Telescope was very sensitive[2], and so could see pulsars that the other telescopes they used could not. And the longer they could collect data from a set of pulsars the lower frequency waves they could probe.

[1]: https://nanograv.org/news/15yrRelease

[2]: https://pirsa.org/20100068 Moving Closer to a Detection of nHz-frequency Gravitational Waves with NANOGrav (Arecibo details at around 10:20)

axus1 year ago

My uninformed recollection from when the warning signs were raised, is there was not even budget to perform emergency maintenance, and they were resigned to the inevitable collapse.

jdhwosnhw1 year ago

I worked and did research atobservatory for my PhD - you are exactly correct. It was well known to the scientists and onsite engineers that a collapse was imminent, for years. Ironically, prior to the collapse the agency responsible for funding performed a study to see would need to do to stop supplying money to the observatory, and found that due to environmental impact, it would actually be more expensive to stop funding. After that study we all suspected that the agency was just waiting for decay or a natural disaster to do the dirty work for them

trogdor1 year ago

Would you link the study you are referring to?

WillPostForFood1 year ago

That's not what the report says. There was ~$14 million allocated for repairs in 2-18 after Hurricane Maria, but the cable sockets weren't identified as needing repairs until the first one failed. The problem wasn't the money, it was the failure to identify the problem. See page 28/29.

SirMaster1 year ago

What was unique about it compared to this one for example?

https://en.wikipedia.org/wiki/Five-hundred-meter_Aperture_Sp...

superkuh1 year ago

Arecibo did high power transmit for radar imaging of asteroids/comets/the sun/etc and ionosphere studies, among other things. FAST is not capable of transmit (it was not designed for it and the suspended receiver mass is much lower). Also, FAST has a RF-noise emitting theme park built just outside of it's grounds.

The uniqueness of Arecibo was this transmit coupled with such a large aperture (so small pattern on sky).

dylan6041 year ago

> Arecibo did high power transmit for radar imaging of

the Sun? that's something I never thought about, and now that I'm thinking about it, still seems very interesting. It seems like there would be so much noise radiating from the burning ball of fusion that radar signals would just get absorbed/lost. And now I've sat here for 5 minutes doing nothing but imagining this.

superkuh1 year ago

The sun is bright, but it's not bright all over in the lower VHF/UHF ranges. The quiet sun has a mimimum radio emission at about 50 MHz and then linearly (mostly) increases up to 10 GHz (and more). So at the lower frequencies radar echos can be detected as long as there are no active flares/etc going on. http://a.superkuh.com/radio-sources.jpg

Although it's emission (due to the transparence of it) is not just a disk but sometimes increases on the limb depending on frequency, http://a.superkuh.com/solar-disk_relative-brightness_vs_sola...

Here's a diagram of how the ray paths from an Earth based radar reflect from the soft target that is the sun, http://a.superkuh.com/what-solar-radio-looks-like.jpg

SirMaster1 year ago

Thanks! To a layman it seemed like they were really similar devices.

bell-cot1 year ago
ctippett1 year ago

Practical Engineering on YouTube has an episode covering the collapse of this structure[1]. It's very good and worth a watch.

[1] https://youtu.be/3oBCtTv6yOw

zinekeller1 year ago

Ah, no, like most journals (although NAP is not stricly a journal, PNAS is), there isn't a direct PDF link (OP: try to click that on a clean browser and you'll be disappointed).

pantulis1 year ago

What an imposing document: the details are overwhelmingly detailed, it really does justice to the marvel of engineering that the Telescope was.

jschrf1 year ago

Perhaps it was a response to the Arecibo message, and the response was "quiet, they'll hear you".

loa_in_1 year ago

Or what we found is the Vogon and their poetry induces beaurocracy issues on the receiving side

kazinator1 year ago

Pages and pages of name-dropping fluff before any substance. That is telling.

dudeinjapan1 year ago

For England, James?

nonfamous1 year ago

From the summary:

>>> The only hypothesis the committee could develop that provides a plausible but unprovable answer to all these questions and the observed socket failure pattern is that the socket zinc creep was unexpectedly accelerated in the Arecibo Telescope's uniquely powerful electromagnetic radiation environment. The Arecibo Telescope cables were suspended across the beam of "the most powerful radio transmitter on Earth." The other investigations failed to note several failure patterns and provided no plausible explanation for most of them. To answer these questions with empirical evidence instead of only the inferences that can be made from the existing data, a more comprehensive and widespread forensic analysis of "good" and "bad" socket workmanship and the low-current, long-term effect on zinc creep is required.

shagie1 year ago

(There was a comment that I was replying to about the realization that this was a transmitter and not just a receiver that was deleted ...)

It was both.

The Arecibo message was transmitted from there - https://en.wikipedia.org/wiki/Arecibo_message

> The entire message consisted of 1,679 binary digits, approximately 210 bytes, transmitted at a frequency of 2,380 MHz and modulated by shifting the frequency by 10 Hz, with a power of 450 kW.

There's also the field of radar astronomy (not radio astronomy) https://en.wikipedia.org/wiki/Radar_astronomy where you bounce of a signal off of an object in the solar system (such as an asteroid)

> Radar provides the ability to study the shape, size and spin state of asteroids and comets from the ground. Radar imaging has produced images with up to 7.5-meter resolution. With sufficient data, the size, shape, spin and radar albedo of the target asteroids can be extracted.

> ...

> In August 2020 the Arecibo Observatory (Arecibo Planetary Radar) suffered a structural cable failure, leading to the collapse of the main telescope in December of that year.

> There is one remaining radar astronomy facility in regular use, the Goldstone Solar System Radar.

userbinator1 year ago

DC can definitely cause a lot of electrolytic damage, but AC usually doesn't. Perhaps something was acting as a diode (these are easily formed from natural processes) and rectifying the signal.

whoopdedo1 year ago

The old "Foxhole radio" trick of using a rusty razor blade as a diode comes to mind. Some metal oxide coatings are semiconducting and under the right conditions will form a Schottky barrier.

Syonyk1 year ago

Fortunately, more novel research comes out of "Huh, that's weird... why did that happen?" than out of things behaving as expected. Though this one does seem rather baffling.

ricksunny1 year ago

I'm interested in hearimg the most speculative hypotheses as to why Arecibo Telescope's socket zinc creep proceeded so quickly compared to expectation.

alexnewman1 year ago

Puerto Rican here. Everything on our island is like that. Preventive maintence is ignored and our insurance companies go bankrupt every big hurricane

wileydragonfly1 year ago

Everything in the Caribbean is like that if you want to be honest.

rovr1381 year ago

> if you want to be honest.

They might only know about Puerto Rico thus limiting themselves to just that.

BobaFloutist1 year ago

If we ever make you a state proper hopefully that will offer an opportunity for that to change.

wileydragonfly1 year ago

They will never be a state. They’re in the sweet spot of self governance and endless federal subsidies. Despite what Wikipedia will tell you, English is not prevalent. Statehood will never happen and I’ve been listening to that drum for over 30 years.

rovr1381 year ago

> They’re in the sweet spot of self governance and endless federal subsidies

Let's be clear, this is not decided by Puerto Ricans. Congress ultimately decides this. How they decide is up to them. Until now, they haven't done much.

mmooss1 year ago

> endless federal subsidies

Wasn't PR denied help with hurricane repair and rebuilding?

astrange1 year ago

Jones Act restrictions on shipping are a lot worse for them than any amount of subsidies. Makes it much harder for ships to come from nearby non-US islands and deliver anything.

alexnewman1 year ago

Half the people speak English here

ordu1 year ago

Aren't you afraid that US insurance companies will go bankrupt?

rovr1381 year ago

If a corporation can't provide a service, they should limit themselves and not provide it, or go for bankruptcy after making bad decisions.

ljsprague1 year ago

Statehood? Why not independence?

BobaFloutist1 year ago

Because they voted for statehood in a referendum.

rovr1381 year ago

FWIW, they have all been flawed.

On the other hand, is there a perfect one?

leptons1 year ago

How would independence be better?

dylan6041 year ago

Because insurance payouts in Florida are so spectacular? Please, don't lead them along with false hope! /s

sideshowb1 year ago

Analysis of recorded audio contains the phrase "for England, James?"

fijiaarone1 year ago

Finally a cogent post. Kids these days.

ionwake1 year ago

[flagged]

KPGv21 year ago

> in developing country

Arecibo is located in the United States. Unless Puerto Rico declared independence recently, that is.

Snide remarks about Texas aside, would you say SpaceX is located in a developing country? Because you'd be hard-pressed to argue that Arecibo was located in a less developed place than SpaceX's South Texas facility.

This is Arecibo: https://i.ytimg.com/vi/Dd6pCS1IBN4/hq720.jpg?sqp=-oaymwEhCK4...

This is the nearest town to SpaceX's facility: https://archleague.org/wp-content/uploads/2021/03/AMRT_TX00_...

ggm1 year ago

It was a bad application of country.

How do you feel about "An economically challenging location with weaker supply chain, infrastructure and potentially lower compliance to standards in engineering."

The GDP of Puerto Rico overall in 1960 was $1.6b. You would expect about 20x growth over the time window, and its more like 100x.

No disrespect to Puerto Ricans intended.

omegaworks1 year ago

[flagged]

ggm1 year ago

Not denying this, but I am unsure that plays to why Arecibo might have had baked in issues. I was trying to talk to the economics behind potentially weaker construction than expected.

I'm told Brazilia has mighty fine buildings, but there are maintenance issues with doing Niemeyer's vision in the Jungle. Le Corbusier's designs for New Delhi certainly had their challenges. So, I don't know that "colonialism" is the root problem here but perhaps it is?

dylan6041 year ago

> Snide remarks about Texas aside,

which is funny as taking Texas alone, it would have a top 10 economy in the world. so to the people running it making decisions that those snide remarks are pointed, it looks to them like they are doing just fine thank you very much.

and this is why my vote today goes to nothing.

leptons1 year ago

Lol, no... as someone who has actually been to Aricebo, that is nowhere near where the telescope is in Puerto Rico. You showed a postcard from the coastal area of Aricebo in your link. The telescope is a very long and winding road away from there, it's very remote, there's nothing really near there at all.

Your links are disingenuous at best, you show possibly the best part of Puerto Rico against the worst part of TX.

rovr1381 year ago

On the other hand, the location is part of what made the telescope good. You'd have too much interference in a city.

defrost1 year ago

Starlink has since expanded interference globally.

No longer can remote areas feel left out from urban radio spectrum interference.

stavros1 year ago

Isn't that what the GP said?

dark-star1 year ago

By some metrics, the US is considered a developing country. For example the U.N. places it at rank 46, below such countries as Romania, Cuba, Bulgaria, Albania:

https://dashboards.sdgindex.org/rankings

Edit: you US people can downvote me as much as you want, that won't change the facts :-D

rootusrootus1 year ago

I can find some ranking metric that your country will be low on the list, I guarantee it. On the other hand, what metric can I find that puts it at #1? There are a few of those that work for the US.

dark-star1 year ago

sure, if you're looking for "any" ranking... but this is not some random dude's "top 100 countries to visit" or something. It's the U.N.'s official list of developed countries. That puts some credibility to it, and it is definitely relevant in this topic since people were discussing whether the US counted as developed country or not. That is certainly more credible than finding "some list where [my] country is low on". I'm sure they are out there but I don't think you can find any list by the UN or any other credible world-wide organization.

Please, feel free to prove me wrong!

Oh, and what list(s) did you have in mind where the US is #1? I can't think of many (at least not positive ones)

selfsimilar1 year ago

It's in Puerto Rico, a US territory that has toyed with formally joining the union as a proper state. This is NOT a developing country.

Filligree1 year ago

[flagged]