Coding Relic - Latest Comments in Soft Errors Are Hard Problems

Re: Soft Errors Are Hard Problems

Olivier — Wed, 08 Oct 2014 19:29:28 -0000

Hi Baruch

The first step to take to understand the source of errors is try to get a physical bitmap of errors location in your DIMM and plot them on a relative map centered on the nearby bump (I'm assuming here that you are using flip chip package). This map of errors might tell you a lot about the origin of the errors. Keep in mind that alpha particles from natural radioisotopes won't travel beyond 100 microns. So you are correct, if alpha particles is the problem, they have to come from within the packaging. Check out the materials in the vicinity of the die. These materials can be tested individually in an alpha counter, that's usually how it's done.You'll have to talk to your package house and ask if they change material from one chip lot to the other since you're seeing chips with several errors and others that are error free. Try to track manufacturing lots as another clue. It is correct to state that testing the whole package for alpha particles emission won't help.
Cosmic ray (high energy neutrons) and thermal neutrons (only if there's abundance of Boron 10 in the die and packaging) are particles external to the chip. Don't worry about betas.

Olivier

Re: Soft Errors Are Hard Problems

DGentry — Tue, 07 Oct 2014 09:49:09 -0000

Alpha particles are so large that they tend not to penetrate well, so the packages of nearby chips will protect them from alpha particles emitted elsewhere within the room.

Beta particles (electrons) and cosmic rays can penetrate the package of chips, but those are from the solar wind not radioactive decay of anything local.

I'm not sure that a radiation counter will detect alpha particles from the packaging materials. The situation I've seen is with the bonding agent emitting alpha particles, which is a "goo" inside the package which cushions the die and the wires between the silicon die and the pins on the package. The particles which hit the silicon die can cause errors, while the plastic of the package surrounding the goo would tend to absorb the rest.

Re: Soft Errors Are Hard Problems

Baruch Even — Tue, 07 Oct 2014 04:24:23 -0000

I see plenty of soft errors in DIMMs and PCIe devices. The distribution is such that some components see no soft errors and others are seeing plenty. I believe that the external forces (various particles) are not even remotely relevant and it points me more to faulty devices.

The theory of radioactive components is super interesting in that regard and I'm wondering if there is a way to check for it. What sort of options do I have to monitor for radioactive emissions in devices? Will they mean much if I can only test outside the cases, besides the racks? Are the failures expected to happen in the device with the problematic compound or will it affect (more weakly) nearby devices as well?

Just noticed the article is 5 years old... doh. I still wonder about my above questions if you care to think about that.

Re: Soft Errors Are Hard Problems

DGentry — Wed, 14 Oct 2009 18:50:57 -0000

Thank you for the detailed information!

Re: Soft Errors Are Hard Problems

Olivier — Wed, 14 Oct 2009 18:23:21 -0000

Very good article Denton, thanks!

Actually cosmic ray don't come from the Sun, but from outer space. It seems counter intuitive, but when there's a solar irruption, it results in less cosmic rays hitting the ground.
Some the industry's main concerns right now are:
- multi bit upsets in memories (mostly SRAM): one particle creates several bitflips. As you know, most advanced ECC detect two and correct one error per word. Memory architecture should be carefully taken care of (using inteleaving or scrambling) to avoid multi bit flips within the same word. But as you say, it might not be feasible for small memory instances spread out within the ASIC.
- more dramatic that the previous effect: the increased sensitivity of FF and registers to Soft Error at smaller technologies (we're talking 65nm here). Mitigation techniques are not as obvious as for memories and can cost a lot of area and power to the designer's budget.
- assessing derating: not all error will actually affect the functionality of the circuit. Either the upset occurred in a part of the circuit not involved in the ongoing function, or the errors has been masked along its propagation path by either timing or logic constraint. Understanding derating for large and complex chips is not a simple problem!

A good overview of the source of the problem can be found in this quick youtube file: http://www.youtube.com/watc...

Re: Soft Errors Are Hard Problems

Some Dude — Fri, 18 Sep 2009 13:45:34 -0000

Here's a reference: "How Cosmic Rays Cause Computer Downtime"
http://www.ewh.ieee.org/r6/...
See slide 34.

Search for [stuck latch cosmic ray] on Google.

Re: Soft Errors Are Hard Problems

DGentry — Fri, 18 Sep 2009 06:49:04 -0000

That is very interesting, I had not heard of cases requiring a power
cycle to recover after a soft error.

Re: Soft Errors Are Hard Problems

Some Dude — Fri, 18 Sep 2009 01:19:20 -0000

One particularly horrible failure mode I've heard about is a "stuck bit" where the value in the bit cannot be changed after a particle hits it, requiring a power cycle.

I work for a company that designs and sells equipment that contains various ASICs supplied by third parties. We test our products and the ASICs they contain by renting time in a facility that has a source of high energy particles (e.g. neutrons), placing our products in the path of the particle beam, and then operating the product with the beam aimed at various ASICS. The results are interesting.

Perhaps this is an extreme way to test the code that handles parity errors in various memories on the ASICs, but until you do it how do you know that the ASIC either detects parity errors, or in the case of ECC, corrects them? Some ASIC vendors are better than others.