Archive for March, 2012


Posted in Manufacturing Improvement with tags , on March 22, 2012 by manufacturingtraining

One of my clients, knowing my interests in high-quality watches and precision production operations, sent this very cool Tag-Heuer video to me…

Enjoy; I know I sure did!


KU Online Courses Scheduled for 2012-2013

Posted in Manufacturing Improvement with tags , , , , , on March 21, 2012 by manufacturingtraining

ManufacturingTraining and the University of Kansas have finalized the course schedule for our next series of six online Manufacturing Optimization courses:

  • Delivery Performance Improvement:  21 August 2012
  • Cost Estimation:  16 October 2012
  • Industrial Statistics:  8 January 2013
  • Quality Management:  5 March 2013
  • Root Cause Failure Analysis:  30 April 2013
  • Cost Reduction and Optimization:  25 June 2013

Each course is 3 weeks long and the University of Kansas will grant Continuing Education credit.  We’ll meet for online lectures twice each week, with interactive assignments and discussion board activities following the lectures.  We’ll be posting more information here and on the website in the near future, so stay tuned for more information on this exciting new professional education opportunity!  In the meantime, if you want advance information on pre-enrolling, you can do by shooting an email to

State of the Art?

Posted in Manufacturing Improvement with tags , , , , on March 14, 2012 by manufacturingtraining

Back to the photo I showed a week or so ago…

An Apache Rotor Blade Bond Joint

The photo above shows a bonded section of an AH-64A Apache helicopter main rotor blade in the area where you see the blue Dykem. It’s where the blade manufacturer and the Army experienced numerous disbonds, and it’s the problem the blade manufacturer had to solve.

An AH-64A Apache at Fort Knox, Kentucky

Before delving into the failure analysis, let’s consider the Apache rotor blade’s design and its history. The Apache helicopter has what are arguably the most advanced rotor blades in the world. They can take a direct hit from a 23mm ZSU-23/4 high explosive warhead and remain intact. During the Vietnam war, a single rifle bullet striking a Huey blade would take out the helicopter and everyone on board. When the Army wrote the specifications for the Apache, they wanted a much more survivable and much less vulnerable blade.

Vietnam-Era Huey Helicopters 

The Apache helicopter prime contractor designed a composite blade with four redundant load paths running the entire rotor blade length. The blade’s advanced design uses titanium, special stainless steels, and honeycomb, but those four redundant load paths were the key to its survivability. If one section of the blade took a hit with a 23mm warhead detonation, the three remaining load paths held the blade together. That actually happened once during the first Persian Gulf war, and the Apache helicopter made it back to its base. It’s an awesome design, but it had a production weakness.

Apache Rotor Blade Sectional View Showing Four Spars 

Let’s also consider the nature of the Apache production approach. Three entities are important here: The US Army (the Apache customer), the prime contractor (who designed the helicopter and its blade), and the blade manufacturer. The blade manufacturer was a built-to-print manufacturing organization. They built the blade in accordance with the helicopter prime contractor’s technical data package.

The manufacturing process consisted of laying up the blade in a cleanroom environment using special fixturing, bagging the blade components in a sealed environment, pulling a vacuum on the bag, transporting the blade to an autoclave, and then autoclave curing.  The autoclave cure was rigidly controlled in accordance with the prime contractor’s specification.

During production startup, many of the blades had a high rejection rate after the autoclave cure. The bond joint (where the stainless steel longitudinal spars overlapped, as shown in our photo above) frequently disbonded.  Eager to get the blade into production, the blade manufacturer, the prime contractor, and the Army pushed ahead.  They believed that due to the “state of the art” nature of the Apache blade’s design, a less-than-100% yield was inherent to the process.  The disbond failures continued into production.  To cut to the chase, the blade manufacturer continued producing the blade for the next decade with an approximate 50% rejection rate.  To make matters worse, blades in service on Apache helicopters only had about an 800-hour service life (the specification called for a 2,000-hour service life).

By any measure, this was not a good situation.  The blade manufacturer had attempted to find the disbond root cause off and on for about 10 years, with essentially no success. While not happy, the Army continued to buy replacement blades, and they continued to send blades back to the prime contractor from the field for depot repairs.  The prime contractor sent the blades back to the blade manufacturer.  In retrospect, neither the prime contractor nor the blade manufacturer were financially motivated to fix the disbond problem.

After a change in ownership, the blade manufacturer realized the in-house blade disbond rework costs were significant. The new management was serious about finding and correcting the blade disbonds. Using fault-tree-analysis-based root cause analysis techniques, the company identified literally hundreds of potential failure causes. The failure analysis team found and corrected many problems in the production process, but none had induced the blade disbonds.  The failures continued. Surprisingly (or perhaps not surprisingly, considering the lively spares and repair business), the helicopter prime contractor did not seem particularly interested in correcting the problem.

After ruling out hundreds of hypothesized failure causes, one of the remaining suspect causes was the bondline width where the longitudinal spars were bonded together. That’s the distance marked on the macro photo with scribe marks on the blue Dykem (the photo I showed you earlier, and the one at the top of this blog entry).  During a meeting with the helicopter prime contractor, the blade manufacturer asked if the bondline width was critical. The prime contractor, evasive at first, finally admitted that this distance was indeed critical. The prime contractor further admitted that if the distance was allowed to go below 0.440 inch, a disbond was likely.

Armed with this information, the blade manufacturer immediately analyzed the prime contractor’s build-to-print rotor blade drawings.  To their surprise, tolerance analysis showed the blade’s design allowed the bondline width to go as low as 0.330 inch. The blade manufacturer inspected all failed blades in house, and found that every one of the failed blades was, in fact, below 0.330 inch.  It was an amazing discovery.

The blade manufacturer immediately asked the prime contractor to change the drawings such that the bondline width would never go below 0.440 inch. The prime contractor refused, most likely fearing a massive claim from the blade manufacturer for a technical data package deficiency spanning several years.  The prime contractor instead accused the blade manufacturer of a quality lapse, stating that this was what allowed the bondline width to go below the 0.440 inch dimension.

The blade manufacturer explained the results of their tolerance analysis again, and once again pointed out that the blade design permitted the disbond-inducing condition. When the prime contractor refused to concede the point (and again accused the blade manufacturer of a quality lapse), the blade manufacturer took a different tack.  As repair facility, the blade manufacturer had blades in house for depot repairs from various points during the Apache program’s life (including the 12th ever blade built, which went back to the first year of production). All of these earlier failed blades had the same problem: They conformed to the technical data package, but their bondline width was below 0.440 inch.

The blade manufacturer, faced with an ongoing 50% rejection rate, decided to hold the blade’s components to much tighter tolerances than required by the prime contractor’s technical data package. By doing so, the blade manufacturer produced conforming blades with bondline widths above 0.440 inch. After implementing this change, the blade disbond rejection rate essentially went to zero.

So what’s the message here?  There are several:

  1. Don’t accept that you have to live with yields less than 100%. You can focus on finding and fixing a failure’s root cause if you are armed with the right tools. Don’t accept the “state of the art” argument as a reason for living with ongoing yield issues.
  2. Don’t think that simply because the product meets the design (i.e., there are no nonconformances) that everything is good. In many cases, the cause of a recurring failure is design related. Finding and addressing these deficiencies is often a key systems failure analysis outcome.
  3. If you are a build-to-print contractor, be wary.  The design agency may not always be completely open to revealing design deficiencies.
  4. It’s easy to become complacent and accept a less-than-100% yield as a necessary fact of life. In some cases, the yield is not just a little below 100%; it’s dramatically less than 100% (as occurred on the Apache rotor blade production program for many years).
  5. There are significant savings associated with finding and fixing recurring nonconformances. You can do it if you want to, and if you have the right tools.

You know, the wild thing about this failure and the Mast Mounted Sight failure mentioned a week or so ago is that the two companies making these different products were literally across the street from each other.  The Mast Mounted Sight was a true show stopper…it stopped production and it probably delayed the start of Operation Desert Storm.  The Apache blade didn’t stop production…it was just a nagging, long-term, expensive rework driver for the Army and the blade manufacturer.  Which one was more expensive?  Beats me, but if I had to guess, I’d guess that the ongoing (but non-show-stopping) nature of the Apache rotor blade failures carried a heftier price tag.

Do you have recurring inprocess failures that you’d like to kill?  Give us a call at 909 204 9984…we can help you equip your people with the tools you need to address these cost and quality drivers!

Selecting a school…

Posted in Uncategorized with tags on March 10, 2012 by manufacturingtraining

A quick input today, folks.   I’ve been an adjunct faculty member at our local university’s College of Engineering for more than 20 years, and a manufacturing associate recently sent a question to me I’d never considered before:

Our grandson has several invitations to visit colleges, and I was wondering if you could send a short list of the most important questions he should ask.

That’s a wonderful question.   Just in case any of you have a similar situation with a child or grandchild going through the college selection process, let me share with you (from the perspective of an insider) what I think any student ought to know when considering a college:

Question 1:  How long does it take to get a Bachelor’s degree?

Many schools have stretched this out to 5 or 6 years, which I think is deplorable.   It ought to take 4 years to get a 4-year degree.  If a student wants to take longer because he or she has to work to pay their way through school, that’s okay.  If the university makes it difficult to get required classes, though, that’s shameful.

Question 2:  In my field of study, what’s the placement rate at graduation?

Many students select fields of study that are literally worse than useless in the sense that their majors hurt them (rather than help them) when they seek employment.  I’m not saying that these fields shouldn’t be taught; I am saying that students need to think about what they can do with their degree.   How well the school prepares a student for finding a job should be a critical factor in the selection process.

Question 3:  What percent of the faculty has practical work experience outside the classroom?

Teachers who have never done anything except teach can’t bring real-world experience to the classroom.   Many schools use adjunct faculty members with full-time industry jobs, and teachers who consult to industry outside the classroom.   This kind of practical experience adds an important dimension to any education.

Question 4:   Which department teaches writing? 

This question is particularly important for students who are engineering or science majors.  If the engineering or science department teaches its own writing classes, the training will probably be much more useful.  If the English department teaches these classes, students will graduate knowing a bit about Shakespeare, but they probably won’t know how organize a proposal or how to select appropriate illustrations for a technical report.  Make no mistake, writing is a critical skill, and any engineer or scientist who graduates with inadequate training in this area is graduating with a serious professional handicap.

Question 5:    How many office hours do the professors make available to students each week?

You’ll want this number to be high.   Students can’t pick up everything they’ll need from lectures, and from my experience, being able to visit with professors and ask questions is critical.

A Four-Step Problem Solving Approach

Posted in Manufacturing Improvement with tags , , , , on March 8, 2012 by manufacturingtraining

In August 1990 the United States starting sending military forces to the Persian Gulf with the intent of expelling Saddam Hussein’s forces from Kuwait.  We called the buildup Desert Shield, and when we actually went to war on 16 January 1991, the name transitioned to Desert Storm.  When Desert Storm finally started, the engagement was decisive.  In short order, Kuwait was free of Iraqi forces.  It was the beginning of the end for Saddam Hussein.

Desert Shield (the buildup) lasted a good 6 months. The question in those days was:  Why the delay?  We had our forces and those of allied nations in place relatively quickly. Why did 6 months elapse before we crossed the border into Kuwait to expel Saddam?

The true reasons for the lengthy delay may never be known, but I can tell you that a key component of our smart munitions delivery capability was not ready in August 1990. You all remember the dramatic videos…munitions being dropped directly down chimneys, one-drop hits, etc.  All that was made possible through laser-guided munitions (along with the bravery and skill of our fighting forces).

One of the key laser targeting devices was the Mast Mounted Sight, shown in the photo above.  It’s the thing that looks like a big basketball on top of the helicopter.

The Mast Mounted Sight contained a laser target designator, an infrared sensor, and a television sensor.  All were slaved to the pilot’s helmet.  Wherever the pilot looked, that’s where all three beams were supposed to point.  The Mast Mounted Sight had been in production and deployed on helicopters for years.  Everyone thought everything was fine.

But it wasn’t.

When the Desert Shield buildup started, the Army tested its Mast Mounted Sight systems a bit more rigorously, and it discovered what it and the manufacturer thought was an alignment error in the laser, IR, and television lines of sight.  This could have been disastrous.  It meant that the pilot might launch a missile based on the television or the IR sensor being on target, but the laser beam would guide the munition to the wrong spot.  If a miss occurred, it would alert the bad guys, and they could return fire against the helicopter.  Mind you, this system had been in production and deployed in the field for years.

The manufacturer went into high gear to find and fix the failure cause. The Mast Mounted Sight contains an internal alignment mechanism, which is supposed to align all three instruments (the laser, IR sensor, and the TV sensor).  The manufacturer spent the next 6 months looking for a problem in the MMS alignment subassembly.  They didn’t find anything.

Hold that thought.

Ever hear the joke about the drunk looking for his car keys at night under a street light?

It goes like this: I offered to help the drunk find his keys, and after we both searched for an hour, we came up empty-handed.

“Gee,” I said, “are you sure you dropped them here?”

“Oh, no,” responded the drunk. “I lost them over there, by those bushes in the dark…”

“Then why are you looking here under the street light?” I asked incredulously.

“Because I can see here,” he answered.

Many times when we have a production shutdown, or even a low-level recurring failure, finding the root cause is elusive. Production shutdowns get a lot of attention.  Recurring nonconformances frequently do not, but they can just as expensive (sometimes more so) than a line-stopping failure.

So how do we go about finding the root cause of a failure?

Many years ago, the smartest man I ever knew once shared a simple four-step problem solving process with me.  It goes like this:

  • Define the problem
  • Define the causes
  • Define the solutions
  • Select the best solution

Where we usually go south when analyzing failures is with that first step: Defining the problem. Frequently, we start jumping to conclusions about potential causes without taking the time to fully understand the problem. The results are predictable: We spend lots of time chasing our tails, and the problem continues.

Need proof?  Try this exercise:  Tell your staff that you walked into a room, flipped the light switch, and the light did not illuminate.   Then ask them what the problem is.  In most cases, folks will immediately start listing potential failure causes: A broken filament, breaks in the wiring, a defective switch, failure to flip the switch properly, etc.  But those are all incorrect answers.

The question should be:  What is the problem?  That should be our first step.  In this case, the problem is that the light bulb does not illuminate.  All of the other suggestions listed above involved jumping to conclusions about potential causes.

Let’s turn back to the Mast Mounted Sight.  After several months of trying to find a failure cause in the MMS alignment mechanism, the failure analysis team finally decided to take a step back. They reviewed the test data again, and to their amazement, they found that the TV and the laser were aligned.  Only the IR sensor was out of alignment.  The failure analysis team had been solving the wrong problem.  Once the problem came into focus, the team looked outside the alignment mechanism, and they found an IR window heater anomaly.  The fix was a simple software patch.  It was implemented on 15 January 1991, and US troops rolled across the Kuwait border on 16 January 1991.

Would you like to know more about our fault-tree-based Root Cause Failure Analysis training program, or perhaps our book on Systems Failure Analysis?   Check out our Root Cause Failure Analysis page, and give us a call at 909 204 9984 if you would like to know more!

Ppass versus Reliability

Posted in Manufacturing Improvement with tags , , , , , , on March 3, 2012 by manufacturingtraining

Many times companies use sampling techniques to assess a production lot’s acceptability.  You know the drill…you pull a specified sample size, and if all of the samples are acceptable you buy the lot.  If any are unacceptable, you reject the lot.  This approach often works for components, assuming the sample represents the rest of the lot.   But what about larger subassemblies or complete systems?  Does it work for them, too?

Here’s the basic question:  Is your acceptance testing approach consistent with your product’s required reliability?

This is an area where a lot of companies (and buying organizations) put themselves in a serious bind without realizing what they are doing.  In the munitions game, for example, it’s pretty common to pull a specified sample and buy the lot if all of the samples go bang.  The problem is that we think if a product’s reliability is high (say, 95%), we ought to be able to pull a sample and have them all work.  That’s not the way it works in the real world, though.   We can’t go with our intuition here; we have to evaluate the probability of passing the acceptance test more rigorously to assure that it is consistent with the required reliability of whatever it is we are testing.

I first ran into this at Aerojet when we were building munition fuzes.   We were failing most of our lot acceptance tests, and we thought we had a pretty good product.   The submunition had a 95% reliability requirement, and in live tests we showed we met that requirement.  We routinely dropped bombs and had more than 95% of the submunitions detonate.

We had a lot acceptance requirement on the fuzes, however, that required firing a sample of 32 with no failures.   We were only passing about one lot out of every five.  What was going on?

What we didn’t realize (at least initially) is that there’s a fundamental difference between demonstrating a product’s reliability and passing a specified-sample-size test with zero failures.   There’s a relationship between a product’s reliability and the probability of passing its acceptance test that can be shown with something called an operating characteristic curve.   For that test I just described (n = 32, acc/rej =0/1), the x-y plot below shows it clearly:

Check the above plot, and you’ll see that with a product reliability of 95%, you’ll only pass the acceptance test about 20% of the time (and that was exactly what we were experiencing).

When we explained this to our Air Force customer, they didn’t like what they were hearing, but they recognized and agreed with the mathematics.  Ultimately, they modified the fuze acceptance test requirement so that it was consistent with the product’s required reliability.  Don’t think that this allowed lower quality munitions to get into the inventory, either.  That particular munition system routinely delivered reliability well in excess of its requirements, and during the 1991 Persian Gulf War, it was the munition that took out the bulk of Saddam Hussein’s Republican Guard tanks.

Another manufacturer was not so lucky.  They manufactured flares for the US Navy, and they encountered precisely the same problem with precisely the same numbers.  The Navy’s reliability requirement was 95% (which the flare met), but they imposed that same lot acceptance requirement (a sample size of 32 flares, accept on 0 failures, reject on 1 or more failures).  Predictably, the company failed 80% of their lot acceptance tests.   Unfortunately, in this case, neither the Navy nor the manufacturer realized what was happening.

I know about that second situation because I was an expert witness when the manufacturer sued the Navy.   When I testified at the Armed Services Board of Contract Appeals, my task was to explain all of the above in a manner that lawyers and the trial judge could understand.  In my experience, lawyers and judges don’t grasp probability and statistics concepts easily, so just stating that the situation was governed by the binomial distribution wasn’t going to cut it.

I went shopping the night before I testified and bought two bags of coffee beans (one with white beans, and one with brown beans).   I put 5 white beans in a bag (representing unreliable product), and 95 beans in the same bag (representing product that would work).  I stuck the bag in my pocket the next morning and went to court.

After explaining the binomial distribution, the nature of the relationship between a product’s reliability and the probability of passing a test, and the x-y plot you see above, I could see that the judge (who was a good guy) had glazed over.  When I finished, I told the judge I could demonstrate the concept for him.  I pulled the bag of coffee beans out of my pocket and explained the contents, and I offered to pull out 32 beans.  The lights came on.  The judge smiled.  He told the Navy’s attorney to pull the beans out of the bag.  The 17th bean was a white one, representing a flare that wouldn’t work (and a failed lot acceptance test).

It was a cool display, it was a deciding factor in the manufacturer winning its $25.4 million claim against the Navy, and that little demonstration was cited as one of the best Armed Services Board of Contract Appeals explanations that year.

So, think about this…when you specify (or agree to) a sample-based test, what’s the reliability of the thing you’re testing, and is it consistent with your test?  If you are failing sample-based acceptance tests, you may simply have an overly-stringent acceptance test.   These kinds of evaluations sound complicated, but Excel makes it a lot easier than it used to be.  The operating characteristic curve is one of the key concepts we should always consider in such situations, and it’s a key part of the root cause failure analysis training we offer.

Hello There!

Posted in Manufacturing Improvement with tags , , , , on March 2, 2012 by manufacturingtraining

Hello, everyone!  Joe Berk here.  I’m the principal consultant and trainer at ManufacturingTraining.   We’re firing up this blog to keep people posted on what’s going on in our world, to tell a few process improvement war stories, and to have fun!

So why a blog?

Well, I write another blog for one of my clients, the California Scooter Company…a great motorcycle manufacturer right here in the United States.   That blog has a huge following, and it’s helped the California Scooter Company enormously.   My thought is that it might be cool to develop a similar following here.

So who are we?

Well, if you’re like me, you probably haven’t had great experiences with consultants.   There are nearly as many consultant jokes as there are lawyer jokes, and where there’s humor, it’s usually based on truth.  To be blunt, when I managed manufacturing organizations I thought the consultants I had rammed down my throat by my bosses were hucksters.

I like to think my organization is different, and our clients tell me that’s the case.  There’s nothing “touchy-feely” about us.   We don’t have any consulting buzzwords, three-letter  acronyms, Greek-lettered improvement programs, or martial arts belts, and there’s a reason for that:  I ‘ve managed large engineering and manufacturing organizations.    I know what it means to have to deliver conforming products on time at or below budgeted cost, and that is the theme underlying all that we do.   If you want to hold hands and sing songs, we’re not your guys.  If you want the hard facts and measurable improvement, I think you’ll want to talk to us.   But don’t do so right away.  Follow the blog for a little bit, and if you have an interest, give us a call.

So this is our first post on the blog.  It’s new, we are going to make it interesting, and it’s focused on manufacturing.   Like I said above, there’s no snake oil here.   It’s the real deal.  I’m a writer and a photo nut, and I’ll do my best to keep interesting stuff posted here for you.   We’ll have photos, videos, graphics, and sometimes just words.   Want to see a sample?   Take a look at what it took to make a motorcycle for Melanie Troxel, the In-N-Out Burger Corporation’s funny car driver…

That was fun.   Sometimes we get to do things like that for some of our clients.

Let me throw a question at you more directly related to process improvement….do you know what this photo shows?   I’ll give you a hint…this product had a rejection rate of about 50%, each assembly cost about $50K, and the company lived with the problem for 10 years before they applied what we offer!   Let me know if you can guess what this is.  Shoot your comments in, and in a another day or two I’ll share with you what it’s all about!

If you’d like to follow the ManufacturingTraining blog, just click on the subscription link up top (it will let you know when new things are posted here).   You’ll have to enter your email address, but this is the only place it will be used and I promise you’ll never be spammed by us!

Stay tuned…there’s lots more coming up!