Inside Apple’s factories [Cook book outtakes]

This post was going to be part of my new book, Tim Cook: The Genius Who Took Apple to the Next Level, but was cut for length or continuity. Over the next week or so, we will be publishing several more sections that were cut, focusing mostly on geeky details of Apple’s manufacturing operations.

A good measure of the size of Apple’s manufacturing operations is its capital expenditure, the amount of money spends on things like buildings and equipment.

Apple’s capital expenditure, or CapEx, is mindboggling. To get an idea of how big it is, take Apple’s new spaceship campus in Cupertino – which is the fourth most expensive building in the world. It cost the company an estimated $5 billion to construct.

Apple spends a similar amount every six months on manufacturing equipment.

Apple’s massive CapEx

Apple is spending on the order of $10 billion a year. To imagine how much money that is, Apple analyst Horace Dediu compares it what the U.S. Navy spends on things like aircraft carriers, which are the biggest pieces of equipment they buy and often take 10 years to build.

“An aircraft carrier costs like four or five billion dollars,” said Dediu. “In the case of Apple then, they probably spend the equivalent of one or two aircraft carriers per year.”

He added: “It’s a tremendous, tremendous amount of money and hard to get your head around … the Navy doesn’t spend as much as Apple does.”

Most of this money appears to be spent on manufacturing machines. Apple’s CapEx spending is in two major buckets – data centers, and manufacturing equipment. Unfortunately, Apple doesn’t break down the relative size of each bucket, but Dediu says the larger bucket is manufacturing equipment and the smaller one is the server business, which is on the same line item that Microsoft, Google and Amazon also spend.

“The only tech company in the U.S. that I know that spends heavily on capital equipment for manufacturing is Intel and in fact you can kind of compare Apple and Intel’s spending. They’re sort of comparable.”

Intel spends money on giant chip manufacturing plants, known as “fabrication” plants or “fabs.” New fabs cost an estimated $3 to $4 billion. “Fabs are very, very expensive,” he said. They “cost in the billions of dollars and they can only buy one or two per year.”

Apple isn’t building fabs, but it is buying huge, huge numbers of manufacturing machines, like CNC routers, which are used to make the cases of iPhones and iPads from solid chunks of aluminum. It’s estimated that Apple has bought up the entire supply of CNC machines for several years in a row. These machines can cost up to $1 million apiece, which means Apple is buying vast numbers of them.

“They have to buy many, many of these machines,” said Dediu. “It’s staggering. You’d be hard pressed to actually work out just how many of these things would be necessary. Anecdotal stories say that you could go to some of these plants and see machines lined up as far as the eye can see. Literally thousands of them in parallel working to make certain parts. So that’s the kind of scale that we’re dealing with.”

The scale is so vast, the spaceship campus didn’t even show up on Apple’s CapEx reports. The cost of the building is mixed in with Apple’s other capital expenditures, and for most other companies, an expenditure of $5 billion dollars would be clearly visible in expenditure reports, even if it were spread over several years.

But in Apple’s CapEx reports, there’s no uptick in spending in the last few years that would indicate it was building one of the most expensive buildings the world has ever seen. Apple is spending so much money every year, the money for the spaceship got buried. Dediu estimated it accounted for only about 10 percent of Apple’s CapEx for five years.

“Even something as phenomenal as a landmark the size of that facility doesn’t register that much,” he said.

Apple’s hybrid outsourcing model

How can Apple be spending so much money on equipment when its manufacturing is contracted out? The answer is that Apple is pioneering a unique hybrid outsourcing model.

Just as Apple likes to control its primary technologies, the company also controls the machines in its suppliers’ factories.

Historically, most manufacturers owned their factories and equipment and manufactured their products in-house. Or they outsourced it to a third-party, which owned the machines. Apple is mixing it up: Apple provides tooling and financing to some of its contract partners, buying the manufacturing equipment that is put in their factories. Apple owns the manufacturing equipment, but the supplier owns the factory and the workforce.

It’s a unique hybrid model. In most outsourcing arrangements, contract manufacturers own and control their own machinery. But it appears that Apple is buying the machinery that goes into partners’ factories. This gives Apple a lot more control. It can dictate the manufacturing process to its own stringent standards.

“On a strategic level I think Apple made the decision that they need to engage,” said Dediu. “They need to be present on the factory floor and being present on the factory floor means to Apple not just sending somebody over to interface with Foxconn but rather to work out how to make the process of manufacturing better.”

It also prevents Apple’s intellectual property from leaking and the unauthorized usage of the machines. One of the problems of working with offshore manufacturers is counterfeiting. It’s not uncommon for a factory to make production runs in off hours, during the night for example, making copies of official products for sale on the black market. “Next thing you know they’re making exact copies because they’re using the exact same machines,” said Dediu.

Apple owning the machines allows the company to dispose of them – or keep them idle – when the product run is finished. “If Apple’s in control of those tools and machines they can say, ‘OK the run is over. Now we’re going to dispose of these and no one else can use them.’ That would be another lever they have in preventing copycats.”

Apple’s model is also likely because of the huge numbers involved. Dediu suspects it is difficult for some of Apple’s manufacturing partners to get the necessary financing to buy the machines they need on such a vast scale. Unable to get loans for traditional sources like banks, Apple steps in. But instead of loaning the contractors the money, the company buys the machines itself. The process is often quicker and easier than going to the bank.

Practice builds

The process of building an Apple product begins with a prototype — or more likely, hundreds fo prototypes — which are designed and built at Apple HQ.

The time scale between the first prototype and first production unit is anywhere between six months to 18 months or longer, depending on the product. Some products take years. During that time, there’re hundreds, if not thousands of issues that need to be found and resolved.

The process is almost entirely manual. It requires engineers to constantly fly out to the factories and study the manufacturing lines for hours, days or weeks at a time, watching very carefully how the product is put together.

The process begins with a series of test builds to test the actual manufacturing process. The first is an Engineering Validation Test (EVT). An EVT build is a test unit – or a series of test units — that are the rough equivalent to an “alpha” build in software – which is the first release of a program in the very earliest testing stages. After EVT, the product enters the DVT testing stage, or Design Verification Testing, and finally there’s PVT, which is Production Verification Testing, which are roughly equivalent to beta tests in software.

“Before you get to production units that you’re building for customers, you do these practice builds,” said Anna-Katrina Shedletsky, a former-Apple product design engineer who spent almost six years at Apple working on four generations of the iPod, and then as a product design lead for the Apple watch. Shedletsky is now CEO of Instrumental, a startup that uses machine learning to help companies find and fix problems on factory assembly lines.

“This is industry standard,” she added. “These practice builds are essentially groups of prototypes that are built, so that you can test and iterate on the design, make sure the design is ready for mass production.”

Problems with the production builds can crop up in lots of ways. Some components may be sensitive to handling, and a high proportion get damaged during installation. Sometimes a part may be badly designed so that it’s possible to put it in backwards, and then it malfunctions when the product is turned on for the first time.

Shedletsky talked about glue, which is widely used in a lot of Apple products. Glueing parts together can be a great way to bond components, but there are lots of ways it can go wrong. The worker might apply too much glue, or too little. There might be gaps, or the glue might spread to other areas of the product where it’s not wanted. The glue might not cure properly if the factory floor is too hot or too cold. Chemicals used in another part of the manufacturing process may eat into the glue and weaken it. There’s lots and lots of things that can go wrong, and some of them are only discovered much later.

“These are the types of things that come up during development that have to get hammered out,” Shedletsky said.

Some production problems can fiendishly difficult to catch. Maybe there’s a screw that occasionally gets forgotten, but it’s only late in the shift, after the worker has returned from a break and is a bit sleepy. The screw is forgotten only in one or two units per shift, which doesn’t seem like a big deal, but it can really add up if the factory is outputting a million units a day. These rare manufacturing glitches are called ‘onesie, twosises.’

“One or two units will have this issue,” she said, “But “If you’re building a million units a day, that’s a lot of units that are going to have that problem. The whole idea is solve it when there’s only one or two. Then you never have to suffer the fallout of tens of thousands of units that might have this issue.”

Designing the Golden Line

The production process begins with setting up the “golden line,” a prototype assembly line that serves as a reference for all the other lines that are set up after it.

Setting up the golden line involves the industrial design team, the product design team, mechanical engineers and engineers from design from manufacturing and operations, as well as the contractors themselves. (see the previous post on how Apple’s industrial design team works with product design and operations).

The idea is to tune the golden line to achieve the best speeds and the best yields. Often, yields and speeds are a trade off. It’s easier to obtain better yields at slower speeds, but mass production requires speed and efficiency. The golden line tries to find the best trade off.

The idea is to set up the perfect line, and once set up, nothing is changed.

“The golden line is constantly evolving to a certain point in development and then it’s locked,” said Shedletsky. “Essentially, it’s golden, don’t touch it.”

The golden line acts as a template for all the other lines that are set up after it. Everything is copied exactly.

It can be tricky to keep the golden line untouched. Sometimes there’s weeks or months between building the line and going to full production. Lots of things can change.

If any changes are made later on, they are tested first on the golden line.

“Every new line has to be compared to that one,” said Shedletsky. “If you’re trying to validate something new, if you’re making a change somewhere else. Let’s say you need to bring up another tool for an injection molding tool, that’s a big part of the enclosure, that you would validate that new tool on the golden line, because you know line is good, you can check to see if new parts are good or not. It’s used in a very careful way.”

One common problem on assembly lines is when an experienced operator quits, and a new one has to be brought in. The new operator has no experience, and is slower. The whole line is outputting fully assembled products every nine seconds, which means that every station on the line has to operate at the same nine second interval. An experienced operator can keep up, but a new one can’t, so instead of substituting one-for-one, three inexperienced operators are brought in to replace the experienced one. That way that station can continue to operate at the same nine second interval.

Market timing and Xmas

For a consumer products company like Apple, the Christmas season is the most important part of the year. It’s when the vast majority of its products are purchased, and new models of the iPhone are released every year in time for the holidays.

The industry standard is that new products should be shipping by September 1st. That gives the product a full three months on shelves during the critical fourth holiday shopping quarter.

In order to do that, factories have to be able to hit peak production in September and October, and the volumes drop in January.

The ramp

After the golden line is set up and thoroughly tested, it gets replicated, a process that’s called ‘ramp.’

The number of lines depends on the speed of the golden line and the number of units that need to be produced per shift.

Shedletsky explained: “A very fast human operation could be nine seconds, which means your line is running at a certain number of units per hour. Maybe you’re producing between 3,000 and 5,000 units per shift on that one line.”

If you need to be making 5,000 units a day, you only use one line. If it’s 20,000 units a day, the plant will need four lines; and if it’s 100,000 units a day, you need to have 20 lines.”

One estimate for iPhone production I saw had Apple’s factories outputting 750,000 iPhones a day, which equates to about 150 lines.

On the first day, the plant sets up a few lines and starts training operators under the guidance of line leaders. The plant slowly builds up to dozens, and then perhaps hundreds of lines. All the time, the Apple’s engineers and the line leaders are carefully watching yields. The idea is to avoid a big pile of unusable products in the “bone pile.”

The bone pile, Shedletsky explained, is an industry term for faulty units that can’t be shipped unless they’re repaired.

“There are people’s job on the operations side who really get into the details on this and they have models that I don’t understand around how you actually tune the knobs to ramp responsibly, where you’re not going to create a huge bone pile of units you can’t ship to customers.”

Companies like Apple start ramping up in the summer in order to hit peak production in September and October.

Ramping takes time, oftentimes weeks and sometimes months. It’s impossible to just turn it on.

Forecasting is key. Ideally, supply exactly matches demand – the company sells every single product it makes.

“There is a whole black magic that goes into figuring out how many products to ship for a given market,” Shedletsky said. “I am not an expert on that. Because you have to make predictions around what your customers want. It’s very expensive to be wrong, to overshoot.”

The launch is a complex process, especially for a company the size of Apple and an iPhone launch, which can see as many as 10 million handsets sold in the first week. Achieving that is a huge operation, involving not just manufacturing, but marketing, logistics, retail stores, channel partners, warehouses, shipping companies and airlines, customs inspections, and so on.

“There’s so much involved in actually getting a product out at a large company,” said Shedletsky.

Sometimes a company like Apple will make tradeoffs in the ramp. If there are problems with ramp, and the lines are producing low yields and there’s a lot of products going on to the bone pile, the company might decide to delay the launch date, or live with lower yields (which costs money).

That’s why every year the availability of the flagship iPhone tends to shift. It’s usually in late September, but occasionally it gets pushed back. In 2018, the iPhone XR, for example, was released in late October.

Challenges of getting good yields

The idea is to maximize yields coming off the assembly line. If there’s one or two issues in a test build of 100 units, that translates into tens of thousands of units when building millions of units.

Sometimes faulty products get scraped, sometimes they get repaired or reworked.

Apple has a specialist team in operations that is devoted to yield studies. The team carefully analyses products coming off the lines for faults. The process takes weeks, even when there are no major issues found.

More often than not though, major issues are discovered, which frequently lead to delays in launching a new product.

“These product delays are really prevalent,” she said. “The top two reasons for that delay that they have self-reported are the late discovered issue and the amount of time it takes once you’ve discovered an issue to actually get to a root cause that can fix and validate that root cause and then be ready to ship again. Often, a lot of these issues don’t become apparent until you get to volume and you don’t get to volume until you’re almost ready to ship. That’s that late discovered issue.”

Example of a production problem; a bent antenna

Shedletsky told a story about one of Instrumental’s customers. They had been bedeviled by a faulty antenna in a product. In an early development build, the product had more than 20 antenna failures, which is a lot. The antenna engineer did a failure analysis, going through each failed product, disassembling the units, trying to find the root cause. They suspected it was a design issue, but they found that the assembly operators had smashed the antenna and bent them as they were installed. It wasn’t an issue with the antenna, it was a process issue.

“This is exactly the type of issue that if you went into production and you’re ramping multiple lines and you have operators who have gone through operator training, but are still very new and green and haven’t done the operation many times yet, could cause that damage and you could have a huge bone pile,” she said.

This may the kind of issue that Apple faced with the AirPower charging mat, which was recently cancelled. Apple said only that the mat didn’t meet it’s high standards. The product had been advertised on the packaging for the latest AirPods, suggesting that it was cancelled at the last minute, which in turn suggests a manufacturing problem.

Size of Foxconn

It’s hard to wrap your mind around the size of Apple’s operations. Foxconn Guanlan, known as Foxconn city, employs more than 800,000 people. That’s almost the population of a city like San Francisco.

Foxconn City is bigger than the Vatican, the Forbidden City and even Jerusalem. It’s self-contained, with its own dormitories, stores, restaurants, bars, karaoke parlors and health facilities.

“These places are huge,” said Shedletsky. “They’re massive.”

At the height of ramp, up to a million iPhones are coming off the lines a day. To achieve that scale, the factory needs dozens of lines for the main assembly, and sometimes hundreds of lines for some of the subcomponents, said Shedletsky.

Shedletsky said each line can make somewhere between 3,000 and 5,000 units a day. If the plant is running two shifts a day, which might be boosted to three shifts during the busiest times, each line can build 10,000 on the line a day. To build a million, the plant needs to have 100 lines.

One-sided supply chain deals

The scale is so big, it sometimes leads Apple to do unethical things, like screwing smaller suppliers.

In order to achieve its massive production numbers, the company has often worked with smaller suppliers to create innovative production methods, then hung them out to dry.

The former operations executive said his team had no choice but to push the supply chain to its limits in coming up with new manufacturing methods. To do this, they sought out small start-ups that had developed innovative methods for manufacturing products or materials. Encouraged by investment from Apple and a promise of a profitable, long-term partnership, these companies invested heavily in maturing their production methods. But once they’d figured it out, Apple dropped them and moved production to Foxconn, according to the executive.

“We would find the niche supplier that would somehow figure out the manufacturing process we needed to make the ID reality and meet the PD specifications at a barely acceptable yield, then we would go to Foxconn and duplicate the complete process and cut the production over once the methods and manufacturing was stabilized,” he said.

The executive said Apple’s budget for this business operation ran in the multiple tens of millions of dollars a year.

“Several of our suppliers went completely out of business because they had so much invested in this creation that when the business was (as was always the case) we were forced by the procurement group to move the manufacturing to Foxconn. It became SOP [standard operating procedure].”

The executive said Apple did this because smaller companies could never handle the ramp volumes. It was strictly business.

“We paid them crazy amounts of money to develop the manufacturing processes and work ridiculous hours and anything they developed belonged to Apple. Anything. That is the deal. Jonathan Ive thinks he invented the rectangle so I can assure you they have patented everything else under the sun. Foxconn became Apple’s bitch in early 2000 and Terry Goh was smart to bet on Apple. When we first went there Apple had part of Building 11 and then the whole building six months later. [By 2013] many buildings were completely dedicated to them. Foxconn has 1,000,000 employees worldwide and it would not be a far off guess to bet that at least half work in an Apple manufacturing facility.”

The executive declined to name the smaller companies. He said he had no knowledge that any of the them tried to sue Apple but that if they had tried, he’s sure they failed.

“The best thing a supplier could do was be very smart up front and lock us in the contract for a certain amount of production and charge us like we were buying cars. [Out of many] only two I know of did this and remained alive after the experience.”

Despite the seemingly amoral supply chain decision-making, the exec admits that most times, the move to Foxconn was justified by the long term implications of working with an inexperienced supplier. Apple could not count on them to deliver. As a result, he never tried to stop it from happening.

“The smaller suppliers were too difficult to deal with and the part logistics were always a problem,” he said. “We could throw it at Foxconn and they would assemble an army at our bidding. I was told to do it anyway and at Apple, you do what you are told to do.”

Secrecy and lack of best practices

Secrecy, of course, is also one of Apple’s prime directives. All of Apple’s staff are sworn to secrecy, and sign big stacks of legal paperwork to ensure their silence. Apple is so secretive, Shedletsky conditioned herself not to use the word “watch” when working on the Apple Watch, even as a verb in completely different contexts.

“People are trained not to talk about what they do,” she said. “I didn’t use the word ‘watch’ as a verb, for the period of time I was working on the watch program. Like ‘to watch TV’. No, we’re going to use some other verb.” (She didn’t say what verb she used as an alternative.)

“Out of everything I’ve ever done in my life, I’ve never seen a more secret environment than working there,” said the former Product Design engineer. “We were constantly under threat of losing our jobs for revealing any shred of anything. And even within Apple, your neighbors often didn’t know what you were working on … the secrecy was like a gun to your head. Make one false move and we’ll pull this trigger.”

The secrecy stops leaks, but it has detrimental effects. One of them is that even within Apple, it’s own processes aren’t well defined. There are no best practices written down. For example, Shedletsky said it wasn’t clear at Apple what exactly an EVT build was.

An EVT, or Engineering Validation Test, is a test unit – or a series of test units — that are built to test the actual manufacturing process. EVT is the rough equivalent to an “alpha” build in software – which is the first release of a program in the very earliest testing stages. After EVT, hardware enters the DVT testing stage, or Design Verification Testing, and finally there’s PVT, which is Production Verification Testing, which are roughly equivalent to beta tests in software.

Shedletsky said it wasn’t clear when one stage was completed and the next began. The criteria for each stage wasn’t clearly defined. It was usually a matter of guesswork.

With less secrecy, it would be possible to write down some best practices, but Apple forbade it. At her company Instrumental, her engineers wrote a guide that spelled out the criteria for each test stage that she said has been printed out and pinned to the walls at factories around the world.

“Because there’s secrecy, there’s confusion,” she said.