Category Archives: Blog

For the last couple of months, my bi-weekly posts have been devoted to Jack Ganssle’s excellent series on the top reasons why embedded projects run into trouble that’s been running on embedded.com. As an engineer who’s been involved in building embedded applications throughout my career, I found myself nodding my head in agreement each time I read one of Jack’s articles.  From his number 10 reason – not enough resources – down through his number 1 – unrealistic schedules – I’ve recognized a number of situations that I’ve personally been involved in. But it’s also been gratifying to recognize that, here at Critical Link, we put a good deal of focus on avoiding problems like the ones Jack has on his list.

With that, here’s my summary of Jack’s Top Two.

2 – Quality gets lip service

Some of Jack’s Top Ten – maybe even all of them – have issues around quality at their core. Here he addresses it directly.

Jack points out that American cars used to have a terrible reputation for quality. Then the Japanese began entering the market with cars that were both higher in quality and lower in price than American vehicles. Then American car manufacturers paid a few visits to Japan and discovered what the Japanese had learned from American quality gurus like W. Edwards Deming: “that the pursuit of quality always leads to lower prices.” These days, quality is no longer a problem for American car makers:

Today you can go to pretty much any manufacturing facility and see the walls covered with graphs and statistics. Quality is the goal, and it is demanded, measured, and practiced.

Alas, Jack finds that, in firmware labs, there tends to be less of a focus on quality. He wants developers to spend more time on test, and not let it wait until the end of a project. Because we all know that, if schedules get tight, something’s got to give and if testing is on the tail end of the schedule, well…

But test is not enough. Quality code happens when we view development as a series of filters designed to, first, not inject errors, and second, to find them when they do appear.

We’re not a firmware lab, per se, but the point about testing and best efforts to “not inject errors” is well taken.

I’m very proud that, at Critical Link, we take quality very seriously across the board. (Sorry for the pun. I couldn’t resist.) I’d be happy to talk to anyone about our quality focus, but to take a shortcut, I’ll note that we’re ISO 9001:2015 certified.

1 – Unrealistic Schedules

Jack’s #1 in this series of articles is one he characterizes as “a fail that is so common it could be called The Universal Law of Disaster:” unrealistic schedules. In his discussion of this problem, he catalogs the reasons why those impossible schedules are something that we universally (and perpetually) have to grapple with as developers. He talks about top-down pressures to hit aggressive dates in order to secure a project.

These pressures aside, he doesn’t let the engineers doing the estimating off the hook:

Too often we do a poor job coming up with an estimate. Estimation is difficult and time consuming. We’re not taught how to do it… so our approach is haphazard. Eliciting requirements is hard and boring so gets only token efforts.

Jumping into coding too quickly is #9 in Jack’s top 10. As I noted in my commentary on that reason, “at Critical Link, our experience as systems engineers means we place tremendous importance on planning and being careful about our requirements.” I’ve found that, if you put the time in on the front end to very thoughtfully figure out what the exact requirements are, you’re rewarded with a much more realistic schedule.

There are, of course, invariably changes to a project. That’s just the nature of the development beast.

There will always be scope modifications. The worst thing an engineer can say when confronted with a new feature request is “sure, I can do it.” The second worst thing is “no.”

“Sure, I can do it” means there will be a schedule impact, and without working with the boss to understand and manage that impact we’re doing the company a disservice. The change might be vitally important. But managing it is also crucial.

“No” ensures the product won’t meet customer needs or expectations. At the outset of a project no one knows every aspect of what the system will have to do.

By the way, the answer to allowing some requirements creep without impacting the schedule is NOT to “throw more resources at it,” which Jack sees as “usually an exercise in futility.” Best to a) do solid planning and careful scheduling from the get-go, and b) keep everyone who needs to be in the loop when the schedule looks like it might slip.

Anyway, that’s it for the Top Ten Reasons Embedded Projects Run into Trouble. Good things to keep in mind, even for us old-timers! Hope that you enjoyed reading this series of post as much as I enjoyed writing them. I think the list is pretty complete, but if you have other problem areas you think should have made the list, please let me know. I’d love to hear from you.

 

Here are the links to my prior posts summarizing Jack’s Reasons 10 and 9, Reasons 7 and 8, and Reasons 6 and 5, and Reasons 4 and 3.

For the last month or so, Embedded.com has been running an excellent series on the Top Ten reasons embedded projects run into trouble, according to Jack Ganssle, an embedded systems guru. I’ve been having a good time counting down with Jack. As I’m writing this summary of his third and fourth reasons, reasons 1 and 2 have not yet been revealed. But stay tuned. I’ll be getting to them in another two weeks.

Anyway, here goes reasons 4 and 3.

4 – Writing optimistic code

Jack starts off this piece in a very humorous way: with a store receipt for a couple of small sailboat parts that showed a charge of $84 Trillion.

This is an example of optimistic programming. Assuming everything will go well, that there’s no chance for an error.

I don’t know if the appearance of an obvious error like this is an example of optimistic programming. But it’s sure an example of poor testing.  But maybe they’re one and the same thing. Anyway, Jack points out that, given that “the resources in a $0.50 MCU dwarf those of the $10m Univac of yore… We can afford to write code that pessimistically checks for impossible conditions.” Which we need to do given that “when it’s impossible for some condition to occur, it probably will.”  Let’s call this point Ganssle’s Law:  the coding equivalent of Murphy’s Law.

Giving advice specific to embedded programming, he suggests that almost every switch statement should have a default case And that assert macros should be universally used:

Done right, the assertion will fire close to the source of the bug, making it easy to figure out the problem. Admittedly, assert does stupid things for an embedded system, but it’s trivial to rewrite it to be more sensible.

Jack’s bottom line:

The best engineers are utter pessimists. They always do worst-case design.

Considering myself a pretty good engineer and a not especially pessimistic sort of guy, I hadn’t thought of it this way. But I’m definitely down with doing worst-case design.

3 – Poor resource planning

Embedded systems developers face challenges that software developers don’t have to deal with in a world of cheap memory, cheap storage, etc.  And that’s resource constraints:

With firmware, your resource estimates have got to be reasonably accurate from nearly day one of the project. At a very early point we’re choosing a CPU, clock rate, memory sizes and peripherals. Those get codified into circuit boards long before the code has been written. Get them wrong and expensive changes ensue.

But it’s worse than that. Pick an 8-bitter, but then max out its capabilities, what then? Changing to a processor with perhaps a very different architecture and you could be adding months of work. That initial goal of $10 for the cost of goods might swell a lot, perhaps even enough to make the product unmarketable.

This ties back to an earlier reason that Jack listed for why embedded projects can go awry. His Number 9: jumping into coding too quickly. The answer to that one was to pay a lot of careful attention to requirements. Same thing applies here, but with the problem that you’re typically subject to financial constraints on what firmware to pick. That and the fact that you also need to avoid built in obsolescence. So:

A rule of thumb suggests, when budgets allow, doubling initial resource needs. Extra memory. Plenty of CPU cycles. Have some extra I/O at least to ease debugging.

Next post, we’ll wrap up with the reasons that get Jack’s nod for first and second place.

Here are the links to my prior posts summarizing Jack’s Reasons 10 and 9, Reasons 7 and 8, and Reasons 6 and 5.

Embedded.com has been running an excellent series of articles by Jack Ganssle on his top ten reasons why embedded projects may find themselves in trouble. I’ve been summarizing Jack’s articles here for my past couple of items. Here’s my take on Reasons 10 and 9, and on Reasons 7 and 8.

It’s not surprising that there are so many ways in which even small, simple projects can go off the rails and some of those reasons are universal.  But embedded development is so complex that projects have pitfalls that are both particular to these types of projects, and are common for projects of all types.  In today’s post, we’ll address one of each.

6 – Crummy analog/digital interfacing

First there’s the good news:  thanks to advancements like multi-layer (and cheaper) PCBs and a mind-boggling array of ICs, analog design has gotten easier over the years. The not so good news:

On the other hand, the more resolution we can get, the more we need so other issues become paramount, like noise…Noise can be fought by many means, but I see too many teams blithely throwing some sort of filter at it. A simple bit of code can average some number of samples, or implement IIR/FIR filters. These are great solutions to some problems, and disasters for others. Any sort of filter will distort the data to some extent and many will delay the result, which can be a problem for fast real-time systems. It’s often better to use careful analog design to minimize the noise before it hits an ADC.

Jack provides a couple of design tips, e.g., using guard traces, optoisolators, or ferrites depending on the circumstances. He ends with a warning that we should put careful analysis into our designs and not forget the basic electronics/circuit theory we studied way back when we were in college. As Jack says, “Electronics is a lot of fun.”

And now for the more universal problem that Jack identifies.

5 – Weak managers or team leads

Jack starts out with Management 101:

Managers are supposed to manage people. That means a lot of things: protect the team from frivolous distractions, make sure they have the needed resources, cull poor performers, and much more.

He then addresses the importance of following firmware standards (and why managers need to enforce the standards and provide the means to ensure conformance, like code reviews or automated tools).

Jack goes on to briefly discuss Deming’s Plan-Do-Check-Act (PDCA) cycle for product improvement, and then gets into why it’s imperative that managers not be constantly interrupting work with “a barrage of text messages, email, IMs and the like.” And Jack really doesn’t like the “open office” concept, which he sees as a productivity destroyer.

Managers must find ways to enhance, not destroy, productivity.

He then gives a shout out to the importance of management placing an emphasis on quality.

Whether you’re a manager or not, Jack’s piece is well worth a quick read as a reminder of the value of good management and the responsibilities that managers have.

A couple of weeks ago, I posted a summary of Jack Ganssle’s first two articles (from a series currently running on embedded.com) discussing reasons Ten and Nine why embedded development projects “run into trouble.”  Here are reasons Eight and Seven:

8 – The undisciplined use of C and C++

As embedded systems developers we quite naturally rely on C and C++ (along with C#, Assembler, Java, Ada, and a variety of scripting languages including Linux shell scripts, Perl scripts, Windows batch files, by the way…). So I was quite naturally intrigued by what Jack has to say about their use.

First off, he reminds us that language-wise, C is right up there with Latin, dating back to 1972. (Which, he points out, makes it older “than most embedded developers”. Just not me.) He notes why C and C++ are so attractive:

C is uniquely suited to deeply-embedded work as it excels at manipulating bits, bytes and hardware. It’s a great choice for real-time work as it doesn’t have some of the problematic features of more modern languages, like garbage collection. C++, of course, brings the advantages of object-oriented programming to the discipline.

But then goes on to talk about why he believes that using them can be dangerous, especially for those lacking in expertise.

C/C++ have unspecified, implementation-defined, and undefined behaviors. Lots of them…There are plenty of ways to get stung even outside of these behaviors. Consider dynamic memory allocation. That’s a very useful feature for RAM-constrained systems. But it’s fundamentally non-deterministic. Malloc and free, malloc and free, and there’s no way to prove that some strange combination of inputs won’t cause enough heap fragmentation that the requested allocation fails.

When there’s no choice but to use dynamic memory, a disciplined use of C is to check malloc’s return value and take some action if malloc was unable to do its job. Yet I almost never see that test performed.

Jack then dives into the perils of pointers. Definitely worth reading him for the warnings he lays out. He ends his article by underscoring that he is, in fact, a proponent of C/C++. Just that he’s against their undisciplined use. Anyway, I’d like to point out that we, too, are proponents of C/C++ and very much committed to their disciplined use.

7. Bad Science

This article is a very cautionary discussion on the importance of understanding the science underlying the products we’re developing embedded systems for. Take nothing for granted, he warns. Don’t assume you get the science. Delve into it and make sure that you’ve got things straight.

His illustrations are interesting. For starters, he discusses a system built in the wayback that used IR light to measure grain proteins.

Spinning filters created 800 distinct wavelengths which impinged on the sample. Lead sulfide detectors read the reflected signal which was digitized and handled by an 8008.

Did you know lead sulfide detectors are more sensitive to temperature variations than to light? We didn’t. The data was garbage, and for months we sought answers before learning this simple fact. More months were lost as we tried cooling them with Peltier plates… and they got too cold. The final solution was a combination of carefully-controlled Peltiers and heating elements.

He provides another example that involves using carbon tetrachloride. Bad decision:

We didn’t know that carbon tet is quite toxic when inhaled; OSHA nailed us.

The substitute they used just plain didn’t work.

The solution to these and other bad science problems: do your research.  For Critical Link, that means working in close partnership with our clients to make sure we understand the underlying science behind their applications, and the implication this has for the embedded solutions we develop for them. (I’m going to do a bit of a brag here: we’re really good at this.)

Jack’s series is really a must-read for engineers in the embedded space, and I’ll be posting summaries of the other reasons over the next couple of months.

A couple of weeks ago, embedded.com started running a series by Jack Ganssle that gives Ganssle’s Top Ten list of “How embedded projects run into trouble.” As an engineer whose worked on a lot of embedded projects over the years, and whose company specializes in them, this series was obviously going to be one that caught my eye. (Not that any project I’ve ever worked on has run into any trouble…) So far, Jack has published articles on his Numbers 10 and 9. Here’s a summary of his thoughts on some common reasons projects go awry:

10 – Not enough resources allocated to a project

I suspect this is a situation we’ve all gotten involved in at some point during our careers. Jack cites a number of culprits here, including failure to invest in tools that help boost productivity. Not a problem at Critical Link: we’re big believers in providing our developers with the tools they need. He then talks about understaffing, pointing out that it “always results in higher costs.” This problem is compounded by the “mundane activities” that engineers can get caught up in “that were once handled by lower-wage workers.”

I like his point about the importance of management support:

A crucial role of a boss is to shield the team from anything that distracts them from getting the work done.

And Jack’s final point really resonates, where he discusses developing firmware, and whether to do it in-house, outsourced, or buy something off-the-shelf.  Sometimes it may seem cheaper to do something with the resources on hand, rather than paying out of pocket. He uses an example in which a colleague told him that:

… his company couldn’t afford the $15k needed to buy a commercial RTOS, so his team would write their own. I checked: the commercial version was 6000 lines of code. Multiply that by $20-$40/line [industry estimate for developing firmware code] and that $15k looks like a blue light special.

This seems to be as much about investing in tools as it is about hiring consultants, but point taken:

Penny pinching always winds up costing more.

Amen to that!

9 – Jumping into coding too quickly

The Ganssle series isn’t completed as yet, but I’m thinking that this point would be closer to my Number 1 than the Number 9 that Jack gives it. At Critical Link, our experience as systems engineers means we place tremendous importance on planning and being careful about our requirements. Here’s a number that he uses:

PMI’s Pulse of the Profession states that 37% of software project failures are due to inadequate requirements.

He notes that there’s a universal

…propensity to start coding as quickly as possible, even well before the functionality and design have stabilized. Yes, in some cases it’s impossible to know much about the nature of a product until a prototype exists, but generally we can get a reasonably-clear idea of what the requirements will be before writing C.

Jack also points out that:

Embedded firmware is a very different sort of beast from other software. We must specify the hardware early in the project, and that can’t be done unless the team has a pretty good idea of what they will build. Get it wrong and costs balloon. Schedules fall apart.

It IS hard to elicit requirements. Sometimes maddeningly difficult. But that doesn’t justify abdicating our responsibility to do as good of a job as possible to figure out what we’re building.

Jack then lists the major reasons “why requirements get shortchanged.” Worth a look at the article.

Embedded development is my territory, so I’m looking forward to the continuation of this series – and to see what Jack’s Number 1 is. Over the next few months, I’ll be posting on the other articles.  Stay tuned!

Moore’s Law seems to be a perennial topic in the tech press. (We’ve done our share of covering it as well. Here’s a 2017 post on the subject, and here’s an earlier take from 2014.)

For anyone who needs a reminder, Moore’s Law – which is now more than 50 years old – can pretty much be summed up as the number of components in an integrated circuit will double every year or so (without any increase in costs). Moore’s Law held for quite a while, but many observers of late have noted that we will eventually reach a point where doubling the number of components on a chip will become physically impossible. Which is not to say that IC’s continue to do more, even if the fancier things they’re doing may be pricy.

But people continue to argue Moore. And one of them is Intel’s CTO Mike Mayberry, who had an interesting piece on the subject a couple of weeks back on EE Times.

The thrust of Mayberry’s article is not to debate whether Moore is dead. It’s to offer a loo

For Mayberry, the future of Moore starts with CMOS scaling.

CMOS scaling is not yet done, and we can see continued progress as we improve our ability to control fabrication. We are not so much limited by the physics as by our ability to fabricate in high volumes with high precision. It is difficult, but we expect it to continue.

Here’s Mayberry’s illustration of what the next generation of Moore’s Law looks like:

Mayberry spends a bit of time describing what Intel’s up to with 3D and with technology that “can drastically improve power-performance.”

Unfortunately, [these new devices] are not a simple replacement for CMOS. So we expect to integrate these in a heterogeneous manner, likely as layers, combining the goodness of scaled CMOS along with novel functions provided by these new devices.

He also mentions that those “novel architectures” are designed to handle the data explosion that’s been going on (and is only, in my view, going to continue to accelerate).

Mayberry’s bottom line on Moore’s Law:

Pulling all this together, we expect the economic benefits of Moore’s Law to continue even as the elements look different than when Moore made his original observation. We need not be distracted by the debate but should continue to deliver ever-better products over the next 50 years.

Whether what Mayberry envisions is the future of Moore’s Law or not, I think that one thing we can count on is IC’s becoming more and more powerful.