80 responses to “Memory Safe C/C++: Time to Flip the Switch”

1. SoK = Systematization of Knowledge

   It’s a weird recent thing. Computer security does not have any good journals. Hence one of the most prestigious Tier-1 conference, IEEE Security & Privacy has introduced this category for presenting surveys instead of novel research output. This category is called SoK and authors need to prefix their paper submission accordingly upon submission.

2. The SoK stands for ‘Systematization of Knowledge’, which is a special track at the IEEE Symposium on Security and Privacy since 2010 that asks for ‘work that evaluates, systematizes, and contextualizes existing knowledge’. (See, for example, http://www.ieee-security.org/TC/SP2013/cfp.html).

3. Thanks Bart and rpw! I’d missed this academic meme.

4. I think the problem is that there is no safe solution. your best bet is to use an as an version of chrome / Firefox, but uaf is still a problem, right?

5. Hi anon, I haven’t yet read the paper but doesn’t SoftBound+CETS give decent protection against use-after-free?

   http://acg.cis.upenn.edu/papers/ismm10_cets.pdf

6. How much of this could be injected into the toolchain on a distro like Gentoo? That would seem to make discovery of compatibility issues much more rapid. There’s also gentoo-hardened, whose maintainers would probably be very, very interested in trying this sort of thing.

7. There is a key disconnect between research and industry here that I see as being a problem in practice: research builds solutions that will work on SPEC; industry has more complex needs. The example I always use when thinking about the engineering challenges for these kinds of tools is Firefox, which has:
   * custom malloc implementation
   * dynamic loading (not even linking!) of libraries
   * JIT
   * high performance sensitivity (engineers chase 1% performance regressions)
   * inline and out-of-line assembly
   * C code that needs to be architecture-specific (basically, reflection)
   * very large (3 million lines of C++; http://quetzalcoatal.blogspot.com/2011/08/not-so-random-mozilla-central-factoids.html has more out-of-date stats)
   * multithreaded
   * inherently imprecise analysis (just about every pointer and every vtable call could be a call to external code)

   It would be hard to turn some solution on by default unless it can cope with pretty much all of these things with minimal or no code modification necessary–these problems are not going to be limited to just Firefox, but other major packages in the distribution, such as SQLite, Apache, OpenJDK, etc.

8. Mike, do you know if Gentoo can be compiled with Clang yet? If so, then there are two options. First, SoftBound+CETS:

   http://acg.cis.upenn.edu/softbound/

   Second, Clang’s address sanitizer, which is a bug-finder rather than a real memory safety solution, but which still should provide some amount of protection. It would be really interesting to see how far we could get compiling Gentoo using these tools.

9. Joshua, there’s no question that there’s some serious engineering work to do and that it’s hard to motivate researchers to do it. Another nasty chunk of work would be involved in making Linux kernel modules memory-safe by default. But anyway, all we can do is try to make these things to work and solve the problems that come up and keep trying…

10. Firefox and Chrome both definitely compile under clang, infact, ASAN is commonly used for bugFinding. I would be interested in seeing a CETS+SoftBound build of Firefox. My suspicion is that it won’t have much overhead; neither browser is CPU bound and both Firefox and Chrome are definitely usable under ASan.

    John: Do you know the authors? Can you try and ask them to get a version of Firefox / Chrome working with their system?

11. anon, I don’t know them but I’ll ask anyhow. Will report the results here, or maybe one of them can comment. (Update: mailed them evening of Tuesday 4/23.)

12. You may not know us, but we read your blog. 🙂

    Thanks for highlighting out work. Let me try to answer some of the questions raised.

    SoftBound+CETS is a research prototype. We’ve run enough code that we’re comfortable with the performance estimates and basic approach, but the prototype is mostly the efforts of one person (Santosh, now a first-semester professor at Rutgers), so he just haven’t been able to put in the engineering effort to making it really bullet proof. So, no, the current prototype is almost certainly not robust enough to compile Firefox or Chrome.

    Also, the most recent results and description of the work can be found in Santosh Nagarakatte’s dissertation from 2012:

    http://www.cis.upenn.edu/~milom/papers/santosh_nagarakatte_phd.pdf

    The SoftBound+CETS implementation code can be found in the SAFECode sub-project of the LLVM main SVN repository. We’ve been trying to keep it current with LLVM as they release new versions of LLVM, but LLVM changes rapidly enough, that likely isn’t possible much longer.

    There is, however, an industrial implementation of something quite similar to Softbound: Intel’s “Pointer Checker” as part of its most recent releases of its C/C++ compiler:

    http://software.intel.com/sites/default/files/m/d/4/1/d/8/Pointer_Checker-Webinar.pdf

    http://d3f8ykwhia686p.cloudfront.net/1live/intel/Intel_PUMag_Issue11_Pointer_Checker.pdf

    From reading the description of how it works, it is extremely similar to the bounds checking proposed in our SoftBound PLDI paper: pointer based with disjoint metadata. Although the bounds checking is similar, the use-after-free checking we described in our CETS paper is much lighter weight than what Intel implemented.

    As for the specific issues asked about, we cover several of them in our various papers, but let me comment further:

    Libraries: the long-term plan is that all the libraries would also be recompiled with SoftBound+CETS, in which case all the static and dynamic linking just works. If the library is not recompiled, a whole host of issues comes up. First and foremost is that calling unsafe code means that the program is executing unsafe code, so all bets are off. The same is true in inline assembly. In some instances, you might be able to create wrappers for unsafe code that allow for interaction, but that is a fairly manual process. We did it for many of the standard C libraries to get our benchmarks to compile, but it isn’t a trivial thing. But I think this issue is true for any such system.

    Multithreading: multithreading support in SoftBound+CETS is okay, but not something we have focused on too much. If the program is free of data races (which is required by the C/C++ memory consistency model) and avoids low-level atomic usage, then adding the necessary synchronization to the SoftBound+CETS data structures is fairly straightforward. But in such an implementation, errant data races could compromise memory safety. That is regrettable, but still better than having no memory safety at all. See Section 6.7 of Santosh Nagarakatte’s dissertation.

    Custom memory allocators: custom memory allocators work if you’re willing to annotate the allocator to mark bounds and participate in the object identification scheme. So, just a few small code changes to get the protection. Without the code changes, the entire allocation pool would just be one big object, so it wouldn’t give the fine-grained protection you would expect (but it wouldn’t really “break” either).

13. I’d also like to say a bit about the history of the project. Our project started out as a *hardware* project, actually. The project was a reaction to a seminar course I taught in 2005 on hardware support for security, which included CCured and Cyclone papers as assigned reading. We wondered: could hardware support help overcome some of the compatibility and performance issues with such approaches? We starting working on it and it resulted in a ASPLOS paper in 2008 with the title: “HardBound: Architectural Support for Spatial Safety of the C Programming Language”.

    http://acg.cis.upenn.edu/papers/asplos08_hardbound.pdf

    We then realized that many of the same ideas we had in hardware could, in fact, be applied to a software implementation as well. We called that “SoftBound”, and published a paper about it PLDI 2009. We had another follow-up paper “Watchdog” on the hardware-centric implementation at ISCA 2012:

    http://acg.cis.upenn.edu/papers/isca12_watchdog.pdf

    Intel has also been working on similar hardware support. For evidence, see US Patent application 2011/0078389 A1:

    “Managing and implementing metadata in central processing unit using register extensions”
    http://www.google.com/patents/US20110078389

    I’ve heard rumors that Intel is planning adding hardware support for this. The fact that Intel released the SoftBound-like Pointer Checker as part of their compiler seems to lend evidence to that possibility.

    So, what will it take? Perhaps if Intel adds hardware supports and really pushes it, maybe that will be what it takes to make C/C++ memory safe.

14. I think your embedded in Academia because you need to get a refund on the shitty education you got so you choose to write about topics in which you or your readers really have no clue about so you can appear like you actually spent your money on something of value where as in all actuality you are no more qualified to write about then a two year old.

    I don’t know who, Laszl ´ o Szekeres ´y, Mathias Payerz, Tao Weiz, Dawn Songz nor do I care because if all they can do is write terribly dry white papers on a topic which they re-invent just to have something to write about then I don’t really want to read it anyway…. let alone understand why I should read material put forth by bumbling idiots who spent tenure with more morons who are all just rubbing each others backs anyway…

    There is nothing new in that paper and in the end their education seems to be as wasted as yours… In the next 20 years you personally and all of your readers can look back at where they are and ask yourself only 1 question… Was the Ivy League Education someone paid for me really worth it in the end?

    Great blog and great ideas… :-/

    Definitely won’t have to worry about my comments again!

15. Hi Milo, thanks for the detailed response!

    It’s excellent that you folks have kept SoftBound+CETS up to date with LLVM.

    Even if SoftBound and related tools have these limitations, it would still be a valuable exercise to start making a Linux distro safe, working through tool problems as they come up. I may be able to devote some resources to this.

    One positive side effect is that the source code modifications supporting one safe C compiler would hopefully carry over and remain useful if another compiler were used later on.

    Randomly, for our Safe TinyOS project I spent a lot of time annotating source code to make it safe using Deputy, which was a lot more hands-on than SoftBound and company. It was actually a really good exercise.

16. Maybe I missed the point; however, wouldn’t it be easier to create a stricter compiler instead of going into billions of lines of legacy code from the outset and incorporating new methodology and packages?

    If the newer compiler detected problems with the memory allocation in the code, surely that would be the time to change it, not before?

17. Sometimes it’s better to start anew. C/C++ passed that point a long time ago. 44 years of patches over patches is enough.

18. One of the things about memory safety is that it really needs to be integrated into a compiler that understands language semantics quite deeply can establish quite a lot of inferences about code, eg that for

    for(i=0;i<n;++i) A[i]=f(B[i]);

    it needs to check that A[0] & A[n-1] and B[0] & B[n-1] fall within a contiguously allocated memory segment and that no other checks are needed. I don't mind potentially paying for memory safety, but I'd be put off applying memory safety checks that are over-the-top and hence have a big performance hit because they are so simply implemented.

19. if possible, avoid using C/C++ ( and other programming languages that allow direct memory access alltogether. Switch to managed ones like C# or even Smalltalk. Currently I am prograamming IOS, Objective-C , using ARC. To prevent maintainance headaches, I am not using mallocs etc. In effect, it is all lower level and should be controlled by the system, so I can concentrate on the app itself, thnks

20. Hi Simon, I am exactly talking about switching to a stricter compiler.

    davetweed, a modern memory safety implementation is pretty smart about this sort of thing. Not nearly as smart as we want it to be, but not that bad. I encourage you to download for example the SoftBound system and try things like this out.

    Ted, the problem is that you haven’t yet taken all of the legacy C/C++ code that runs on a modern OS, and rewritten it in a managed language. Please do that, then get back to us. Thanks!

21. @Matthew Flannery:

    Congratulations on submiting your Phd thesis on “Aspects of Social Fitness within groups of Computing Science Wannabes”.

22. The reality is this…. Good memory safe code comes from good programmers, not from modifying C, C++. Not C# as that statement really shows ignorance of the subject (no disrespect meant for the language). Dave Tweed nailed it. Code should be written and the compiler should catch the issues. Then the programmer should fix them! Thirty years of code only prove one thing, C++ and C are the best choice, and what mono and .net are written in. If .net and c# are safe then this only proves the C++ likewise is safe when a good programmer uses it.

    It’s like fixing a car, if you want to trust the kid at the tire shop to fix your vehicle, that’s what you get. Programmers that use C# to avoid memory issues are like the kid. C#, a good runtime language is limited. So those who think this is the answer don’t understand programming. They dabble with code that they think will be safe, while a pro knows whether or not it is safe.

23. Memory will always reside in memory. Think about that. With enough determination, rogue code will be able to read and alter that memory, despite your best efforts at protecting me from myself. The solution is to use better memory protection at the O/S level. Here is a tidbit about Windows. I don’t know how Linux implements memory protection. http://msdn.microsoft.com/en-us/library/windows/desktop/aa366785(v=vs.85).aspx

24. Perhaps the solution should be implemented at the hardware level.

    Intel’s 16-bit 80286 processor had a segmented virtual memory mode in which it was possible for each process to allocate up to 8192 segments of any size from 1 byte up to 65535. I recall a memory allocation validator that allocated a segment for every memory allocation, so the processor’s memory protection hardware would cause an exception if you tried to access unallocated memory.

    A similar implementation on 32-bit (or 64-bit) processors would be much more difficult – the number of pointers in a large program is much higher. Furthermore, even if segments are available, the cost of switching between them is much higher in modern processors. However, many programs are not CPU limited. Perhaps we should use some of the unused cycles on security, by re-introducing segmentation and paying the cost to switch between segments.

25. Hi John,

    you are absolutely right when you say:

    “there’s no question that there’s some serious engineering
    work to do and that it’s hard to motivate researchers to do it”

    In my opinion, a large part of the trouble with motivating researchers
    is that among large segments of the research community, there is the
    curious view that memory safety is a “solved problem”…

    Ranjit.

26. leave C++ alone. Hire good programmers. And use another language if you want memory safety.
    C++ is designed for performance not memory safety.
    Its like arguing that we should put revolution per minute governors on cars because they are unsafe. Just a fancy word for going 15 MPH.

27. Hi Daniel, please read Milo Martin’s comment 13: it sounds like Intel may be adding hardware support for bounds checking.

28. One comment that needs to be made here is that the whole point of C (and to some extent C++) is that it *isn’t* memory-safe: it should be understood that C is more of a high-level macro assembler than it is a high-level language — one often uses it specifically in situations where you don’t want memory-safe execution, such as writing operating systems or firmware. As was pointed out earlier in this thread by Doc Maynard, memory safety comes from good programmers: meanwhile, leave C alone and let it be what it is: a highly-flexible tool which, in the right hands, can be used to achieve things which only machine code would let you do otherwise.

29. @Nickels
    The problem is that today’s performance tweak is tomorrow’s exploit. The last thirty years have shown that when a “good programmer” leaves out (for example) bounds checking to improve efficiency because he feels it would be safe, that code is exactly the point where the hacker finds his next exploit.

    I’m not proposing Nanny State fixes to the problem, but anything that can add to the tool belt to improve safety like ASLR or NX is a step towards a safer computer. The most important thing though is educating programmers on good practices from day one.

    Unfortunately, that is one thing we haven’t figured out effectively yet.

30. Who are all these people?!

31. Dan, you are not correct. The vast majority of C/C++, even in an OS kernel, is intended to be memory safe. The problem is that sometimes, despite our intentions, it isn’t memory safe. A “safe-by-default” compilation mode — with an easy escape hatch for the small minority of code that actually needs to be unsafe — is an effective defense against implementation errors.

32. John, my response to this post was too large for a comment. So I blogged it here http://blog.leafsr.com/2013/04/memory-safety-protections-and-real-world.html

33. Chris, thanks for the pointer. I think your point of view is a reasonable one and that of course we should pursue these kinds of mitigations, but I also think there are plenty of situations where memory safety is a better choice.

34. John, Memory safety is required when all bugs are exploitable. But attackers have to exist and persist in an imperfect world the same as defenders. If you starve them of exploitable bugs its game over. This is (should be) the industry approach, academia continues to strive for program purity. A noble, but lost cause in my opinion.

35. Chris, one reason we may differ is that I’m not a security person and I deeply hate all memory corruption errors, not just those that lead to vulnerabilities. My view is that security might be the lever we use to push memory safety into deployment but it’s hardly the only benefit.

36. I have heard C# (2000) is a better language than C (from the 1950s) because it does all this stuff for you? Perhaps time to stop the laziness and learn a new code paradigm?

37. Jason, your comment and some of these others fall well below the usual standard here. These are getting in the way of a useful discussion. I expect better, and will start deleting if I have to.

    Did my article get linked to some site other than HN/Reddit today, perhaps?

38. I want to stress that hardware support might be the key to reducing the performance penalty. Our simulations (based on a ton of simulation and compute-intense workload assumptions) show just a 17% overhead on average for full memory safety (bounds checking and use-after-free checking). This is versus the 2x overhead introduced by the software-only implementation in the compiler.

    There are lots of code in which I as an end-user would gladly pay 17% to avoid security vulnerabilities and catch memory corruption bugs. As I mentioned above, based on Intel patent filings, rumors, and Intel’s software-only pointer checker tool, there is reason to suspect that Intel might be planning on such hardware support. That could really change the calculus.

39. John – That point is not lost on me at all. There is a lot of good that will come of memory safety, especially if you can bring it to all those hundreds of millions of lines of existing C/C++ code. This is part of why I am so interested in approaches like NaCl/PNaCl. Their end goal is the same (bring safety to existing code) but obviously with a drastically different approach. As always I look forward to your next post.

40. Ranjit said:

    In my opinion, a large part of the trouble with motivating researchers
    is that among large segments of the research community, there is the
    curious view that memory safety is a “solved problem”…

    True, though with huge credit to SoftBound etc., the engineering is hard enough for even those of us who completely believe this is a problem to find a good cost-benefit argument for the needed resources. It’s a little bit like testing research’s issue of making it worthwhile to not just say “we tried it on some container classes, and it looked awesome.” Or when model checking papers had to produce less toy examples.

41. Cognitive bias can slip into such discussions.

    If you ask someone, will you give up X to get Y, they will often say “no”. Is it worth slowing down your code by 50% to get memory safety? No! (They value the performance they already have more than security.)

    But if you ask them the reverse question, will you give up the security of memory safety for a 50% performance improvement, they might say; What? Are you crazy, make my machine more insecure because of some performance. That’s crazy. I could get fired for that. (They value the security they already have more than performance.)

    This sort of cognitive bias is well documented; just by framing the same situation in a slightly different way, people will answer it differently:

    http://en.wikipedia.org/wiki/Endowment_effect

    Of course, I’m not saying it makes sense to give up 10x performance for security, etc. But as such tools have gotten better, we are close and closer to the point in which the endowment effect begins to play a role.

42. why would i need to allocate/access memory directly (well, except for harware registers etc.) when i can use arrays, scalars, objects, collections, sets, ordered lists etc. .? Let the compiler figure it out, so i can concentrate on the essentials? Take a look at Smalltalk for instance (e.g. Squeak)

43. Milo, I agree re. the endowment effect! That is really my main motivation for pushing an agenda where safety is opt-out instead of opt-in.

    The HW performance figure is really impressive, although I feel compelled to insert something here about my inherent and large distrust for simulation results in computer architecture research :).

44. I will say that I think the motivation problem somewhat solves itself, as long as anyone “cares” — eventually, academic research standards also adjust to require more plausible paths from an idea to engineering implementations.

45. Ted, I am not talking about new code, where arguments like yours are applicable. I’m talking about the kind of code that is causing problems like these:

    http://www.ubuntu.com/usn/

    Notice that this is page 1 out of 42. Some of these vulnerabilities would not have happened if these programs were executing in a memory safe environment. That is all.

46. regehr, apologies I am only a software manager of a team of 15 aerospace developers? We stopped using C when a memory leak almost caused a disaster which would have perhaps led to loss of life. Feel free to continue in the stone age of development!

47. regehr, apologies I am only a software manager of a team of 15 aerospace developers? We stopped using C when a memory leak almost caused a disaster which would have perhaps led to loss of life. Feel free to continue in the stone age of development!

48. Jason, see my comment #45. None of this is about liking C/C++ or writing more of it. This is about how our entire trusted computing base for Windows, Linux, etc. is all unsafe code. We are stuck with a lot of it.

    Much embedded software is still getting written in C, I’m happy that your team’s code is not!

49. John, oh yes, certainly don’t really trust the 17% number. I also don’t trust the results from such hardware simulations experiments. To quote Einstein: “An experiment is something everybody believes, except the person who made it.” 🙂

    But even a simple back-of-the-envolope calculation supports that low overhead for doing such checking in hardware is not unreasonable.

    Let’s assume a simple in-order processor in which all instructions take a single cycle. In such a setting, the inputs to the calculation are primarily: (1) the percentage of all instructions that access memory (say, one third) which is multiplied by the bounds check cost (say, one cycle) and (2) the percentage of all memory operations load/store pointers from the shadow space (say, 25%) multiplied by the cost (say, two cycles). That give us 1 + (33% * 1) + (33% * 25% * 2) = 1.5 So that gives a ballpark estimate of 50% operation overhead, which is likely to result in relatively less performance overhead in an out-of-order dynamically scheduled superscalar, as these operations are all off the critical path.

    In fact, there are some aspects of our simulations that are pessimistic, so the overhead could actually be lower depending on how the hardware implements the checking.

    I suspect that Intel will only release such hardware if the overheads are low enough to be deemed acceptable. In fact, one possible situation that could delay hardware support is if Intel implements it, finds its performance is lacking, and then pushes it back. They did this with multi-threading, for example. All Pentium IV chips had multi-threading, but it was disabled on the first-genreation Pentium IV chips due to bugs and/or poor performance.

50. Jason, I don’t think anyone working on memory safety on C/C++ would argue against using modern programming languages if they fit your needs! Yes, of course!

    But, for various reasons—both legacy reasons and deficiencies with the current suite of modern memory-safe languages—C/C++ is still used. Java is almost 20 years old now, and I really would have expected it to crowd out C/C++ completely, but it has not. If anything, we have seen a resurgence in interested in C++ and “native code” for mobile applications. Java has taken over in some key niches (web services, for example) but my assessment is there is a reason that our web browsers, compilers, and databases are not written in C/C++ (and not Java or C#).

    The question becomes: why? What are the deficiencies—real or perceived—with the modern languages available today vs C/C++. I think performance is actually a big one. Java, for example, adds layers of indirection and thus makes it extremely difficult to express some data structures and computations efficiently. There was a paper “Four Trends Leading to Java Runtime Bloat” by Nick Mitchell et al that described some of the cases. A search for java bloat will result in several interesting academic papers discussing such issues. Of course, some of the same bloat can happen in other languages (say, C++) but there are language design aspects of Java that make it really difficult to inline objects to reduce such overheads. Like C++ or not, its templates and inline objects can results in some really efficient use of memory.

    Based on some recent research we’ve been doing, we have some results that indicate we can create a language nearly as efficient as C++ but also totally memory safe and type safe, hopefully capturing the best of both worlds…