This chapter explains how to use the Debug version of SmartHeap. SmartHeap is shipped with two sets of libraries: Runtime and Debug. The Debug library supports all the same APIs as the Runtime library, plus numerous debugging APIs you can use to isolate and diagnose heap related programming errors. We recommend that you use the Debug SmartHeap library during development and testing in order to catch memory bugs as early as possible.
The Debug version of SmartHeap is designed to perform comprehensive heap error detection with minimal, if any, changes to your application. In addition to the errors that Runtime SmartHeap detects (see §2.5.1 for a list), Debug SmartHeap detects the following errors:
Invalid parameters. All parameters are validated, including memory pools, pointers, handles, flags, and buffers.
Double freeing and other references to previously freed memory. You can defer freeing memory so that references to freed memory are guaranteed to be caught. Normally, free blocks are recycled for subsequent allocations, so writing to a previously freed block often overwrites data in another part of an application where the block is now in use again.
Writes into free memory. Free blocks are always filled with a unique pattern; SmartHeap thus detects a write into any byte of any free block.
Overwrites before or after an allocated block. SmartHeap detects an overwrite as small as one byte beyond the requested size, even if the block’s actual size is larger than requested. You can customize the size and value of “guards” that are placed before and after every allocation, including fixed-size allocations.
Overwrites over internal heap data structures.
Overwrites over the handle table, caused by writing directly to a SmartHeap handle.
Leakage. Debug SmartHeap includes a leakage reporting facility. You can group or “checkpoint” allocations and find and report all allocations not freed in one or more groups.
Retaining a reference to an allocation passed to realloc. Debug SmartHeap realloc always moves the allocation in order to catch this error.
Modifying an allocation marked as “read-only.” You can mark individual allocations as read-only — SmartHeap then reports an error if the allocation’s contents are changed.
Freeing/reallocing an allocation marked as “no free” or “no realloc.” An allocation that should not be freed/realloced during some stretch of a program can be marked as such, and SmartHeap will report as an error any attempt to free/realloc the allocation.
Freeing a variable-size allocation with MemFreeFS, the fixed-size de-allocation API.
Windows, 16-bit · References to non-shared memory owned by a different task.
Each allocation in Debug SmartHeap keeps track of the following information:
The SmartHeap API that created the object (for example, MemAllocPtr or malloc).
The file name, line number, and pass count* where the object was allocated. This information allows you to set a breakpoint at the nth pass over the line where the allocation occurs, allowing you to track any SmartHeap allocated object from its creation and follow that object’s life in the debugger to the point where an error occurs.
An allocation count that uniquely distinguishes the block from all other historical and future allocations in the current task or process.
A checkpoint — the grouping or context of an allocated block.
The actual requested size and any flags used in the creation of the object — so SmartHeap can report the exact parameters to the allocation call at any time in the future.
A checksum used if the block is marked as read-only.
Flags indicating whether the block is marked “no free,” “no realloc,” or “defer freed.”
When an error is detected, SmartHeap reports the following information:
The SmartHeap API/file/line/pass count of the call that detected the error.
The creation SmartHeap API/file/line/pass count, checkpoint, and allocation count of the object that was corrupted, if applicable.
The SmartHeap API/file/line/pass count where the object was last known to be “OK” (for example, not overwritten yet).
The memory address where the corruption was detected and, if the address is valid, a hex and character memory dump of the contents of the corrupted location.
SmartHeap reports errors with any combination of the following (you can specify which):
A prompt to the user interface (message box or command-line prompt, depending on the platform), where the user can abort/retry/ignore.
Output to the debugging console (secondary monitor or debugger log window).
Output to a log file, which is closed after each error to ensure that no information is lost if the application crashes.
User-defined error handler that takes control of I/O and handling.
User-defined tracing of entry/exit of all SmartHeap APIs.
Setting up for Debug SmartHeap can be as simple as relinking with the Debug SmartHeap library. Recompiling your source files with the MEM_DEBUG manifest constant will give you the most detailed debugging information. SmartHeap provides special debugging macros for malloc and C++ operator new that you can optionally include in your source files that call those functions.
You do not need to recompile your application or make any calls to the debugging APIs to take advantage of SmartHeap’s memory error detection facility. Just relink with the Debug SmartHeap library, usually named “libshXXXd.a” (for UNIX platforms) or “hawXXX.lib” for Windows. In some older versions of SmartHeap, these libraries may be named “shlXXXXd.lib” (for statically linking Debug SmartHeap into your application) or “shdXXXXd.lib” (for the Debug SmartHeap DLL). For the exact library file names, see the Getting Started and Platform Guide.
Linking with the Debug SmartHeap library causes SmartHeap to perform full error detection. SmartHeap will report overwrites, double freeing, writes into free blocks, and everything else listed in §3.1, except for the source file, line, and pass count.
For SmartHeap to report the source file, line, and pass count when it detects errors, you need to recompile your source modules for SmartHeap debugging. You can compile any subset of your source files for debugging and link a combination of modules that were compiled for either Debug or Runtime SmartHeap. SmartHeap reports file/line/pass count only for the calls that originated from modules compiled for debugging.
Windows 32 bit If your application is compiled with Visual C++ PDB debugging information, Debug SmartHeap obtains file names and line numbers directly from the debugging information. So if you aren’t calling Debug SmartHeap APIs, there is no need to recompile your application for Debug SmartHeap
When SmartHeap detects an error, the resulting error report can include the source file, line number, and pass count both for the location at which the allocation was created and the location at which the error was detected. (If more than one allocation is created from the same source file and line, then the pass count identifies on which pass of that source line a specific allocation was created.)
If you want SmartHeap to report the source file, line number, and pass count, you need to recompile your source modules for Debug SmartHeap. You can compile any subset of your source files and then link modules (as described in the previous section) regardless of whether they’ve been recompiled for Debug SmartHeap. SmartHeap reports file, line, and pass count only for calls that originated from modules that you’ve compiled for Debug SmartHeap.
Note You can’t use the following procedure to recompile source modules that include:
Overloaded declarations or definitions of operator new.
Calls to operator new with the placement syntax (even in header files).
If you do recompile these modules with DEFINE_NEW_MACRO defined, you’ll encounter a compiler syntax error. Instead, use the procedure in the next section, “Recompiling persnickety C++ modules.”
For each file for which you want SmartHeap to report source file, line number, and pass count:
If you’re coding in C, define the macro constant MEM_DEBUG on the compiler command line (for example, DMEM_DEBUG=1).
Or
If you’re coding in C++, define the macro constants MEM_DEBUG and DEFINE_NEW_MACRO on the compiler command line (for example, DDEFINE_NEW_MACRO=1 DMEM_DEBUG=1).
Note MEM_DEBUG causes SmartHeap’s debug-only APIs (dbgMemXXX) to be prototyped. When MEM_DEBUG is not defined, smrtheap.h defines these dbgMemXXX functions as empty macros. This lets you have calls to SmartHeap debugging APIs in your source code without the need for #ifdef statements. When you compile for runtime, the debugging calls are automatically compiled out.
Include one or more of the following SmartHeap header files. If you want file and line information for:
SmartHeap APIs, include smrtheap.h.
ANSI C APIs (malloc, calloc, realloc, and free), include shmalloc.h.
C++ operator new, include smrtheap.hpp.
Note shmalloc.h and smrtheap.hpp both include smrtheap.h, so normally you need to include only one of these header files in each of your source files. If you want to debug both ANSI C and C++, you’ll need to include both shmalloc.h and smrtheap.hpp. You can include shmalloc.h to define a debugging malloc even if you don’t have SmartHeap override the malloc function. This is because shmalloc.h defines malloc, etc. as macros rather than functions.
The #include directive must appear after any compiler or library header files that declare malloc (for example, stdlib.h) or new (for example, new.h or afx.h):
#include <smrtheap.h>
Note If you’re using the Debug Microsoft Foundation Class Library, you don’t need to include smrtheap.hpp to have file and line information for operator new. SmartHeap DEBUG_NEW is completely compatible with MFC DEBUG_NEW.
HeapAgent note If you already include heapagnt.h, then you don’t need to also include shmalloc.h or smrtheap.hpp unless you’re using SmartHeap APIs in your application. The HeapAgent header file defines debugging versions of malloc and new when MEM_DEBUG is defined.
Recompile the file.
When you’ve finished steps 1 through 3 for all files, relink with the Debug SmartHeap Library, as described in the previous section.
Note Compiling for Debug SmartHeap does not in itself cause SmartHeap to detect any additional errors — all errors are detected by simply linking with Debug SmartHeap. Compiling for debugging provides file/line/pass count information if errors are detected and provides access to the dbgMemXXX functions. If you want to have the extra error checking of Debug SmartHeap without changing any of your source files or recompiling, then you only have to link with the Debug SmartHeap library. You can always recompile individual modules for debugging to track down specific errors at a later time.
Important! SmartHeap can detect additional errors, and can detect some errors earlier (for example, overwrites and double freeing), when you adjust Debug SmartHeap settings with such routines as dbgMemSetSafetyLevel, dbgMemSetCheckFrequency, and dbgMemPoolDeferFreeing. These functions are described later in this chapter and in Chapter 4, “Reference.” The module in which you call these debug-only APIs must be compiled with MEM_DEBUG defined for the routines to have any effect. However, your other modules need not be compiled with MEM_DEBUG to get the benefit of more rigorous checking — just compile the modules that call dbgMemXXX routines.
C++ syntax makes it difficult to record the file and line from which C++ operator new are called. operator new isn’t called with the C function call syntax. As a result, it can’t be defined as a macro that passes additional parameters with function syntax, as is malloc and all of the other SmartHeap APIs.
C++ operator new can accept extra parameters if called with the placement syntax. SmartHeap therefore defines a special macro that causes calls to operator new to pass file and line information with the placement syntax, like this:
new x 1 new(__FILE__, __LINE__) x
Unfortunately, if you have calls to operator new that already use the placement syntax, then SmartHeap’s normal macro substitution for operator new will cause a syntax error for those calls, as follows:
new(y) x 1 new(__FILE__, __LINE__)(y) x
Windows 16-bit Similarly, the SmartHeap operator new macro will expand incorrectly for calls to __huge operator new in 16-bit Microsoft Visual C++:
new __huge x 1 new(__FILE__, __LINE__)__huge x
If your application includes overloaded declarations or definitions of operator new, includes calls to operator new with the placement syntax, or includes calls to __huge operator new in 16-bit Microsoft Visual C++ (even in header files), use the following procedure to enable SmartHeap to include source file, line number, and pass count information in reports.
Define the macro constant MEM_DEBUG on the compiler command line (for example, DMEM_DEBUG=1). Omit the macro constant DEFINE_NEW_MACRO, since that causes SmartHeap to define a macro for operator new.
Include smrtheap.hpp. The #include directive must appear after any compiler or library header files that declare new, such as new.h or afx.h:
#include <smrtheap.hpp>
Note If you’re using the Debug Microsoft Foundation Class Library, you don’t need to include smrtheap.hpp to have file and line information for operator new. SmartHeap DEBUG_NEW is completely compatible with MFC DEBUG_NEW.
HeapAgent note If you already include heapagnt.h, then you don’t need to also include smrtheap.hpp. The HeapAgent header file defines a debugging version of operator new when MEM_DEBUG is defined.
Replace instances of the symbol new for which you want file and line information with the macro constant DEBUG_NEW. If MEM_DEBUG is defined (see step 1), smrtheap.hpp defines DEBUG_NEW as a special version of the new operator that passes file and line information to SmartHeap. If MEM_DEBUG is not defined, smrtheap.hpp defines DEBUG_NEW as new.
For placement calls to operator new, define new as the macro DEBUG_NEW for selective portions of code, and undefine the new macro immediately before the conflicting instances of the symbol new.
For example:
#define
MEM_DEBUG 1
// make sure DEFINE_NEW_MACRO isn't defined
#ifdef
DEFINE_NEW_MACRO
#undef DEFINE_NEW_MACRO
#endif
#include
<smrtheap.hpp>
operator new(size_t, void
*myArg);
operator new(size_t, void *myArg)
{ return myArg;
}
// define new as DEBUG_NEW _after_
// smrtheap.hpp, and
_after_ any
// declarations and definitions
// of overloaded
operator new
#define new DEBUG_NEW
void
main()
{
char *str1 = new char[20];
// undefine new macro
immediately before
// calls to operator new that use the
//
placement syntax
#undef new
char *str2 = new(str1)
char[20];
char *str3 = new(str1, str2) char[20];
// then
restore macro for subsequent calls
// to new
#define new
DEBUG_NEW
char *str4 = new char[20];
}
Recompile the file.
When you’ve finished steps 1 through 4 for all files, relink with the SmartHeap Library, as described at the beginning of this chapter.
Alternatively, you can avoid having #ifdef’s in your code by using DEBUG_NEW, DEBUG_NEW1, DEBUG_NEW2, and DEBUG_NEW3 in place of new for memory allocations. These macros pass file and line information to operator new, including placement syntax variants thereof, when MEM_DEBUG is defined, but expand simply to new otherwise.
DEBUG_NEW1 is used for placement syntax calls to new with one parameter, DEBUG_NEW2 is used for calls with two parameters, and DEBUG_NEW3 is used for calls with three parameters.
The example in step 3 above would be simply:
#define
MEM_DEBUG 1
// do not define DEFINE_NEW_MACRO
#include
<smrtheap.hpp>
operator
new(size_t, void *myArg);
operator new(size_t, void *myArg)
{
return myArg; }
void
main()
{
char *str1 = DEBUG_NEW char[20];
char *str2 =
DEBUG_NEW1(str1) char[20];
char *str3 =
DEBUG_NEW2(str1,
str2) char[20];
char *str4 = DEBUG_NEW char[20];
}
Important! If you are using Borland C++ 3.1 for DOS or Windows, or Borland C++ 1.0 for OS/2 2.x, then Debug SmartHeap new is not supported for allocations of arrays of objects. The reason is that the Borland C++ compiler does not support the placement syntax for allocating arrays of objects that have non-default destructors — a runtime crash occurs in the call to delete[] when freeing such arrays. For this compiler, you must #undef the new macro before calls to new that allocate object arrays, just as you do for calls with the placement syntax as described above. No other C++ compilers are known to have this problem. Borland has fixed the problem in BC++ 4.0 and higher for DOS and Windows, and in BC++ 1.5 and higher for OS/2.
This section explains how to use SmartHeap to detect a number of common memory bugs. Tips are also provided on how to prevent many of these bugs.
The Debug version of SmartHeap provides a mechanism for controlling how much time SmartHeap spends performing error checking. The function dbgMemSetSafetyLevel provides a course granularity of control. This function allows you to set the safety level to one of three values:
MEM_SAFETY_SOME, the lowest level, performs only checks that are O(c), such as validating parameters and checking guards for overwrites.
MEM_SAFETY_FULL, the default, performs additional O(n) checks such as scanning free lists to detect double freeing.
MEM_SAFETY_DEBUG checks the entire heap on every SmartHeap entry-point — making it O(n²).
The MEM_SAFETY_DEBUG safety level is very useful for detecting overwrites but can cause severe performance degradation. This is mitigated by another API, dbgMemSetCheckFrequency, which causes SmartHeap to validate the heap only on every nth entry point to the library. The default check frequency is every 10th call to SmartHeap. You can use dbgMemPoolSetCheckFrequency to control the check frequency for an individual memory pool.
See dbgMemSetSafetyLevel in §4.2, “Function reference,” for complete details on controlling the safety level in your application.
When an allocation is freed in Debug SmartHeap, it is not immediately recycled, but placed in a queue. When the queue reaches the deferred freeing queue length, the oldest item in the queue is recycled. This allows Debug SmartHeap to catch bugs such as double-frees, reads, and writes to previously freed memory.
You can use dbgMemSetDeferQueueLen or dbgMemPoolSetDeferQueueLen to control how many allocations are retained in the queue. You can use dbgMemSetDeferSizeThreshold to control the size of allocations whose freeing is deferred — blocks larger than the threshold are recycled immediately.
Debug SmartHeap provides routines to obtain information about pointers, memory pools, and debugging settings — dbgMemPtrInfo, dbgMemPoolInfo, and dbgMemSettingsInfo, respectively. dbgMemTotalSize and dbgMemTotalCount return the total size of all SmartHeap memory pools and the total number of heap allocations, respectively.
The best way to deal with bugs is to avoid creating them in the first place.
Important! To ensure compile-time checking, always include one of the SmartHeap header files in each of your modules that contains references to SmartHeap functions. SmartHeap memory pool and handle data types are declared as pointers to structures in smrtheap.h to maximize compile-time type checking. You should not have any casts to or from either of these types. Neither should you dereference pointers of these types or perform address arithmetic on them.
Note SmartHeap’s runtime error handling mechanism kicks in where the compile-time error checking leaves off. You should always develop, debug, and test your application with safety level set to MEM_SAFETY_FULL. This will detect a host of errors, from invalid parameters, to double freeing, to invalid shared-memory references, and many others. If you suspect a memory overwrite, try setting the safety level to MEM_SAFETY_DEBUG.
SmartHeap provides a number of APIs you can use for parameter validation. MemCheckPtr and MemPoolCheck are available in both Runtime and Debug SmartHeap. In addition, in Debug SmartHeap only, you can use dbgMemCheckPtr to validate both heap and non-heap pointers. Debug SmartHeap also provides dbgMemCheckAll, which validates all SmartHeap memory pools in a single call.
Double freeing is a common error that can cause you to waste hours or days tracking down the source of the problem. This is because the symptoms often appear long after the double freeing took place.
Depending on the implementation of the memory manager, double freeing may corrupt an entry in the free store (heap, free list, etc.) or cause the same memory block to be allocated twice later on. Either way, the symptoms are likely to be strange, and different in each instance of double freeing. Moreover, the problem may appear to “go away” with slight (but unrelated) changes to your program, only to resurface later in a different guise.
Debug SmartHeap can detect double freeing the moment it occurs when safety level is set to MEM_SAFETY_FULL. This detection is available in all SmartHeap APIs that pass a memory pointer or handle as a parameter. SmartHeap will report any reference to a previously freed allocation with the MEM_DOUBLE_FREE error.
Everything said about double freeing applies to memory overwrites as well, except the symptoms are even more obscure, and the source of the problem is often harder to pin down.
Following are a few of the myriad causes of memory overwrites:
When you create an allocation, you specify a size that is too small for the data the allocation is to contain.
You forget to recompile a module after changing the size of a structure or other object — code that is either allocating, writing to, or referencing fields of the structure then go outside of the structure’s bounds.
You write to a wild, dangling, or previously freed pointer.
You misuse a union or other overlapping area of memory so that one object steps on another.
You fail to NUL-terminate a character string, and subsequently copy such a string to another memory location.
Debug SmartHeap includes a number of facilities that are specifically designed to detect overwrites:
SmartHeap pads each block it allocates with leading and trailing guards. The guards are a variable-length region filled with a known, unlikely pattern of bytes. You can specify the size and fill value of guards with dbgMemSetGuardSize and dbgMemSetGuardFill, respectively (default fill is four bytes; default fill char is 0xFC).
Each time a block is referenced — when it is passed as a parameter or traversed during a full heap check — the leading and trailing guards are checked. If any guard byte doesn’t match the fill pattern, then SmartHeap reports it with the MEM_OVERWRITE or MEM_UNDERWRITE error code.
SmartHeap also validates its internal data structures. If they are found to be corrupt, the error MEM_BAD_BLOCK or MEM_BAD_FREE_BLOCK is reported.
You can detect memory overwrites by checking the validity of memory pools from time to time. SmartHeap provides the function MemPoolCheck for exactly this purpose. This function checks each entry in a given memory pool for validity, regardless of the safety level in effect.
You can set the SmartHeap safety level to MEM_SAFETY_DEBUG, so that MemPoolCheck is called at every nth call to a SmartHeap allocation or de-allocation API. Use dbgMemSetCheckFrequency to set the value of n; the default is every 10 calls. Setting safety to MEM_SAFETY_DEBUG will pin down most memory overwrites, but if you are overwriting memory somewhere in your code that is far from the nearest allocation or free, you sometimes need to do more.
Note We recommend that you place calls to MemPoolCheck at strategic points in your application, inside a MEM_DEBUG compiler flag that will be turned off in your final production version. These calls will act like an insurance policy, informing you immediately if you are causing a memory overwrite rather than having your customers inform you somewhere down the road.
Two common types of overwrites are writing into a block that has already been freed and writing into a block that moved when it was reallocated. This error can be exceedingly hard to track down because, by the time the overwrite occurs, the block may already be in use in another part of your application.
Debug SmartHeap fills blocks with an uncommon pattern as soon as they are freed. This makes it easy for you to spot free blocks and, more importantly, allows SmartHeap to detect an overwrite to any byte of a previously freed block. You can specify the fill value with dbgMemSetFreeFill (the default is 0xDD).
If you want to avoid having Debug SmartHeap detect writes to free memory, you can disable filling of free memory by calling dbgMemSuppressFreeFill. One reason you might want to do this is if a third-party library for which you don’t have source code is responsible for the write to free memory.
SmartHeap also provides a facility to help with the problem of overwrites into blocks that have been freed and subsequently allocated again. dbgMemPoolDeferFreeing causes all SmartHeap freeing APIs to mark blocks free rather than actually freeing them. The blocks are filled with the free pattern. Until you call dbgMemPoolFreeDeferred, these “defer freed” blocks remain as traps waiting for an overwrite into a free block.
When SmartHeap detects such an overwrite, it reports the error MEM_FREE_BLOCK_WRITE.
You shouldn’t assume anything about the contents of a memory block after it has been freed. With most memory managers, data previously stored in a (now) free block may still be there, leading to the possibility of a subtle bug that shows symptoms only after a completely unrelated change to the application.
Debug SmartHeap fills free blocks with a distinct pattern (default 0xDD, customizable with dbgMemSetFreeFill), thus eliminating any chance of relying on data remaining in free blocks. If you read and dereference a pointer stored in a free memory block in Debug SmartHeap, a memory access violation will result, making it very easy to track down such bugs.
Reads of free blocks commonly occur in code that frees one or more elements while traversing a linked data structure. For example:
/*
example list link structure */
struct Link
{
struct Link
*next;
int value;
};
/*
erroneous free-list function */
int FreeList(struct Link
*link)
{
while (link)
{
free(link);
link =
link->next; /* ERROR! link freed!! */
}
}
/*
correct free-list function */
int FreeList(struct Link
*link)
{
struct Link *toFree;
while (link)
{
toFree
= link;
link = link->next; /* read BEFORE freeing
*/
free(toFree);
}
}
A dangling pointer is a pointer to memory that was previously freed. Freeing a dangling pointer results in double freeing, as described in §3.3.3. Writing to the memory pointed to by such a pointer can result in a memory overwrite, as described in §3.3.5.
A wild pointer is similar to a dangling pointer: it may have been previously freed or may simply be an uninitialized sequence of random bits in a pointer variable.
If you dereference or modify the memory that a dangling or wild pointer points to, you’ll eventually encounter mysterious symptoms or crashes.
Debug SmartHeap fully validates each pointer that you pass to the library. If the pointer is not a valid address, or not a valid block of a SmartHeap memory pool, or if the block is currently free, an error will be reported. This is one safeguard against dangling pointers, but you aren’t likely to pass every pointer in your program to a SmartHeap function.
SmartHeap can help you find problems with uninitialized allocations. When the safety level is MEM_SAFETY_FULL or higher, Debug SmartHeap fills all blocks with an unlikely pattern as soon as they are allocated. You can specify the fill value with dbgMemSetInUseFill.
Note The function MemCheckPtr helps you detect dangling pointers. You can also use MemPoolCheck, as described in §3.3.4 above, to validate each block in a pool: if a memory pool is corrupted, writing to a dangling pointer is often the culprit.
Sometimes errors in a program occur not because of the wrong operation (for example, an illegal overwrite or invalid parameters) but because the correct operation is carried out at the wrong time.
For example, with a write into a free block, the write might be correct but the block might have been freed too early. In such a case it can be difficult to determine just where and when the block was freed.
To detect frees or reallocations that occur at the wrong time, you can use the dbgMemProtectPtr function to mark an individual SmartHeap-allocated block as “no free” or “no realloc.” SmartHeap then reports any attempt to free or reallocate the block with the error MEM_NOFREE or MEM_NOREALLOC, respectively.
A common symptom of a memory bug is an unexpected change to data. Some interactive debuggers have the ability to catch changes to data.
Debug SmartHeap provides a portable mechanism for handling this programmatically. You can use dbgMemProtectPtr to mark a memory block as read-only. If the contents of the block change, SmartHeap reports the error MEM_READONLY_MODIFIED.
SmartHeap maintains a checksum for blocks marked read-only. Whenever the block is validated (when it is passed as a parameter or when the heap is checked), SmartHeap recomputes the checksum to detect a change in value. This mechanism can be slow, particularly if you mark many blocks read-only, and/or if MEM_SAFETY_DEBUG is enabled. However, this may be the only way to track down a subtle overwrite bug.
Scribbling on the stack is a type of memory overwrite. It is caused by writing beyond the bounds of a local variable allocated on the stack. Stack scribbling can’t be detected by SmartHeap, because SmartHeap manages only dynamic memory on the heap (the stack is automatic memory).
The symptoms of this problem depend on what is scribbled upon. If you overwrite an adjacent local variable that isn’t subsequently referenced, there may be no symptom. If you overwrite the function return address, a memory protection violation (also known as a UAE) is almost certain.
The following example shows a typical stack-scribbling scenario:
int
DisplayError(char *msg, int num)
{
char buf[64]; /* stack
variable */
/* format and display the message;
we may
overwrite buf here!! */
sprintf(buf, "Error #%d: %s.",
num, msg);
MessageBox(GetFocus(), buf, NULL, MB_OK);
}
You can often avoid potential stack scribbling by dynamically allocating arrays, strings, and other variable-size objects rather than assuming what the maximum size will be in a local variable declaration.
The following solution avoids potential stack scribbling:
#include
<smrtheap.h>
/* StringPool is a global variable
previously
initialized with MemPoolInit. */
extern MEM_POOL
StringPool;
int DisplayError(char *msg, int num)
{
/*
allocate buffer in which to format msg */
char *buf = (char
*)
MemAllocPtr(StringPool, strlen(msg)+20, 0);
/* return
if allocation failed */
if (buf == NULL)
return FALSE;
/*
format and display the message */
sprintf(buf, "Error #%d:
%s.", num, msg);
MessageBox(GetFocus(), buf, NULL,
MB_OK);
/* free the buffer */
MemFreePtr(buf);
return
TRUE;
}
To avoid a related problem, running out of stack space, turn your compiler’s stack-checking probe switch on during development.
Leakage is memory that is no longer in use but that has not been freed. This memory is permanently unavailable and, therefore, wasted. Leakage is sometimes referred to as “garbage.”
Leakage is an insidious problem. It will usually not cause your program to crash or produce other blatant symptoms. What it will cause is the premature exhaustion of memory. Users who use your application for extended periods will inevitably run out of memory.
Because of its low-key nature, leakage can be very difficult to eradicate. If you are dealing with complex dynamic data structures, it takes disciplined programming to write completely “garbage-free” code. Some languages (such as Lisp, BASIC, or Actor) provide an automatic garbage collector. However, garbage collectors have their own problems, such as doubling memory requirements, and incurring a substantial performance penalty.
It is much easier to prevent leakage than it is to remove leakage already introduced. Every time you dynamically allocate memory, you should consider when it will be de-allocated and, preferably, right then and there, write the appropriate code to free the memory. Don’t be lulled into complacency by the comparatively large address space afforded by virtual memory. If your application leaks, your users will eventually run out of memory.
SmartHeap provides a number of functions that help you prevent, diagnose, and eliminate leakage. The dbgMemReportLeakage function gives you an immediate report of any unfreed blocks. You can use this function in conjunction with dbgMemSetCheckpoint to generate a leakage report for a particular group of allocations or a particular sub-process within your application. See the dbgMemReportLeakage function in §4.2, “Function reference” for an example.
You can prevent leakage by partitioning your data structures into separate memory pools, as described in §2.4.2, “Freeing related allocations.” By using MemPoolFree to free the entire contents of a memory pool, you avoid the need to free each element individually, and you avoid the possibility of forgetting to free some elements.
Another function that can be useful is MemPoolSize. This function returns the total number of bytes allocated from the system by a given pool. If your application reaches a steady state of memory use from a particular pool, such that, on average, the memory allocated is equal in size to the memory freed, but MemPoolSize reports a gradual increase in memory allocated, this is indicative of leakage.
Constipation is a corollary of leakage. Constipation is memory, sitting free in a private memory pool, that exceeds its owner’s memory requirements. Constipated memory is not permanently unavailable, wasted memory: it can be used by one party but is not available for general use.
How can this situation come about? By having a memory pool full of available memory that you never use. In fact, constipation is the one down side to having many memory pools for different data structures. For example, if you initialize a new memory pool with MemPoolInitFS, you can specify the initial number of blocks to allocate. If you specify a number that is larger than you will ever use, the memory pool will be constipated.
SmartHeap guards against constipation by returning pages with no blocks in use to the operating system (multiple pages are transparently maintained by SmartHeap for each memory pool). MemPoolSetFloor tells SmartHeap to trim such unused pages only above the threshold you specify (the default floor is zero).
There are several steps you can take to avoid constipation:
Do not specify a floor with MemPoolSetFloor that is larger than your current memory needs — otherwise if you allocate and free memory from a pool, then don’t subsequently use the freed blocks, you will be causing constipation.
When calling MemPoolInitFS, specify a value equal to your near-term needs. The pool will be expanded automatically, as required.
Do not call MemPoolPreAllocate with an excessively large size — hogging resources like this just brings overall system performance down.
Use MemPoolSetCeiling to cap the size of your memory pools. SmartHeap will then try harder to shrink your pools if they become bloated.
If you free a large number of small blocks from a memory pool, with no intention of re-allocating the memory in the near future, call MemPoolShrink to return the newly-freed memory to the operating system.
Windows 16-bit · For 16-bit Windows, handle the WM_COMPACTING message in each of your top-level window procedures. Windows sends this message when the system is spending a lot of time compacting the global heap, which is indicative of low memory. If your application receives this message, immediately shrink all of your memory pools to give back any free memory.
The error handling mechanism in Debug SmartHeap is identical to that in Runtime SmartHeap (see §2.5, “Runtime error handling”). In Debug SmartHeap, however, more errors are detected, and more information is provided to your error handler about the source and cause of the error.
In Debug SmartHeap, you can customize error handling without writing your own error handler. You can use dbgMemSetDefaultErrorOutput to send output to any combination of a user interface prompt, log file, secondary monitor, or audible beep.
You can create your own error handling routine using the function MemSetErrorHandler.. You can then control error I/O and recovery completely. Debug SmartHeap provides functions to format error output to a string to simplify error reporting. These are dbgMemFormatErrorInfo and dbgMemFormatCall.
To trace all calls to SmartHeap APIs, you can use dbgMemSetEntryHandler and dbgMemSetExitHandler. Debug SmartHeap will call back to the trace hook you establish on the entry and exit of each call to the SmartHeap library.
On entry, SmartHeap identifies the API and all parameters, and sets a boolean that indicates whether the parameters are valid. On exit, SmartHeap passes both the input parameters and the return value of the SmartHeap function.
You could have the trace hook send a log of all SmartHeap calls to a file or secondary monitor to determine the exact history of memory management calls.
The following pages show the shtestd.c sample application that illustrates Debug SmartHeap error reports. Following the listing of the test program is the output it generates. Note that the line numbers correspond with those reported by SmartHeap in the error report.
The output shown here is for the 32-bit Windows version of SmartHeap. The output will differ slightly on other platforms — for example, it won’t include thread identifiers on non-multi-threaded platforms.
/* Sample app illustrating Debug SmartHeap error detection/reporting */
#include <stdio.h>
#include <stdlib.h>
/* Note: SmartHeap header files must be included AFTER any files that
* declare malloc or operator new
*/
#ifdef __cplusplus
#define DEFINE_NEW_MACRO 1
#endif
#include "smrtheap.h"
#ifndef MEM_DEBUG
#error shtestd.c must be compiled with MEM_DEBUG defined
#endif
#define TRUE 1
#define FALSE 0
int main()
{
unsigned char *buf;
int i;
unsigned char c;
dbgMemSetSafetyLevel(MEM_SAFETY_DEBUG);
dbgMemSetDefaultErrorOutput(DBGMEM_OUTPUT_FILE|DBGMEM_OUTPUT_CONSOLE
|DBGMEM_OUTPUT_BEEP|DBGMEM_OUTPUT_PROMPT,
"shtestd.out");
dbgMemSetCheckFrequency(1);
dbgMemDeferFreeing(TRUE);
dbgMemSetCheckpoint(2);
/* the following allocation is never freed (leakage) */
#ifdef __cplusplus
buf = new unsigned char[3];
#else
buf = (unsigned char *)malloc(3);
#endif
/* invalid buffer */
dbgMemSettingsInfo(NULL);
/* invalid pointer parameter */
free((void *)ULONG_MAX);
/* underwrite */
c = buf[-1];
buf[-1] = 'x';
realloc(buf, 4);
buf[-1] = c;
/* overwrite */
buf = (unsigned char *)malloc(3); /* more leakage */
c = buf[3];
buf[5] = 'z';
realloc(buf, 1);
buf[5] = c;
dbgMemSetCheckpoint(3);
/* write into read-only block */
buf = (unsigned char *)malloc(10); /* more leakage */
*buf = 'a';
malloc(1); /* more leakage */
dbgMemProtectPtr(buf, DBGMEM_PTR_READONLY);
*buf = 'b';
buf = (unsigned char *)realloc(buf, 11);
*buf = 'a';
dbgMemProtectPtr(buf, DBGMEM_PTR_NOFREE | DBGMEM_PTR_NOREALLOC);
free(buf);
realloc(buf, 44);
/* double free */
buf = (unsigned char *)malloc(1);
for (i = 0; i < 3; i++)
free(buf);
/* write into free block */
c = *buf;
*buf = 'a';
calloc(1, 3); /* more leakage */
*buf = c;
dbgMemReportLeakage(NULL, 2, 3);
return 1;
}
Here is a listing of the shtestd.out file generated by the above program:
MEM_BAD_BUFFER: Invalid buffer parameter.
Error detected in: dbgMemSettingsInfo(00000000)
at line 44 of file shtestd.c, pass #1
checkpoint 2, alloc #33, thread 0xc2
Parameter is NULL pointer.
MEM_BAD_POINTER: Invalid memory pointer parameter.
Error detected in: free(FFFFFFFF)
at line 47 of file shtestd.c, pass #1
checkpoint 2, alloc #33, thread 0xc2
Error at or near address FFFFFFFF which contains:
<illegal address>
MEM_UNDERWRITE: Memory before beginning of allocated block overwritten.
Error detected in: realloc(00760B2C, 4)
at line 52 of file shtestd.c, pass #1
checkpoint 2, alloc #33, thread 0xc2
Object created by: malloc(3)
at line 40 of file shtestd.c, pass #1
checkpoint 2, alloc #33, thread 0xc2
Error at or near address 00760B2B which contains:
78 EB EB EB FC FC FC FC-FC FC FC FC DD DD DD DD x...............
Pool last verified in: malloc
at line 40 of file shtestd.c, pass #1
MEM_OVERWRITE: Memory after end of allocated block overwritten.
Error detected in: realloc(00760B2C, 1)
at line 59 of file shtestd.c, pass #1
checkpoint 2, alloc #35, thread 0xc2
Object created by: malloc(3)
at line 56 of file shtestd.c, pass #1
checkpoint 2, alloc #35, thread 0xc2
Error at or near address 00760B31 which contains:
7A FC FC FC FC FC 0D DD-DD DD DD 00 00 00 00 02 z...............
Pool last verified in: malloc
at line 56 of file shtestd.c, pass #1
MEM_READONLY_MODIFIED: Memory block marked as read-only was written to.
Error detected in: realloc(00760BD4, 11)
at line 70 of file shtestd.c, pass #1
checkpoint 3, alloc #38, thread 0xc2
Object created by: malloc(10)
at line 65 of file shtestd.c, pass #1
checkpoint 3, alloc #37, thread 0xc2
Error at or near address 00760BD4 which contains:
62 EB EB EB EB EB EB EB-EB EB FC FC FC FC FC FC b...............
Pool last verified in: dbgMemProtectPtr
at line 68 of file shtestd.c, pass #1
MEM_NOFREE: Attempt to free memory block marked as no-free.
Error detected in: free(00760C14)
at line 73 of file shtestd.c, pass #1
checkpoint 3, alloc #39, thread 0xc2
Object created by: realloc(00760C14, 11)
at line 70 of file shtestd.c, pass #1
checkpoint 3, alloc #39, thread 0xc2
Error at or near address 00760C14 which contains:
61 EB EB EB EB EB EB EB-EB EB EB FC FC FC FC FC a...............
MEM_NOREALLOC: Attempt to realloc memory block marked as no-realloc.
Error detected in: realloc(00760C14, 44)
at line 74 of file shtestd.c, pass #1
checkpoint 3, alloc #39, thread 0xc2
Object created by: realloc(00760C14, 11)
at line 70 of file shtestd.c, pass #1
checkpoint 3, alloc #39, thread 0xc2
Error at or near address 00760C14 which contains:
61 EB EB EB EB EB EB EB-EB EB EB FC FC FC FC FC a...............
MEM_DOUBLE_FREE: Free memory block referenced.
Error detected in: free(00760C54)
at line 79 of file shtestd.c, pass #2
checkpoint 3, alloc #40, thread 0xc2
Object created by: free(00760C54)
at line 79 of file shtestd.c, pass #1
checkpoint 3, alloc #40, thread 0xc2
Error at or near address 00760C54 which contains:
DD DD DD DD DD DD DD DD-DD DD DD DD DD DD DD DD ................
MEM_DOUBLE_FREE: Free memory block referenced.
Error detected in: free(00760C54)
at line 79 of file shtestd.c, pass #3
checkpoint 3, alloc #40, thread 0xc2
Object created by: free(00760C54)
at line 79 of file shtestd.c, pass #1
checkpoint 3, alloc #40, thread 0xc2
Error at or near address 00760C54 which contains:
DD DD DD DD DD DD DD DD-DD DD DD DD DD DD DD DD ................
MEM_FREE_BLOCK_WRITE: Free memory block was written to.
Error detected in: calloc(3, 1)
at line 84 of file shtestd.c, pass #1
checkpoint 3, alloc #40, thread 0xc2
Object created by: free(00760C54)
at line 79 of file shtestd.c, pass #1
checkpoint 3, alloc #40, thread 0xc2
Error at or near address 00760C54 which contains:
61 DD DD DD DD DD DD DD-DD DD DD DD DD DD DD DD a...............
Pool last verified in: free
at line 79 of file shtestd.c, pass #1
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: dbgMemReportLeakage(NULL, 2, 3)
at line 87 of file shtestd.c, pass #1
checkpoint 3, alloc #41, thread 0xc2
Object created by: malloc(1)
at line 67 of file shtestd.c, pass #1
checkpoint 3, alloc #38, thread 0xc2
Error at or near address 00760B2C which contains:
EB FC FC FC FC FC FC FC-FC FC FC 0D DD DD DD DD ................
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: dbgMemReportLeakage(NULL, 2, 3)
at line 87 of file shtestd.c, pass #1
checkpoint 3, alloc #41, thread 0xc2
Object created by: realloc(00760B64, 4)
at line 52 of file shtestd.c, pass #1
checkpoint 2, alloc #34, thread 0xc2
Error at or near address 00760B64 which contains:
EB EB EB EB FC FC FC FC-FC FC FC 0D 00 00 00 00 ................
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: dbgMemReportLeakage(NULL, 2, 3)
at line 87 of file shtestd.c, pass #1
checkpoint 3, alloc #41, thread 0xc2
Object created by: realloc(00760B9C, 1)
at line 59 of file shtestd.c, pass #1
checkpoint 2, alloc #36, thread 0xc2
Error at or near address 00760B9C which contains:
EB FC FC FC FC FC FC FC-FC FC FC 0F 00 00 00 00 ................
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: dbgMemReportLeakage(NULL, 2, 3)
at line 87 of file shtestd.c, pass #1
checkpoint 3, alloc #41, thread 0xc2
Object created by: realloc(00760C14, 11)
at line 70 of file shtestd.c, pass #1
checkpoint 3, alloc #39, thread 0xc2
Error at or near address 00760C14 which contains:
61 EB EB EB EB EB EB EB-EB EB EB FC FC FC FC FC a...............
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: dbgMemReportLeakage(NULL, 2, 3)
at line 87 of file shtestd.c, pass #1
checkpoint 3, alloc #41, thread 0xc2
Object created by: calloc(3, 1)
at line 84 of file shtestd.c, pass #1
checkpoint 3, alloc #41, thread 0xc2
Error at or near address 00760C8C which contains:
00 00 00 FC FC FC FC FC-FC FC FC DD DD DD DD DD ................
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: Termination
checkpoint 3, alloc #41, thread 0xc2
Object created by: malloc(1)
at line 67 of file shtestd.c, pass #1
checkpoint 3, alloc #38, thread 0xc2
Error at or near address 00760B2C which contains:
EB FC FC FC FC FC FC FC-FC FC FC 0D DD DD DD DD ................
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: Termination
checkpoint 3, alloc #41, thread 0xc2
Object created by: realloc(00760B64, 4)
at line 52 of file shtestd.c, pass #1
checkpoint 2, alloc #34, thread 0xc2
Error at or near address 00760B64 which contains:
EB EB EB EB FC FC FC FC-FC FC FC 0D 00 00 00 00 ................
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: Termination
checkpoint 3, alloc #41, thread 0xc2
Object created by: realloc(00760B9C, 1)
at line 59 of file shtestd.c, pass #1
checkpoint 2, alloc #36, thread 0xc2
Error at or near address 00760B9C which contains:
EB FC FC FC FC FC FC FC-FC FC FC 0F 00 00 00 00 ................
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: Termination
checkpoint 3, alloc #41, thread 0xc2
Object created by: realloc(00760C14, 11)
at line 70 of file shtestd.c, pass #1
checkpoint 3, alloc #39, thread 0xc2
Error at or near address 00760C14 which contains:
61 EB EB EB EB EB EB EB-EB EB EB FC FC FC FC FC a...............
MEM_LEAKAGE: Memory block has not been freed.
Error detected in: Termination
checkpoint 3, alloc #41, thread 0xc2
Object created by: calloc(3, 1)
at line 84 of file shtestd.c, pass #1
checkpoint 3, alloc #41, thread 0xc2
Error at or near address 00760C8C which contains:
00 00 00 FC FC FC FC FC-FC FC FC DD DD DD DD DD ................
* Used in conjunction with an interactive debugger, the SmartHeap pass count allows you to breakpoint at the precise point in your application when and where an object was allocated that will later be corrupted or fail to be freed.
SmartHeap
Programmer’s Guide