[Raymond Chen] wondered why the x86 ENTER instruction had a strange second parameter that seems to always be set to zero. If you’ve ever wondered, [Raymond] explains what he learned in a recent blog post.

If you’ve ever taken apart the output a C compiler or written assembly programs,  you probably know that ENTER is supposed to set up a new stack frame. Presumably, you are a subroutine and some arguments were pushed on the stack for you. The instruction puts the pointer to those arguments in EBP and then adjusts the stack pointer to account for your local variables. That local variable size is the first argument to ENTER.

The reason you rarely see it set to a non-zero value is that the final argument is made for other languages that are not as frequently seen these days. In a simple way of thinking, C functions live at a global scope. Sure, there are namespaces and methods for classes and instances. But you don’t normally have a C compiler that allows a function to define another function, right?

Turns out, gcc does support this as an extension (but not g++). However, looking at the output code shows it doesn’t use this feature, but it could. The idea is that a nested function can “see” any local variables that belong to the enclosing function. This works, for example, if you allow gcc to use its extensions:

[CODE language=C]
#include

void test()
{
int a=10;
/* nested function */
void testloop(int n)
{
while (n–) printf(“%d\n”,a);
}
testloop(3);
printf(“Again\n”);
testloop(2);
printf(“and now\n”);
a=33;
testloop(5);
}

void main(int argc, char *argv[])
{
test();
}
[/CODE]

You can see that the testloop function has access to its argument, a local variable, and also a local variable that belongs to the test function. We aren’t saying this is a good idea, but it is possible, and it is common in certain other languages like Pascal, for example.

In some cases, this situation is handled by providing a linked list of stack frames. However, the Intel designers decided to do it differently. When you provide a non-zero second argument to ENTER, it copies an array of stack pointers into your local variable space. This makes your code potentially more efficient as it executes but exacts a penalty on function calls for nested functions.

As [Raymond] points out, though, it may be that no one uses this feature. Certainly, gcc doesn’t. If you want to make sure, try these commands with the above program in nest.c to check out 32-bit x86:

gcc -m32 -g -o nest nest.c   gcc -m32 -s -c nest.c# now look at nest.s and/or disassemble nest using gdb

Of course, if you write your own assembly, you could use the feature as you see fit. The x86 has some crazy instructions. If you’ve ever wondered if you should learn assembly language, our commenters would like a word with you.