Support for large memory pages was added to Java beginning in SE 5.0. For details see Java Support for Large Memory Pages. This article also mentions possible negative effects of using large page memory.
Note that in the knowledge article Deployment/Launch Checklists (Doc ID 1363505.1) in the Sun JVM section it says:
Use big memory pages if your OS supports it. Don't forget to set -XX:+UseLargePages if using a Sun JVM
To begin thinking about tuning this parameter, determine how much memory your local implementation of ATG currently consumes. You can see this by looking at the process on a running instance. This number will be:
heap size + perm size + small overhead
The small overhead is something on the order of 500MB. If you have configured heap size to vary from a min to a max (-Xms and -Xmx are different), note that it will grow over time until the max is reached.
Another tool for viewing processes and their memory usage is the top command. Top gives a summary of the memory free and used at the "top" of its output. You can run top by thread using the –H option to get more detail.
An interesting article specific to the Oracle Database (DB) and not related per se to ATG is available in the Oracle Database Administrator's Reference 11g Release 2 (11.2) for Linux and UNIX-Based Operating Systems. Specifically see G.2.2 Configuring HugePages on Red Hat Enterprise Linux and Oracle Linux. This article has several interesting points to consider when configuring for Oracle DB and is cited here as another resource to consider when configuring this parameter.