Large page usage is primarily intended to provide performance improvements to large memory high performance computing applications by bringing larger blocks of memory into cache when memory is accessed. The benefits of large pages are mainly due to better hardware prefetch and fewer Translation Lookaside Buffer (TLB) misses.

• Not all jobs will benefit from large pages.
• Performance will vary on our multi-user sytems due to memory contention with other jobs and the structure of the large page system. Currently 16 GB of memory is set aside for large pages. When this 16 GB of memory is used by large pages, the application requesting large pages will receive smaller pages. If you want to run benchmarks with large pages, please contact help@msi.umn.edu. The technical support staff will work with you to run your benchmarks.
• Do not use large pages for shell scripts.
• Codes with many fork()/exec() slow down when used with large pages.
• Gaussian Jobs do not benefit from large page usage. A number of performance tests have shown worse performance when using large pages.
• Large pages support is available on Power4 nodes as follows:

  Node Type	Large page	Memory for large pages	Total Memory	Memory Available*
  1.3 GHz 32-processor Regatta	yes	16 GB	64GB	47.5 GB
  1.3 GHz 24-processor Regatta	No	-	24GB	23.5 GB
  1.7 GHz 32-processor Regatta	No	-	64 GB	63.5 GB
  1.7 GHz 32-processor Regatta	No	-	128 GB	127.5 GB
  1.5 GHz p655	No	-	16 GB	15.5 GB

  * This includes memory for large pages. If you are not using large pages, subtract the amount of memory reserved for large pages from this number.

• You need to ask to be enabled to use Large Pages by sending an email to help@msi.umn.edu. Unauthorized applications are prevented from using large pages. This request is done only the first time, you want to use large pages.

• Running a Large page job:

  There are several ways to run executables using large pages. We recommend the following:

  • After compiling, mark the executable for large pages:
       /usr/ccs/bin/ldedit -b lpdata ./a.out
    The executable remains marked for large pages, until you unmark the large pages.
  • Run the job:
    • Serial Jobs:
        ./a.out
    • Parallel Jobs:
        poe ./a.out
  • You can unmark the executable:
       /usr/ccs/bin/ldedit -b nolpdata ./a.out

• Submitting large page jobs in the queue using Loadleveler

  Below are examples of Loadleveler scripts for running an executable. Note this script marks the executable for large pages. If it is already marked, you can delete the ldedit command. You may want to umark the executable for large pages after you run the executable.

  • Serial Job

    #!/bin/csh
    #@ initialdir = /homes/r30/runesha/BENCHMARK/BENCH-SEPTEMBER-2002/TMJ/LargePages
    #@ input      = test1.in
    #@ output     = test1.out
    #@ error      = test1.err
    #@ wall_clock_limit = 10:00:00 
    #@ resources = ConsumableMemory(500) 
    #@ requirements = (Feature == "LargePages")
    #@ queue
    /usr/ccs/bin/ldedit -b lpdata ./a.out
    time ./a.out
    

  • Parallel Job

    #!/bin/csh
    #@ initialdir   = /homes/r30/runesha/BENCHMARK/BENCH-SEPTEMBER-2002/TMJ/LargePages
    #@ output       = mpi_$(jobid).out
    #@ error        = mpi_$(jobid).err
    #@ job_type     = parallel
    #@ wall_clock_limit = 1:00:00
    #@ requirements = (Feature == "LargePages")
    #@ resources    = ConsumableMemory(12000)
    #@ node         = 1
    #@ tasks_per_node = 2
    #@ node_usage   = shared
    #@ queue
    /usr/ccs/bin/ldedit -b lpdata ./main.x
    time poe ./main.x