Documentation for JIRA 4.4. Documentation for other versions of JIRA is available too.
Java applications like JIRA and Confluence run in a "Java virtual machine" (JVM), instead of directly within an operating system. When started, the Java virtual machine is allocated a certain amount of memory, which it makes available to applications like JIRA. By default, Java virtual machines are allocated 64Mb of memory, no matter how many gigabytes of memory your server may actually have available. 64Mb is inadequate for medium to large JIRA installations, and so this needs to be increased. Seeing OutOfMemoryErrors in the logs is symptomatic of this.
This page addresses how to increase Heap Space memory. Confirm that you're not receiving Perm Gen or GC Overhead errors.