Modify the cronExpression property of the mailPollTrigger bean in schedulingSubsystemContext.xml file.

In Confluence installation directory, this file is jarred in <confluence installation directory>/confluence/WEB-INF/lib/confluence-x.x.jar.
Unzipping the file and placing it WEB-INF/classes folder will over ride the configuration from the jarred version.

For example, the expression below will fire at 12pm (noon) every day:

 <property name="cronExpression">
   <value>0 0 12 * * ?</value> 
 </property>


Cron expression

You may also like to view an information on how to create Cron expressions to provide the ability to specify complex time combination.
http://quartz.sourceforge.net/javadoc/org/quartz/CronTrigger.html