That's a good point and I think you are right, there is no built in 'reminder' options for approvers that I have found.
I would suggest you could have 2 ways to build something that would cover this;
The first would be to have a workflow set on time entry, that would notify the approver for each submission. This I believe would not be practical as they would probably be inundated by notifications.
The second (And I think better) option would be to build a report that can then be run and distributed regularly (Daily or weekly, depending on needs) that gives an overview of the time entries that are awaiting approval for each approver. This could be scheduled to run automatically or initiated as required using a custom action.