Same exact thing is happening to me...(server owner) friend can stay in as long as they want, but I get disconnected after 5-10mins.
EDIT: I had setup a friend who is on my server with admin on the server, removed him, so now I am the only admin, and that seemed to work. For me at least.