Luke Wilson Says He's 'Starting to Feel Panicky' About Turning 50: 'When Did This Happen?'

Luke Wilson's 50th birthday is fast approaching and the actor's feelings about the milestone are apprehensive at best.

Wilson, who turns 50 on September 21, tells PEOPLE his previous birthdays "had really no impact on me." Reaching 50 is whole other thing for the Legally Blonde star, whose new film 12 Mighty Orphans is now playing in select theaters.

"Thirty didn't matter to me. Forty didn't matter to me," he says. "I just steamrolled right through it, but 50. I don't know if it's getting to me because I am sore when I wake up in the middle of the night, and I am forgetting the names of people I know."

He continues, "So I don't know. It's definitely one of those things has had me thinking, 'Oh, okay. So this is where the phrase midlife crisis comes from.' I'm definitely starting to feel panicky about some stuff."

Wilson says he hadn't "really talked" to his two older brothers Owen, 52, and Andrew, 56, about his birthday. He did, however, rely on friend, tennis player Pete Sampras, for some advice.

"He's going to be 50 in August. And we were just together yesterday, and we're just having a hilarious conversation about how sore we were and trying to stay in shape and what you eat and what happened? When did this happen?" Wilson says. "How did it happen?"

Never miss a story — sign up for PEOPLE's free weekly newsletter to get the biggest news of the week delivered to your inbox every Friday.

He adds, "Because Pete's in unbelievable shape and still works out a lot. It just makes me feel better just to hear him say, 'I don't know when it happened.' Because I'm the same way. I'm like, 'It was just yesterday we got to L.A..' Or, 'It seems like just yesterday we were in Texas making our first movie.'"

12 Mighty Orphans is in theaters this Friday.