Money manager Kenneth Starr downplayed the amount he allegedly stole from his celebrity clients after federal agents found him hiding in a closet in his Upper East Side condo, authorities said yesterday.

“At the time he was handcuffed, the agent made some comment about, ‘This is your time to get things straight. You’re at the crossroads. You stole $30 million,’ ” prosecutor William Harrington said in Manhattan federal court. “The defendant responded, saying, ‘Not that much.’ ”

Harrington said it was unclear if Starr was read his rights before making the confession.

Meanwhile, Judge Shira Scheindlin gave Starr two weeks to try to convince the SEC to unfreeze some of his assets so he can replace his public defender with a private lawyer.