Perez, who returned as Real Madrid president in 2009, was speaking after leaving the role for the first time, a decision he said in the leaked audio was partly down to Raul.
- Don't have ESPN? Get instant access
Casillas and Raul both left the club during Perez's second tenure but have since returned following their retirement as players. Casillas works for the Real Madrid Foundation and Raul is the coach of the B team, Castilla.
"He is not good enough to be the goalkeeper for Real Madrid, what do you want me to say?" Perez says of Casillas in the audio, which first appeared in El Confidencial. "He is not. And he never has been.
"It is the big mistake we made. What has happened is that [broken audio] they adore him, they want to speak with him. ... I don't know. They defend him so much. But, well, he is one of the big frauds of Madrid. The second is Raul. The two biggest frauds at Madrid are first Raul and then Casillas.
"[Zinedine] Zidane's a great guy. And [David] Beckham and Ronaldo. ... But then there are the others that think that ... let's see. Raul is a bad person.
"He thinks Madrid is his and he uses everything there is in Madrid, developing everything for his own benefit. Him and his agent. They are to blame for Madrid not doing well.
"Look, I left, among other reasons, because of him. Because he, now he considers his time ending, says: 'Before I go, I am going to end Madrid.' He is a negative guy, he is destroying Madrid and the morale of the players so that people say: 'It's because Madrid are bad, not Raul.' It's terrible how bad this guy is."
Later on Tuesday, Madrid released a statement on behalf of Perez in which he claimed the audio was leaked because of his involvement in promoting the recently failed European Super League project.
The statement read: "Given the story in El Confidencial, in which comments are attributed to me, I think it's necessary to clarify:
"The comments were made in conversation with D. Jose Antonio [Abellan], who has been unsuccessfully trying to sell them for years. It's surprising that given the time that has passed, they have appeared today in the newspaper El Confidencial.
"The comments are taken out of the context in which they were made. That they are reproduced now, after so many years, I believe is due to my role as one of the promoters of the European Super League.
"I have placed the issue in the hands of my lawyers who are studying possible legal action."
Raul left Real Madrid in 2010 after 18 years at the club, going on to play for Schalke, Al Sadd and the New York Cosmos before returning in a coaching capacity in 2018. He took over the club's reserve team the following year.
With 323 goals, he was Madrid's top scorer of all time when left, although he has since been overtaken by Cristiano Ronaldo, who netted 450 goals in 11 years at the Bernabeu.
However, Raul remains the club's record appearance holder, with 741 for the first team.
Casillas is second to Raul, with 725 appearances. The former goalkeeper joined the club's academy in 1990 and left 25 years later, joining Portuguese side Porto in 2015 before retiring in 2020 following heart problems.