Athletic director Greg McGarity announced the decision in a statement released by the school. McGarity moved quickly on Fox, the ninth-year coach, only one day after the Bulldogs' exit from the Southeastern Conference Tournament.
Georgia has received only two NCAA bids in Fox's tenure, losing in the first round in 2011 and 2015.
"In the end, I felt like we have not reached our full potential as a basketball program," McGarity said. "I really thought we were on the way to turning the corner this year. We just did not achieve the level of success as a program that I believe we should at the University of Georgia."
Georgia was 39-29 in 2015-16 and 2016-17, making back-to-back NIT appearances.
McGarity said Georgia will "move as quickly as possible" in its national search for Fox's replacement.
Intermediaries on behalf of the Bulldogs began reaching out to gauge the interest of potential replacements over the past several days, sources told ESPN's Jeff Goodman and Jeff Borzello. Todd Turner and Collegiate Sports Associates are expected to be retained to help with the search.
Fox was 163-133 at Georgia.
The Bulldogs (18-15) likely needed to win the SEC tournament to land an invitation to the NCAA tournament and possibly save Fox's job.
"For the past nine years, we have had the good fortune to coach some terrific young men," Fox said. "This chapter has closed, and I am grateful to the many who have helped us along the way but especially to the young men who wore the red and black. We've been able to reach a couple of NCAA tournaments, a couple more NITs and graduate our players. Hopefully, our time here can be the groundwork for more success."
McGarity said the decision was "very difficult, especially when you are talking about someone like Mark Fox."
"We have worked alongside each other for eight seasons, and I have the utmost respect for Mark, his staff, our student-athletes and the support staff," McGarity said. "They have always represented the University of Georgia in a professional manner."
There was no immediate indication of who would coach the team if Georgia receives a bid to the NIT. Assistant coach Jonas Hayes, who may be a candidate to replace Fox, could serve as interim coach.
This season, a poor conference record hurt the Bulldogs' NCAA hopes. Georgia finished 7-11 in SEC regular-season games, tied for 11th in the 14-team league.
Expectations grew that Georgia could return to the tournament when forward Yante Maten returned for his senior season. Maten was selected SEC Player of the Year, but his consistent production was not enough to save Georgia's season as the Bulldogs were hurt by problems in their backcourt.
Georgia beat Vanderbilt and Missouri in the first two rounds of the SEC tournament before falling to Kentucky.
Fox, 49, previously coached at Nevada, where he led the Wolf Pack to three NCAA tournament appearances in five seasons.
Information from The Associated Press was used in this report.