Fulham's patchy league form carried over into the cup as they were knocked out by Leicester in a 4-3 thriller at the King Power Stadium.
Hugo Rodallega capitalised on some lax defending to give them an 18th-minute lead, but Leicester scored three in quick succession either side of half-time to turn it around -completely.
Wes Morgan levelled in the 41st minute on the rebound after Maarten Stekelenburg failed to hold Anthony Knockaert's free-kick, and Chris Wood put the Foxes ahead just before half-time when Philippe Senderos's clearance was blocked and the ball fell kindly.
Miguel Ignasi headed in a third early in the second half before Rodallega's almost instant response for Fulham set up a late rally, which looked complete when Giorgos Karagounis fired in an 86th-minute free-kick.
But no sooner were they level than Lloyd Dyer struck what would prove the winner.
Stoke needed penalties to see off the 10 men of Birmingham at St Andrews after the hosts scored twice in the last five minutes of the 90 to force extra time, and then equalised again in the 118th minute to make it 4-4.
Oussama Assaidi got the match off to a fine start with a 30-yard screamer in the 10th minute and although Tom Adeyemi levelled for Birmingham, they seemed doomed as Stoke took control after Wade Elliott was sent off on the stroke of half-time as several players clashed.
Mark Hughes' side used the extra man to set up Crouch's 55th-minute header and Arnautovic's 71st-minute strike, with Crouch striking the crossbar with an overhead effort soon after.
That near miss allowed Peter Lovenkrands to tie it with goals in the 85th and 90th minutes, and although Kenwyne Jones put Stoke back in front four minutes into extra time, Olly Lee rifled home another late equaliser.
But Birmingham's brave effort fell short as Mitch Hancox and Callum Reilly missed their first two penalties, and Stoke won the shoot-out 4-2.
West Ham edged out Championship leaders Burnley 2-0 at Turf Moor with both goals coming from the penalty spot late on.
Matthew Taylor got the first in the 76th minute after Jason Shackell brought down Kevin Nolan while Jack Collison tucked away the second in stoppage time after being fouled by Keith Treacy, who saw red to complete Burnley's misery.
Download ESPN's new UK sport app, a fresh and powerful new way to follow your favourite UK sports news, scores and video.