Amadou Diallo and Danny Cashman struck in each half to leave the score 1-1 after 90 minutes, but the Seagulls progressed through to round four after claiming victory from the spot.
West Ham took the lead with just six minutes played through Diallo, after Jeffrey Ngakia saw his initial effort saved and the former clinically fired home on the rebound.
It was almost 2-0 a couple of minutes later after Diallo turned provider for Emmanuel Longelo, who slid his attempt over the bar from deep inside the box.
The Hammers continued their lively start as Longelo showed some neat footwork to work his way into a shooting position, but his speculative attempt was saved by Roco Rees.
Diallo had another chance to get his and West Ham’s second after he raced to the byline down the left-hand side, but his subsequent strike went straight across the face of goal.
Albion created their best opening of the half when Matt Everitt made a driving run down the left and picked out Cashman, who elected to tee up Ryan Longman in the box, but the latter’s strike hit the post.
Diallo continued to be involved in the thick of the action, and his fierce strike was kept out by Rees once again.
Haydon Roberts was then called into action to block Sean Adarkwa’s shot, as the pacey West Ham attack continued to cause Albion problems.
Cashman broke into the box in a bid to restore parity to the scoreline, but his shot was blocked and the hosts cleared the danger from the resulting corner.
The Seagulls started the second half brightly, as Roberts strode forward from defence, ran into space and saw his shot rise over the target.
Mark Beard’s and Mikey Harris’s side were soon rewarded, as Cashman levelled the scores with 55 minutes played, after he cut in off the left flank and saw his close-range strike beat Serine Sanneh at his near post.
Cashman could have turned the game on its head moments later with another close-range strike, but this time his effort went wide.
Albion continued to apply the pressure and looked more likely to get a potentially crucial second goal, after Roberts saw his header go narrowly over the bar from Peter Gwargis’s teasing corner.
It was the Hammers that were next to come close though, after Longelo’s cross from the right found Bernardo Rosa deep in space, but the forward fired over from deep inside the area.
Albion came close to winning the contest late on in a spell of pressure, as the Hammers defence needed to be alert to clear Kasper Lopata’s header, before Longman fired wide with a long-range strike.
The game headed into extra time and the Seagulls pressed forward quickly through substitute Jack Spong, who fired wide with a looping effort from distance.
Gwargis then curled a perfectly-weighted cross to find Roberts unmarked inside the box, but the centre-half’s header hit the post.
Marc Leonard found Cashman deep inside the area moments later, only for the Seagulls number nine to fire inches wide of the far post.
The Seagulls thought they had a chance to win the game in extra time when Cashman was brought down inside the area by Joshua Okotcha, but Jenks saw his penalty saved.
Late chances fell to Longman and Adarkwa for either side, but the teams couldn’t be separated and the tie went to penalties.
Gwargis, Cashman, Leonard, Roberts and Jenks were all successful from the spot before Adarkwa missed the decisive kick, and Albion progressed.
ALBION: Rees; Tanimowo, Lopata, Roberts, Packham; Jenks, Carre (Spong 90+1); Longman, Gwargis, Everitt (Leonard 60); Cashman.
SUBS NOT USED: Bentley (GK), Clark-Eden, Tolaj.