When the war ended, the Goldstone Ground was restored and normal football began again in 1919. However, 1919/20 was the last season of the Southern League in its prime form, and the First Division clubs were taken on by the Football League to form a Third Division (South) in 1920.
It took Albion 38 years to win the section and secure the one promotion berth available. Much of the interest between the wars was therefore reserved for the FA Cup. The club became renowned giant-killers, defeating First Division sides Oldham Athletic, Sheffield United, Everton and Chelsea in front of large crowds at the Goldstone Ground; and winning away at Grimsby Town, Portsmouth and Leicester City. A 1933 cup game against West Ham United brought 32,310 spectators to the Goldstone, a record that lasted 25 years.
Albion kept playing throughout the Second World War and never failed to fulfil a fixture, but it was a struggle to survive. The directors of the nearby greyhound track took control of the club in 1940 to stave off financial ruin, while the German Luftwaffe did its best to disrupt proceedings. Three games at the Goldstone Ground were abandoned because of air-raid warnings, and the North Stand was bombed in August 1942 (but no one was hurt).
Under the wartime regulations, Charlie Webb, who was manager from 1919 until 1947, called upon players of other clubs serving with the Army in the area, and even had to draft soldiers from the crowd on occasion to complete an eleven.
The resumption of normal football after the war saw attendances escalate to unprecedented numbers. In 1947/48, Albion finished bottom of the Football League for the only time and had to be voted back into the competition by their fellow members, but the average gate was over 11,000, a record. The following season it rose to more than 17,000.