Warner Communications Inc. said yesterday that it would exercise its option to buy the 50 percent of Warner Amex Cable Communications Inc. owned by its partner in the venture, the American Express Company. It will pay $450 million and assume Warner Amex's $500 million in debt.

At the same time, another player has entered the bidding for MTV Networks Inc., which is two-thirds owned by Warner Amex and one-third owned by the public. The latest bid raises the ante by seeking not only MTV, but possibly other Warner Amex operations as well.

On Thursday, Forstmann Little & Company, an investment firm, and a group of MTV's management made a $470 million bid for MTV. Two-thirds of that, about $310 million, would go to Warner Amex.

But according to an announcement yesterday, Warner now has another, more complex option that involves Viacom International Inc.

Viacom has long wanted to expand its cable businesses and has been negotiating with Warner for some time about Warner Amex.

Option to Buy

Yesterday, Warner said that the two companies had reached an agreement giving Viacom an option to buy various parts of Warner Amex if Warner chooses to sell.

Under the terms of the agreement, Warner can force Viacom into one of two acquisition choices: acquiring the MTV interest for $310 million in cash, or buying, for $450 million, the half of Warner Amex that Warner is buying from American Express. That purchase would include one-third of the MTV stock. Unlike the Forstmann Little offer, the Viacom bid for MTV would not be extended to the public shareholders.

Warner Communications would retain the ownership of the 19 percent of Showtime/The Movie Channel Inc. that Warner Amex owns, and certain other receivables. Warner Communiations already owns 31 percent of Showtime/The Movie Channel Inc., which owns the two pay-television services of the same name.

'The Only Choice'

''Viacom wants to own either of those assets, and that is the only choice it has been given,'' said Gordon Crawford, a senior vice president at the Capital Research Group.

Warner has 90 days in which to exercise the Viacom option. Subsequently, Viacom has 30 days to make up its mind which way to go.

Forstmann Little said yesterday that it would extend its bid for MTV until Aug. 14.

Other bidders may yet appear, and the prices could rise, although until the sale of Warner Amex is concluded any purchase of MTV will be complicated by the need to get the consent of executives from both Warner and, presumably, American Express. What remained unclear, analysts said, was whether directors would rubber-stamp such a sale. Any dissenting company could hinder a transaction.

The bids come at a crucial time for Warner Communications.

The Warner Communications chairman, Steven J. Ross, has long indicated an interest in the cable business. However, Herbert J. Siegel, Warner's largest shareholder, with 29.6 percent of the stock, has generally been opposed to such an investment. Arranging a number of options, through which Warner could sell off part of the assets of Warner Amex and reduce the cost of the acquisition, apparently made the deal more attractive to Mr. Siegel.

''The key to the whole thing is that Steve Ross has bought American Express out,'' Mr. Crawford said. ''Now he can either sell MTV, or he can resell the half of Warner Amex that he just bought, at a large profit. Although he is reselling it for $450 million, he is going to pull out assets, including a 19 percent interest and some receivables, which would probably give him $50 million in profits.''

Bidding Door Left Open

Whether Mr. Ross really intends to hold onto the company, sell the entire half or sell off cable systems piecemeal, he has succeeded in leaving the door open for a bidding war.

Warner has its financing for the Warner-Amex acquisition in place. It said a consortium of banks led by First Boston had committed the funds necesary to carry out the transaction.

Forstmann Little said its offer for MTV is also firm and fully financed.

Viacom has not said how it would finance an acquisition.

Viacom is the nation's 11th-largest multiple system cable operator, with 800,000 subscribers. Warner Amex has 1.2 million subscribers. Acquiring half the company would give Viacom 1.4 million subscribers. Additionally, Viacom owns 50 percent of Showtime/The Movie Channel Inc. The company also has a minority interest in an advertiser supported service called Lifetime.

The prices for cable companies have escalated. Time Inc. and Tele-Communications Inc., which had a deal to buy Warner Amex from American Express - if that partner had bought up the entire company -twice raised their bid. The Warner acquisition, however, appears to have shut Time and Tele-Communications out of further bidding.