Showtime/The Movie Channel agreed yesterday to pay an estimated $100 million to $150 million for a five-year distribution agreement with Carolco Pictures. The amount will depend on how much the films make in theatrical release.

The deal, which begins with this summer's release of ''Rambo III'' and ''Red Heat,'' gives Showtime/ The Movie Channel, the second-largest pay cable network after Home Box Office, a stronger position in the fiercely competitive market. It will take at least a year before either title appears on the network, since films are first released on video cassette about six months after their theatrical debut, and only later appear on pay cable.

The Carolco agreement gives Showtime the exclusive pay cable rights in the United States for all of the studio's releases through 1992, which will be at least 50 films. It also includes the satellite rights.

The deal comes as Showtime is struggling to find new customers. While Home Box Office and Cinemax, both owned by Time Inc., had 21 million subscribers at year-end, Showtime and The Movie Channel had a combined total of 8.4 million viewers, according to Paul Kagan Associates, an industry research group.

Home Box Office also lured away Paramount Pictures late last year after that studio's exclusive agreement with Showtime had ended. Terms of the negotiations have not been disclosed.

Showtime continues to have exclusive agreements with Touchstone (Disney), Orion, Cannon, Atlantic and several other studios.