The United States formally ended 38 years of association with the United Nations Educational, Scientific and Cultural Organization (UNESCO) yesterday, announcing that it will withdraw at the end of the month because the agency has been politicized leftward and is financially irresponsible.

However, Gregory J. Newell, assistant secretary of state for international organization affairs, said U.S. membership could be renewed if the 116-nation institution makes certain changes in its operations.

In the meantime, he said, an "observer panel" will be set up at the State Department to monitor UNESCO's performance, and the United States will assume the formal status of an "observer" at UNESCO headquarters in Paris.

The U.S. withdrawal was promised a year ago and has been debated around the world ever since. Critics have charged that it was motivated more by the Reagan administration's conservative ideology than by UNESCO management problems, which all sides acknowledge are serious. Newell noted that UNESCO had made improvements over the past year but said they had not gone far enough.

"An unacceptable gap clearly remains," Newell said. "When UNESCO returns to its original purposes and principles, the United States would be in a position to return to UNESCO."

Conservatives encouraged the withdrawal as a step toward possibly dropping out of the United Nations, which they see as little more than a forum for anti-American speeches.

Asked if the UNESCO decision signaled such a plan, Newell said it "should certainly be a warning to other organizations that this administration will not tolerate the problems that caused us to remove ourselves from UNESCO." But he added that there are no current plans to withdraw from any other international body, including the Food and Agriculture Organization, which has been targeted by federal budget-cutters.

The move will deprive UNESCO of $47 million in U.S. funds for fiscal 1985, or about 25 percent of its $200 million annual budget. Newell said that he would propose continued U.S. spending at that level for other programs to accomplish UNESCO's goals but that budget requests were under review.

"Our chief goal is not UNESCO's good health or UNESCO reform; it is international development," Newell said. He added that he would propose diverting part of the $47 million to the United Nations Development Program, the Agency for International Development and the U.S. Information Agency.

Other sources close to the debate said it was questionable whether the administration would continue to spend $47 million on international science, cultural and education programs because the sum would be "like a piece of cheese in front of a mouse" at the Office of Management and Budget, which is looking for ways to slash federal spending.

UNESCO officials in Paris said they will seek voluntary contributions and make budget cuts to cover the loss of U.S. funds, and may call a special governing board session in January.

UNESCO spokesman Doudou Diene expressed "regret" at the U.S. decision, saying he did not see how it would help efforts to reform UNESCO. He said the United States "did not help in discussing reforms because it did not submit any list of proposals until last July" and even then "did not indicate that these reforms were a condition for remaining in UNESCO."

Newell said the United States had been "abundantly specific," listing its requests in March, April, May and finally to the board in July.

"What would have kept us in was some kind of informal commitment by the executive board to make our requests to the general conference next November . We didn't get it," said a State Department official close to the debate. "We were exceedingly explicit."

Diene disagreed. "What is very sad is that nobody expressed that," he said in a telephone interview. "The point remains, what exactly does the U.S. want?" He said if the answer is political, "it will be better to be there when the executive board and the conference meet next year."

In a recent speech, Newell called UNESCO's financial management "atrocious," saying 80 percent of its budget is spent at the Paris headquarters. Its benefits, he said, "can be had at a far lower price than that of the obloquy we endure in Paris."

Chartered in 1946 "to contribute to peace and security by promoting collaboration among the nations through education, science and culture," UNESCO operates 182 programs around the world, spending 40 percent of its budget on education.

It has organized large-scale literacy efforts in developing countries and set up international scientific research teams; it gives technical help to developing nations and has organized the preservation of major natural, historic and architectural sites worldwide.

Newell said U.S. participation in the Universal Copyright Convention, the Oceanographic Commission and some educational exchanges will not be affected and that an arrangement will be sought to allow continued work in UNESCO's widely praised "Man in the Biosphere" program of environmental research.

But Newell has charged that UNESCO promotes "Soviet-inspired" world disarmament in some of its education programs, boosting the needs of states over the rights of individuals and demanding a "new international economic order" critical of free-market capitalism. Poor, Third World nations have used UNESCO forums to vote sanctions against Israel, praise revolutionary organizations and to denounce and routinely outvote the United States.

The administration charged last December that UNESCO had "extraneously politicized virtually every subject it deals with; exhibits hostility toward the basic institutions of a free society, especially a free market and a free press; and demonstrated unrestrained budgetary expansion."

In a July 13 letter to UNESCO Director-General Amadou-Mahtar M'Bow of Senegal, Newell asked that the organization's major financial sources be given control of the budget, that certain "politicized" programs be abandoned and that the organization adopt a simplified, zero-growth budget. No one was able to keep track of everything UNESCO was doing, Newell wrote, and "the result is a combination of annoyance, frustration, indifference, helplessness and pretense."

Although many nations, including much of Europe, urged the United States to reconsider its position, others joined in making similar complaints. Many concerns focused on M'Bow, a debonair, 63-year-old diplomat whose elegant life style and frequent world travel led to charges that he had overspent and mismanaged UNESCO's budget for personal gain during his 10 years in office. He denied the charges and refused to resign.

Newell refused to discuss M'Bow yesterday. "Our complaints involve programs and policies. We don't deal in personalities," he said.

Prodded by the United States, Britain also has announced that it would withdraw at the end of 1985 if reforms are not forthcoming.

UNESCO has always been controversial. Congress suspended the U.S. contribution in 1974 after the organization recognized the Palestine Liberation Organization and condemned Israel. It was later restored, but UNESCO's reaction to its critics has been uneven.

UNESCO defenders say it did what it could this year, setting up problem-solving groups and cooperating with a management review by the General Accounting Office. It hired a Washington public relations firm with ties to the Reagan administration, Wagner & Baroody, at $15,000 a month to plead its case.

But only one executive board session was scheduled after July, and some U.S. demands required the approval of the biennial UNESCO General Conference next November. The board meeting recommended a two-year freeze on the UNESCO budget and put several dubious programs on a review list.

The industrialized bloc of U.S. allies failed to support some of the U.S. proposals, including those to change voting procedures. The 10 European Economic Community nations wrote President Reagan earlier this month, asking him to postpone U.S. withdrawal for a year.

The kind of rhetoric that U.S. diplomats had found offensive has been notably absent this year from UNESCO documents. The rhetoric had included routine denunciations of Israel and references to the "New International Economic Order," a phrase that conservatives regard as a thinly veiled call for world revolution.

UNESCO also withheld funding for a privately organized conference on "protection of journalists" that critics had called an effort to control the media, and dropped a proposal to license reporters. But Newell called these moves "a tactical retreat," not a policy change.