The claim developed slowly. Rome's prestige in the early centuries rested on being the church of Peter and Paul and the capital's see, and other churches consulted it; the language of jurisdiction over them is later. Eastern churches accepted a primacy of honour and place Constantinople, Alexandria, Antioch and Jerusalem alongside Rome in a pentarchy of ancient sees. What they did not accept was that Rome could decide for the others.
The Eastern Orthodox objection is therefore double. Universal jurisdiction, they hold, makes the pope a bishop over bishops, which contradicts the collegial structure they read in the early councils; and infallibility locates in one person a guarantee the East attributes to the whole church, discerned over time and recognised by reception rather than declared in advance. The 1870 definition, coming as it did after 1054 and after the failed reunion councils of Lyons and Florence, made reconciliation substantially harder and prompted a Western secession as well: the Old Catholic churches formed in refusal of it.
The Protestant objection is prior and simpler. If Scripture is the final norm, then no office can be exempt from correction by it, and the Reformers argued that the papacy had in fact taught error, on indulgences among other things. The dispute is not really about whether Peter was prominent among the apostles, which the texts plainly show, but about whether his prominence was an office transmitted to a particular see.
Oriental Orthodox churches, separated since the fifth century over christology, share the Eastern objections. Anglicanism, having rejected papal jurisdiction in the sixteenth century, has since produced ecumenical statements exploring what a universal primacy might look like if it were not defined as it was in 1870. Recent popes have themselves invited proposals on the question, without altering the definition.