BRUSSELS— Alphabet Inc.’s GOOG -0.88% Google started its appeal Monday to overturn a $5 billion antitrust fine imposed by the European Union, contending that its Android operating system for mobile devices has boosted competition rather than foreclosing it.

The tech giant presented oral arguments in Luxembourg before the EU’s second-highest court, in its appeal to overturn the 2018 decision from the bloc’s antitrust enforcer. In that case, EU authorities found Google had illegally abused the market power of Android to push companies that manufacture and distribute Android phones into agreements aimed at entrenching and expanding the dominance of the Google search engine on mobile devices.

That decision was the largest of three antitrust fines totaling more than $9 billion that the EU has levied against Google in the past half decade. It also ordered changes to the distribution agreements buttressing one of the company’s biggest growth engines: search ads on mobile phones.

“Android has created more choice for everyone, not less,” a Google spokeswoman said. “This case isn’t supported by the facts or the law.”

A verdict isn’t expected for months, and it can be appealed to the EU’s top court, the Court of Justice. Still, the litigation is a new test for the EU’s competition and digital-policy chief, Margrethe Vestager, who already faces a pending appeal of an earlier decision against Google’s alleged abuse of the dominance of its search engine to favor its online-shopping ads.

Ms. Vestager has since opened a new antitrust probe into Google’s ad-tech business, along with a wave of cases exploring whether companies including Facebook Inc., Apple Inc. and Amazon.com Inc. abuse their dominance to drive out smaller rivals. The companies deny wrongdoing.

A spokeswoman for the European Commission, the bloc’s top antitrust regulator, declined to comment.

While the appeal is under way, Google has had to comply with the decision, offering all users of new Android phones in the EU a choice screen of alternative search engines. But so far Google’s market share for search on mobile phones has remained relatively stable, according to Statcounter.

At issue in the hearing this week is whether Google’s Android is indeed dominant, as European regulators argue, and whether Google’s distribution deals for the operating system and its Google Play app store for Android were anticompetitive.

The Commission has argued that Google used those agreements to block the rise of potential competitors and secure the dominance of its cash cow search engine on mobile phones—an outcome far from assured at the time.

The Commission found Google had abused its control of the Android operating system by forcing phone makers to pre-install Google Search and Google’s Chrome browser if they also wanted to include Google’s Play store for apps, by far the most common way to get Android apps.

The Commission also found Google’s so-called antifragmentation agreements—deals that discourage official Android manufacturers from selling devices that run unofficial, modified versions of Android—illegally blocked the development and emergence of competing operating systems.

“Instead of an antifragmentation agreement, it should be called an anticompetition agreement,” said Thomas Vinje, a lawyer representing FairSearch, a group representing Oracle Inc. and other companies whose 2013 complaint led the Commission to open a formal case investigating Android in 2015.

Apple and Google have one of Silicon Valley’s most famous rivalries, but behind the scenes they maintain a deal worth $8 billion to $12 billion a year according to a U.S. Department of Justice lawsuit. Here’s how they came to depend on each other. Photo illustration: Jaden Urbi

Google argues that those analyses are flawed. It says Android devices must compete with Apple’s iPhone and iPad, and the Commission was wrong to largely exclude them from its analysis. The company argues that its antifragmentation agreements are necessary to keep Android phones compatible with apps, and aren’t a barrier to creating competing operating systems.

Google also says the allegation that it blocked competing apps is false because manufacturers typically install many rival apps on Android devices, and consumers can easily download others. The company says it has a right to recoup the money it spends developing Android, which it makes available free to manufacturers, by encouraging them to install Google Search, from which the company makes the bulk of its revenue.

Google and the Commission will be joined in this week’s arguments by nearly a dozen outside companies and trade groups that have filed their own supporting briefs in the case. Google’s supporters include the Computer & Communications Industry Association and two handset manufacturers.

Arguments on the Commission’s side will include some from German publisher groups and complainants in the case, including FairSearch.

By: Sam Schechner at sam.schechner@wsj.com and Daniel Michaels at daniel.michaels@wsj.com