Denmark is braced for lengthy and challenging coalition talks after neither Mette Frederiksen’s leftwing bloc nor the rightwing parties managed to obtain a majority in Tuesday’s election.

After a bruising night for her Social Democrat party, which despite remaining the biggest party in the Danish parliament had its worst general election since 1903, the prime minister went to Amalienborg palace on Wednesday morning to submit her government’s resignation to the king.

Later in the day, parties began arriving at the palace in order of size, starting with the Social Democrats, to tell the king who they think should have the role of “royal investigator”, whose task it will be to try to form a government.

Speaking in a debate involving the 12 party leaders in Copenhagen on Wednesday, Frederiksen said voters had handed leaders a “troublesome” party situation but a “government must be formed”.

She added: “The world is not waiting for us out there, and it has only become even more restless than when the election was called.”

Frederiksen said she would start exploring the possibility of forming a left-leaning government with the support of Lars Løkke Rasmussen’s centre-right Moderates.

The failure of the left-leaning “red bloc” and right-leaning “blue bloc”, which won 84 seats and 77 seats respectively, to get a majority in the 179-seat parliament left the Moderates, with 14 seats, in a potentially powerful position to play a key role in forming a new coalition, putting Rasmussen, a committed centrist, in the position of kingmaker.

In his election night speech he appealed to Frederiksen and Troels Lund Poulsen, the leader of the liberal Venstre party, with whom he has been in coalition for more than three years, to “come down from the trees” and join him in the centre ground.

“What is clear – with all conceivable reservations – I think is that there is no red majority to the left of us, and there is no black-blue majority to the right of us,” he said, to cheers.

Rasmussen, was the foreign minister in the last government and has twice been prime minister. He said before this election that he did not want to be the prime minister but would like to be appointed royal investigator – although this role is usually held by the person who goes on to lead the government.

Frederiksen addressed her party at the Social Democrats’ party at Christiansborg in the early hours of the morning, saying the results were not as good as she had hoped but were “OK”.

“We reach out for responsibility – even when it comes at a price. I am still prepared to take on the job as Denmark’s prime minister. There is just no indication that it will be easy,” she said.

Poulsen said he was still a candidate for prime minister and ruled out forming a coalition with the Social Democrats. He told supporters: “We need a new government. And that’s also why I’m happy that Venstre has become the largest blue party.”

Coalition negotiations are expected to take weeks. Among the election’s biggest winners were the Green Left, who for the first time became the second largest party in Folketinget, the Danish parliament. They are believed to have benefited from leftwing voters deserting the Social Democrats after their three years in a centrist coalition, during which time Frederiksen doubled down on her hardline stance on immigration.

At the opposite end of the spectrum, the far-right Danish People’s party (DPP) increased its support since the last election from five to 16 seats. The number is still far from the party’s peak levels of support in 2015, when it won 37 seats and 21% of the vote.

Naaja Nathanielsen, a high-profile minister from the Greenlandic party Inuit Ataqatigiit (IA), won one of the Arctic island’s two seats in the Danish parliament. The other was won by Qarsoq Høegh-Dam, of the independence party Naleraq.

The royal palace released a statement saying the king had received the prime minister, and representatives of the political parties would be asked to meet at the palace.

“After explaining the election results and the parliamentary situation, the prime minister submitted the government’s resignation and advised that representatives of the political parties that have been elected to the Folketing should now be given the opportunity to speak about the upcoming government formation,” it said.