Etydhv

The Sanctuary spell reads:

[...] any creature who targets the warded creature with an attack or a harmful spell must first make a Wisdom saving throw. On a failed save, the creature must choose a new target or lose the attack or spell.

Imagine a case where two orcs (allied to each other) are standing next to each other and next to me (their enemy). One tries to attack me and fails the save. Does he have to attack his Orc friend, or can he choose not to attack anyone?

The spell is ambiguous. You can either read

the creature must choose a new target or lose the attack.

So, you can choose either of these options. Or it can read

the creature must choose a new target or lose the attack.

So the creature needs to choose a new target, and if unable to, it loses the attack.

He can choose not to attack

Either way you put it, there is always the option of "losing the attack", which is ultimately the same as choosing not to attack (if it wasn't an option, it wouldn't be stated).

Repeating your quotes but using brackets to separate the options:

the creature (must choose a new target) or (lose the attack)

the creature must (choose a new target) or (lose the attack)

Almost (but not entirely) certainly yes, he can choose not to attack.

First, I assume your real question isn't "can they choose not to attack", it's "can they be forced to attack a target they don't want to attack".

I think your first interpretation is pretty clearly the intended one, but we're not here to discuss RAI... If your second interpretation were intended, the text of the spell should say something like "if able" instead of "or lose the attack". But for the sake of argument, assume your second interpretation holds. Sanctuary covers targeting the warded creature with "an attack or harmful spell", and those go by different rules.

In the case of an attack, the Making an Attack section of the rules specifies that the first step is

1. Choose a target. Pick a target within your attack's range: a creature, an object, or a location.

So if your example orc (me) tries to attack a warded creature (you), I fail the save, and you tell me I have to choose a new target (like my friend standing next to me), I can just say no, I guess I'm going to target the empty air beside you, or the ground in front of my feet, or anything else within the reach of that attack other than you and my friend, not necessarily a creature. That's a pretty iron-clad way to "not attack" even if an effect is forcing me to attack anyway.

In the case of a spell, the targeting rules are a little more complicated, since each individual spell is going to determine whether its target is a creature, object, or location. Of course, Sanctuary only applies when the warded creature is specifically chosen as a target. Accordingly, the rules here are less helpful, but many harmful spells do actually require you to select "a creature" as your target, unlike the Attack action. This does seem to leave the door open for the DM to rule according to your second interpretation and try to force me to, say, target my orc friend with a Chaos Bolt I intended for you, if they are the only other creature within range of the spell.

However, there are also rules for handling spells cast with invalid targets. That section in Xanathar's Guide to Everything gives the example of casting Charm Person (which targets "a humanoid you can see within range") on a creature that is not actually a humanoid, and the ruling is just that the effects of the spell don't happen. So back in our orc example, if I cast Chaos Bolt at you, fail the Wisdom save, and the DM tells me I need to pick a new target, I glance nervously at my orc friend, verify that the three of us are the only valid targets (i.e. the only creatures in range), and announce that I'm targeting that rock way over there. I mark off the spell slot as expended, the rock does not take any damage (since it's not a valid target), and that's that.