It is a fact that all screaming children of all ages are, without exception, revealing a lack of discipline in either the child or the parent(s). There is no justifying or otherwise excusing such behavior; even babies can be quieted, or at the very least removed from a location. It's the duty of a parent(s) or guardian(s) to inconvenience themselves and their day's too-tight schedule not only for a child's sake but as a courtesy to others (if I wanted a screaming child I'd give birth to one or adopt it). Oh, and here's the point where I add that there's a difference between crying and screaming, and a lot of people who like to imagine they're good parents would do well to learn it. Not all adults are born to be parents, but it's usually a teachable quality, and it's okay to ask for help if you can't figure out what the hell you're supposed to be doing.
That said, children are humans, and like with adult humans, what amounts to public humiliation is going to be handled in different ways; some will conform to avoid further humiliation, others will resent and continue to act out in an effort to strike back. Neither is desirable, the latter for obvious reasons; the former can result in a person who doesn't take up for themselves when they should (i.e., the typical victim of an abusive relationship), because they're afraid of how people will react.
With that in mind, nonphysical methods should always be tried first, especially if the child is old enough to understand an abstract concept like good manners. Spanking should always be avoided if at all possible, particularly if it's preceded by frustration on the part of the one meting out the discipline, because the question then becomes not what the CHILD was doing wrong, but what the ADULT was doing wrong. An adult should always be in control—not merely of the child and the situation, but themselves as well.
There's no arguing that discipline is imperative, as it keeps society functioning, but an adult may be in as much need of it as a child, and it's important to find the actual reason for a lack of discipline and deal with THAT (perhaps your nephew was jealous of his cousin getting a present and there was room for you to deal with that more constructively . . . but did you ask to find out?); that is, treat the cause rather than the symptom. Discipline of any sort should never be administered by someone who's in an angry mindset; the line between discipline and abuse is very fine, and easily crossed by someone in that state of mind. It might seem like justice at the time, but after the fact a calm mind often regrets the degree of punishment that was dealt.
tl;dr version: What's best for a child should always be foremost on your mind when you have the responsibility of care. It's not okay to spank a child, in public or private, just because you want to shut them up as fast as possible or because you don't want to throw off your schedule. That's you being selfish, not you thinking of what's best for the child. It's up to you to make time for your nephew—as much time as necessary—so that you both know what to expect from each other. He's old enough to understand that there are consequences for bad behavior, and it's up to you to make sure he knows what privileges or rewards he stands to lose for any misbehavior. However, he might come from a permissive or inconsistent background, and he needs to know before you crack down that things have changed. And it's up to you to realize that a permissive or inconsistent background (if that's a problem) means he's going to need reminders and patience, because he's having to learn a new way of social interaction.