Here are some ways you can deal with your sons violent tendencies. The most powerful motivator is your ability to use descriptive praise. In the example you have given me of your son balling up his fist as if to hit you the fact that he didn't is something that you can praise as you describe his behaviors. Descriptive praise needs to be very specific to the behavior and happen as quickly as possible after the aggression has been averted.
The second thing that is very important is to be proactive rather than reactive. Take the time to sit down and figure out what went wrong in the past even if it was something that just happened recently. This will help your son have some alternate behaviors to choose from rather than just looking angry and taking an aggressive stance. You may ask him what the right thing should be in a situation like that when he's no longer as angry and can think clearly. This is particularly helpful in situations where you know frustration leads to this angry type of stance. Another way of looking at this technique is called the A B C's of aggression. Simply put my your son is not as angry or frustrated you ask him to look at what caused him to be angry, what behaviors he decided to choose and the consequences of those behaviors. In this situation you could address the fact that he was angry and having to go to practice when he was frustrated by not being able to understand his lessons in mathematics. The behaviors were for him to take an aggressive stance and make a fist. The consequences are that it made you feel scared that he would do something that might hurt you. The more he is able to understand the consequences of his behaviors the better he has an ability to choose behaviors that would be more appropriate. This is a good time to discuss alternate behaviors such as telling you how frustrated he was by not being able to complete his homework when he was able to do this so well in the past. Another way would be for him to say how frustrated he feels that having to go to sports practice when he knows he needs to spend more time working on homework. In this way you are allowing him to be more assertive rather than aggressive.
The third way of responding when your son is upset or annoyed is to be respectful and help him get over his upset because he knows he's being heard. In the example you gave you might reflect on your own behaviors and see if there was some other way your insistence that he goes to sports practice could be stated with the understanding that he's having frustration in doing his homework. Simply put you may say it's time to go to practice but will work on the homework together or I'll find some help for you to solve your problem. Your son then knows that you are listening to his frustration and he does not have to resort to angry impulses. This is particularly true when the anger is not specifically tied to one thing. Being a reflective listener allows you to understand why he's upset. Most people become aggressive because they feel they are not getting your point across. Likewise the quickest way to end aggressive acts on someone's part is to become an empathetic and clear listener to their concerns. This usually results in a reduction of tension and frustration which will no longer fuel the angry outburst.
It is important to keep in mind your sons body posture and help him learn the warning signs of his becoming angry and frustrated. Usually there are physiological science that people display before they become angry. Helping your son understand those warning signs will give him the opportunity of expressing himself in other ways. This is been particularly true in helping men deal with domestic violence. Studies have shown that they can be taught to be more assertive they do not rely on aggression to get their point across. In your son's case in this example you may have feelings that the frustration caused such as tension in his neck, stiffness in his arms or some other bodily features. This is particularly helpful in young boys who are just beginning to learn some of their bodily sensation.
Lastly is the issue of consequences and rewards. Unlike being punished which is a negative consequence and reduces all behaviors going through the situation that led up to his angry expression may help him see that he can get his point across without getting you scared. This helps them understand the consequences of his behaviors as well as the potential rewards of acting in a more assertive way about his feelings. If he sees that all he had to do was tell you how frustrated he was and how angry it made him feel rather than showing it to you he will see the rewards of being assertive in getting your point across.