2 ways come to mind (have used both w decent success).
1- assuming all your customers are using a dns server you control (ie the mt dns), you can use regex static entries, and normal static entries (and use a rule to redirect all udp 53 dns to your mt dns, to keep ppl from setting their own customer dns server ip).
Or regex on layer 7 to drop any torrent dns queries (regardless of which dns server your clients are using). this works well, or as a good point to add a client to a 24h address list (then slow their speeds or limit only their connection count). clients using encrypted dns or a dns server through a vpn, will ofcorse be immune to this.
see here: viewtopic.php?t=59234
2- limit connection count per minute per IP, ie usually when one runs a torrent download it will use alot of connections, and/or alot of connections to many different IPs (either limit connections per minute per IP, or use this to add their IP to a address list, then filter that address list) - clients using a vpn for all their traffic, will be immnue to this (as it looks like only 1 or a few connections to a single IP, from your standpoint). only option then is to queue the clients local IP (for either just the vpn traffic/vpn serverIP , or just specific traffic, or for all their traffic)
what ever you do, you need to test it and monitor that it does not affect your legit customers', legit traffic. (ie you wont find a copy-paste that works 100% of the time and has 0 side effects, but you can get pretty close to that if you take the time). i know i didnt post rules / export in here, bc most of mine have been customized for each location, so just serach the forums (or better use google with: