Ah, you were close, but not quite. I live on the other side of the border, behind enemy lines in IL. Find Beloit, WI on the map and go diredctly south about 20 miles and you hit Rockford, IL. That's my metro area -- Rockford, Loves Park, Machesney Park.
Sounds like the M4 you have is nicely tricked out. If you're getting that kind of groups with the rig you have, then why bother? If you had the AR equivalent of a designated marksman's rifle (DMR) with a heavy barrel, flattop receiver, and pricey scope, then the trigger kit would make much more sense. However, you're doing tactical work here and that's the M4's main forte.
If I was going to Afghanistan or Iraq and was carrying an M4/M16 for social encounters, I would prefer an upgraded package. Those upgrades would be: (1) caliber and (2) gas operating system.
For better terminal effects on the target then consider the 6.8x43 SPC round (6.8 Remington). All that's required is a switch iof uppers and magazines. (The 6.8 magazine looks like a standard 30-round 5.56x45 NATO, but it's made of steel and holds 25 rounds.) Alternatively there's the 6.5 Grendel, but I like the 6.8 SPC better. The 115-grain, .270 caliber bullet delivers TWICE the energy to the target as a 62-grain, .224 caliber bullet. In this department, I'd also consider a gas piston operated upper receiver -- I can think of at least three offerings in this area -- where the conventional gas tube/bolt carrier arrangement is replaced by a gas piston operated bolt carrier like the FN FAL. For me, the state of the art offering in an M4/M16/AR -type rifle would be a gas piston upper and the 6.8 SPC caliber. THAT's just about perfect.