It depends on the unit’s role and what it’s trying to beat, speed wise. Generally, if your unit comes with high base speed, max it out. Such units usually get some speed from their weapon. If it’s low, dump speed (don’t add any). If it’s middling, usually leave it alone, but sometimes it can be good to pump for specific units and roles.
Very slow < 20
Very fast 50-60
Ultra fast 60-70
Anyone can add 6-7 speed from their A and 4-7 from their S slot, for 10-14 speed. Visible buffs from C or Rallies add 6, usually. And Lulls and Canto effects can add 3 in the B slot. All told, with only a Rally or C slot and no other supportive effects, you can add 20-24 speed, which is enough to climb 2-2.5 tiers in speed. You will need further support to move up a third tier, and four tiers is pretty hard to do without pretty much your entire AR team buffing speed.
As one example, L!Lilina has middling/slow speed to start, but because she wants visible attack, the best way to get that, Life and Death, also boosts speed. So, mine ends up with 39 speed, which is just under my Fast tier. Not fast enough to double any speedsters and enough to beat middling and slow ones. Typically, the slower ones run auto-doubles or big retaliations (Bonfire B!Hector does both), so the best option is to shut that down with Windsweep. Now she safely initiates against most units because speedsters are also usually squishy and AoE + a hit is enough.