Typically you will eventually unlock / run 2 specialty gyms. Whatever stat will be your highest will be a gym that just trains that and needs to be kept 25% above your second highest stat. Whatever is your 2nd and 3rd highest stats will your other special gym where those two stats together need to be kept higher than your other two stats.
This break down works well if you have a dump stat.
Ex: If you got the dex and def gym and then later you picked up the str gym for str to be your highest stat and def and dex to be your 2nd and 3rd stats.
Note: it is typical to have either dex or def be your dump stat as str and spd are both important in landing hits that actually do dmg.
In answer to your question: the only downside is paying for it and potentially getting locked out of it or not needing it as you grow.