No, it's fine at level 6. The most important thing is Charged Blow at this stage, not the charges themselves.
WK charges, as well as TB and DW charges, are different from attacks already imbued with an element (for example mage attacks and Inferno/Blizzard). Charges will increase in both neutral damage and elemental advantage, whereas elemental attacks start off with a 1.5x multiplier for elementally weak and increase in neutral damage.
Simply put, at level 6, you will not be gaining a 150% damage increase by hitting a fire weak monster. That happens at level 20.