For MuseScore 3, the Style dialog has moved to the Format menu. Also, in current versions of MuseScore, custom signatures now work more completely, correctly determining the pitch of notes.

Part 1 - Customization

Part 2 - Creating Custom Key Signatures