No, you will not be charged the annual membership fee or for your 4 free passes annually, as long as you maintain your Smart Diamond membership.
The credit card information is requested on registration in the event that you purchase additional passes after using your 4 free passes or if you purchase guest passes for fellow passengers.
More information on the airport lounge passes can be found on the Priority Pass website.