Exhibit 99.4

LETTER TO REGISTERED HOLDERS OF COMMON STOCK

PACIFIC CAPITAL BANCORP

Up to 726,975,565 Shares of Common Stock To Be Issued Upon Exercise of Subscription Rights

[—], 2010

Dear Shareholder:

This letter is being distributed by Pacific Capital Bancorp (“us”, “we”, “our” or the “Company”) in connection with the offering (the “Rights Offering”) by the Company of shares of our common stock (“Common Stock”), which will be issued upon the exercise of non-transferable subscription rights (the “Subscription Rights”), which are being distributed, at no cost, to all holders of record of Common Stock as of 4:01 p.m., New York City time, on August 30, 2010 (the “Record Date”). The Subscription Rights and the Rights Offering are described in the prospectus dated [—], 2010, which is enclosed with this letter (the “Prospectus”).

In the Rights Offering, we are offering up to an aggregate of 726,975,565 shares of Common Stock to be issued upon the exercise of the Subscription Rights, which is described further in the Prospectus. The Subscription Rights will expire, if not exercised earlier, at 5:00 p.m., New York City time, on [— ], 2010, unless we elect in our sole discretion to extend the period of the Rights Offering beyond this date (as such date may be extended, the “Expiration Date”) or cancel the Rights Offering earlier.

As described in the Prospectus, you will receive, at no charge, 15.335 Subscription Rights for each share of Common Stock you owned on the Record Date. Subscription rights may only be exercised in whole numbers; we will not issue fractional rights and will round all of the Subscription Rights down to the nearest whole number for each holder of record. Each whole Subscription Right will allow you to subscribe to purchase one share of Common Stock at a subscription price of $0.20 per share. For example, if you owned 1,000 shares of Common Stock on the Record Date, you would receive 15,335 Subscription Rights and would have the right to purchase 15,335 shares of Common Stock for $0.20 per share. However, if you owned 500 shares of Common Stock on the Record Date, you would receive 7,667 Subscription Rights and would have the right to purchase 7,667 shares of Common Stock for $0.20 per share.

You should be aware that there is no over-subscription right associated with the Rights Offering. In addition, no shareholder, including SB Acquisition Company LLC, a wholly owned subsidiary of Ford Financial Fund, L.P. (the “Investor”), will backstop the Rights Offering. Neither you nor any shareholder, including the Investor, will have the opportunity to purchase additional shares not purchased by other shareholders in the Rights Offering.

You will be required to submit payment in full for all of the shares of Common Stock you wish to buy pursuant to the exercise of your Subscription Rights to BNY Mellon Shareowner Services (the “Subscription Agent”), by no later than 5:00 p.m., New York City time, on the Expiration Date. Any excess subscription payments that you may pay to the Subscription Agent in the Rights Offering will be returned, without interest or penalty, to you by the Subscription Agent as soon as practicable following the completion of the Rights Offering.

Your Subscription Rights are evidenced by a subscription rights certificate and election form (the “Subscription Rights Certificate and Election Form”). Your Subscription Rights are non-transferable, meaning that you may not sell, transfer or assign your Subscription Rights Statement to anyone else.

Enclosed for your additional information are copies of the following documents:

The first three documents listed above provide additional information on the Rights Offering, the Company and the steps you must take if you wish to exercise all or some of your Subscription Rights. You should read all of these documents carefully in their entirety.

Your prompt action is requested. To exercise your Subscription Rights, you must deliver your properly completed and signed Subscription Rights Certificate and Election Form (or the Notice of Guaranteed Delivery if you are following the guaranteed delivery procedures), together with your payment in full of the total subscription amount that is required for all of the shares that you intend to purchase in the Rights Offering to the Subscription Agent as described further in the Prospectus. Your properly completed and signed Subscription Rights Certificate and Election Form or Notice of Guaranteed Delivery, in either case accompanied by full payment of your total subscription amount, must be received by the Subscription Agent, and your payment must clear, by no later than 5:00 p.m., New York City time, on the Expiration Date. Once you have exercised your Subscription Rights, you may not cancel, revoke or otherwise amend the exercise of your Subscription Rights (unless we are required by law to permit revocation). Any Subscription Rights that are not exercised prior to 5:00 p.m., New York City time, on the Expiration Date will expire and you will have no further rights under your Subscription Rights Certificate.

Additional copies of the enclosed materials may be obtained from BNY Mellon Shareowner Services, which is acting as our information agent in the Rights Offering. You may also contact BNY Mellon Shareowner Services if you have any questions on the Rights Offering or require any assistance in exercising your Subscription Rights, by telephone, if you are located within the U.S., Canada or Puerto Rico, at (866) 339-6260 (toll free) or, if you are located outside the U.S., at (201) 680-6579 (collect).

Very truly yours,

PACIFIC CAPITAL BANCORP

2