Exhibit 99.5

LETTER TO BROKERS AND OTHER NOMINEE HOLDERS

PACIFIC CAPITAL BANCORP

Up to 726,975,565 Shares of Common Stock To Be Issued Upon the Exercise of Subscription Rights

[—], 2010

To Security Dealers, Commercial Banks,

Trust Companies and Other Nominees:

This letter is being distributed to securities dealers, commercial banks, trust companies, and other nominees in connection with the rights offering (the “Rights Offering”) by Pacific Capital Bancorp (the “Company”) of shares of its common stock (“Common Stock”), which will be issued upon the exercise of non-transferable subscription rights (the “Subscription Rights”), which are being distributed to all holders of record (the “Recordholders”) of Common Stock as of 4:01 p.m., New York City time, on August 30, 2010 (the “Record Date”). The Subscription Rights and the Rights Offering are described in the prospectus dated [—], 2010, which is enclosed with this letter (the “Prospectus”).

In the Rights Offering, the Company is offering up to an aggregate of 726,975,565 shares of Common Stock to be issued upon the exercise of the Subscription Rights, which is described further in the Prospectus. The Subscription Rights will expire, if not exercised earlier, at 5:00 p.m., New York City time, on [—], 2010, unless the Company elects in its sole discretion to extend the period of the Rights Offering beyond this date (as such date may be extended, the “Expiration Date”) or cancel the Rights Offering earlier.

As described in the Prospectus, each Recordholder will receive, at no charge, 15.335 Subscription Rights for each share of Common Stock owned on the Record Date. Subscription rights may only be exercised in whole numbers; the Company will not issue fractional rights and will round all of the Subscription Rights down to the nearest whole number for each Recordholder. Each whole Subscription Right will allow the holder thereof to subscribe to purchase one share of Common Stock at a subscription price of $0.20 per share. For example, if a Recordholder owned 1,000 shares of Common Stock on the Record Date, the Recordholder would receive 15,335 Subscription Rights and would have the right to purchase 15,335 shares of Common Stock for $0.20. However, if you owned 500 shares of Common Stock on the Record Date, you would receive 7,667 Subscription Rights and would have the right to purchase 7,667 shares of Common Stock for $0.20 per share.

You should be aware that there is no over-subscription right associated with the Rights Offering. In addition, no shareholder, including SB Acquisition Company LLC, a wholly owned subsidiary of Ford Financial Fund, L.P. (the “Investor”), will backstop the Rights Offering. Neither you nor any shareholder, including the Investor, will have the opportunity to purchase additional shares not purchased by other shareholders in the Rights Offering.

Each holder of Subscription Rights will be required to submit payment in full for all of the shares of Common Stock that the holder wishes to buy pursuant to the exercise of the holder’s Subscription Rights to BNY Mellon Shareowner Services (the “Subscription Agent”), by no later than 5:00 p.m., New York City time, on the Expiration Date. Any excess subscription payments that any holder may pay to the Subscription Agent in the Rights Offering will be returned, without interest or penalty, to the holder by the Subscription Agent as soon as practicable following the completion of the Rights Offering.

The Subscription Rights are evidenced by subscription rights certificates and election forms (each, a “Subscription Rights Certificate and Election Form”) registered in the Recordholder’s name. Subscription Rights are non-transferable, meaning that they may not be sold, transferred, or assigned by the Recordholder to any other party.

We are asking persons who hold shares of Common Stock beneficially and who have received the Subscription Rights distributable with respect to those shares through a broker, dealer, commercial bank, trust company, or other nominee, as well as persons who hold certificates of Common Stock directly and prefer to have such institutions effect transactions relating to the Subscription Rights on their behalf, to contact the appropriate institution or nominee and request it to effect the transactions for them. In addition, we are asking beneficial owners who wish to obtain a separate Subscription Rights Certificate and Election Form to contact the appropriate nominee as soon as possible and request that a separate Subscription Rights Certificate and Election Form be issued.

If you exercise the Subscription Rights on behalf of beneficial owners of Subscription Rights you will be required to certify to the Company, the Subscription Agent, and the Information Agent (as defined below), with respect to each beneficial owner of Subscription Rights on whose behalf you are acting, as to the aggregate number of Subscription Rights that have been exercised and the corresponding number of shares of Common Stock subscribed for in the Rights Offering.

All commissions, fees, and other expenses (including brokerage commissions and transfer taxes), other than fees and expenses of the Subscription Agent and BNY Mellon Shareowner Services (the “Information Agent”), incurred in connection with the exercise of the Subscription Rights will be for the account of the holder of the Subscription Rights, and none of such commissions, fees, or expenses will be paid by the Company, the Subscription Agent, or the Information Agent.

Enclosed are copies of the following documents:

Your prompt action is requested. To exercise the Subscription Rights, you must deliver the properly completed and signed Subscription Rights Certificate and Election Form (or the Notice of Guaranteed Delivery if you are following the guaranteed delivery procedures), together with payment in full of the total subscription amount that is required for all of the shares subscribed for in the Rights Offering, to the Subscription Agent as described further in the Prospectus. The properly completed and signed Subscription Rights Certificate and Election Form or Notice of Guaranteed Delivery, in either case accompanied by full payment of the total subscription amount, must be received by the Subscription Agent, and your payment must clear, by no later than 5:00 p.m., New York City time, on the Expiration Date. Failure to return the properly completed Subscription Rights Certificate and Election Form (or the Notice of Guaranteed Delivery) with the correct payment will result in your not being able to exercise the Subscription Rights held in your name on behalf of yourself or other beneficial owners. A Subscription Rights holder cannot revoke the exercise of Subscription Rights (unless we are required by law to permit revocation). Subscription Rights not exercised before 5:00 p.m., New York City time, on the Expiration Date will expire.

2

Additional copies of the enclosed materials may be obtained from the Information Agent by calling, if you are located within the U.S., Canada or Puerto Rico, (866) 339-6260 (toll free) or, if you are located outside the U.S., (201) 680-6579 (collect).

Very truly yours,

PACIFIC CAPITAL BANCORP

NOTHING CONTAINED IN THE PROSPECTUS OR IN ANY OF THE ENCLOSED DOCUMENTS SHALL CONSTITUTE YOU OR ANY PERSON AS AN AGENT OF PACIFIC CAPITAL BANCORP, THE SUBSCRIPTION AGENT, THE INFORMATION AGENT, OR ANY OTHER PERSON MAKING OR DEEMED TO BE MAKING OFFERS OF THE SECURITIES ISSUABLE UPON VALID EXERCISE OF THE SUBSCRIPTION RIGHTS, OR AUTHORIZE YOU OR ANY OTHER PERSON TO MAKE ANY STATEMENTS ON BEHALF OF ANY OF THEM WITH RESPECT TO THE RIGHTS OFFERING EXCEPT FOR STATEMENTS MADE IN THE PROSPECTUS.

3