Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the reference to our firm under the caption “Experts” in the Registration Statement (Form S-1) and related Prospectus of Pacific Capital Bancorp for the issuance of non-transferable subscription rights and to the incorporation by reference therein of our reports dated March 12, 2010 (except for the effects of discontinued operations as described in Note 7, as to which the date is October 5, 2010), with respect to the consolidated financial statements of Pacific Capital Bancorp for the year ended December 31, 2009, included in its Current Report on Form 8-K filed with the Securities and Exchange Commission on October 5, 2010, and our report dated March 12, 2010, with respect to the effectiveness of internal control over financial reporting of Pacific Capital Bancorp, included in its Annual Report (Form 10-K) for the year ended December 31, 2009, filed with the Securities and Exchange Commission on March 12, 2010.

/s/    Ernst & Young LLP

Los Angeles, California

October 5, 2010