Exhibit 5.1

May 17, 2012

Capital One Financial Corporation

1680 Capital One Drive

McLean, Virginia 22102

Registration Statement on Form S-3 (File No. 333-181047)

Ladies and Gentlemen:

We have examined the Registration Statement on Form S-3, File No. 333-181047 (the “Registration Statement”), of Capital One Financial Corporation, a Delaware corporation (the “Company”), filed with the Securities and Exchange Commission (the “Commission”) pursuant to the Securities Act of 1933, as amended (the “Securities Act”), in connection with the offering by the selling stockholder identified in the Registration Statement of up to 54,028,086 shares of the Company’s common stock, par value $0.01 per share (the “Shares”).

We have examined the originals, or photostatic or certified copies, of such records of the Company and certificates of officers of the Company and of public officials and such other documents as we have deemed relevant and necessary as the basis for the opinions set forth below. In our examination, we have assumed the genuineness of all signatures, the legal capacity and competency of all natural persons, the authenticity of all documents submitted to us as originals and the conformity to original documents of all documents submitted to us as copies.

Based upon the foregoing examination and in reliance thereon, and subject to the assumptions stated and in reliance on statements of fact contained in the documents that we have examined, we are of the opinion that the Shares have been validly issued and are fully paid and non-assessable.

We render no opinion herein as to matters involving the laws of any jurisdiction other than, to the extent relevant for our opinions herein, the Delaware General Corporation Law. This opinion is limited to the effect of the current state of the Delaware General Corporation Law and the facts as they currently exist. We assume no obligation to revise or supplement this opinion in the event of future changes in such laws or the interpretations thereof or such facts.

We consent to the filing of this opinion as an exhibit to the Registration Statement, and we further consent to the use of our name under the caption “Certain Legal Matters” in the Registration Statement and the prospectus that forms a part thereof and under the caption “Validity of the Common Stock” in the prospectus supplement, dated May 17, 2012, relating to the offering of the Shares. In giving these consents, we do not thereby admit that we are within the category of persons whose consent is required under Section 7 of the Securities Act or the Rules and Regulations of the Commission.

Very truly yours,

/s/ Gibson, Dunn & Crutcher LLP