Exhibit 99.3

EXECUTION VERSION

SECOND AMENDED AND RESTATED INVESTOR RIGHTS AGREEMENT

AMONG

XENITH BANKSHARES, INC.,

as the Company

BCP FUND I VIRGINIA HOLDINGS, LLC,

as the sole Institutional Investor

AND

THE HOLDERS

(as defined herein)

DATED AS OF FEBRUARY 10, 2016

TABLE OF CONTENTS

SECOND AMENDED AND RESTATED INVESTOR RIGHTS AGREEMENT

THIS SECOND AMENDED AND RESTATED INVESTOR RIGHTS AGREEMENT (this “Agreement”) is entered into as of February 10, 2016 by and among (i) Xenith Bankshares, Inc., a Virginia corporation (the “Company”), (ii) BCP Fund I Virginia Holdings, LLC, a Delaware limited liability company (“BCP Fund”), and (iii) those shareholders who purchased shares of common stock, $1.00 par value (the “Xenith Common Stock”), of Xenith Corporation, a Virginia corporation (the “Predecessor”), in the Xenith Private Placement (such Persons are referred to herein collectively as the “Holders” and individually as a “Holder”).

WHEREAS, in connection with the Xenith Private Placement, the Predecessor and the Holders entered into an Investor Rights Agreement, dated as of June 26, 2009 (the “Original Investor Rights Agreement”), in order to provide the Holders, including the Institutional Investors, with certain rights with respect to the election of directors and the registration of their shares of Xenith Common Stock, including those shares issuable upon exercise of any options to purchase shares of Xenith Common Stock and any warrants exercisable into shares of Xenith Common Stock; and

WHEREAS, the Predecessor entered into an Agreement of Merger, dated May 12, 2009, as amended (the “Predecessor Merger Agreement”), with First Bankshares, Inc. (“FBS”), pursuant to which the Predecessor merged with and into FBS (the “Merger”), with FBS being the surviving corporation in the Merger; and

WHEREAS, the Predecessor, BCP Fund and the D&O Holders entered into the First Amendment to the Investor Rights Agreement, dated as of December 21, 2009 (the “First Amendment” and, together with the Original Investor Rights Agreement, the “Investor Rights Agreement”), that limited the obligations of the Holders under Article 2 of the Original Investors Rights Agreement to the Institutional Investors and the D&O Holders only, and relieved the remaining Holders of their obligations under Article 2; and

WHEREAS, the Merger was effected on December 22, 2009; and

WHEREAS, at the effective time of the Merger, FBS changed its name to Xenith Bankshares, Inc. and assumed the rights and obligations of the Predecessor under the Investor Rights Agreement by operation of law; and

WHEREAS, under the terms of the Predecessor Merger Agreement, all shares of Xenith Common Stock outstanding immediately prior to the effective time of the Merger were cancelled and converted automatically into shares of common stock of the Company (“XBKS Common Stock”) and all options and warrants to purchase shares of Xenith Common Stock outstanding at the effective time of the Merger were converted into options and warrants to purchase shares of XBKS Common Stock; and

WHEREAS, the Company, BCP Fund and the Holders entered in an Amended and Restated Investor Rights Agreement, dated as of December 23, 2010 (the “First Amended and Restated Investor Rights Agreement”), pursuant to which the Investor Rights Agreement was amended and restated in its entirety; and

WHEREAS, this Agreement is amending and restating the First Amended and Restated Investor Rights Agreement in its entirety in anticipation of the transactions contemplated by the Agreement and Plan of Reorganization, dated as of February 10, 2016, by and between the Company and Hampton Roads Bankshares, Inc., a Virginia corporation (“HRB”) (such agreement, the “Merger Agreement,” and such transactions, the “Anticipated Merger”); and

WHEREAS, as of the date of this Agreement, BCP Fund is the sole Institutional Investor; and

WHEREAS, the Company and BCP Fund, as the sole Institutional Investor, desire to amend and restate the First Amended and Restated Investor Rights Agreement pursuant to Section 4.1 thereof solely for administrative convenience and without any intention of affecting (adversely or otherwise) any rights or obligations of the Company, the Institutional Investors or the other Holders under the Original Investor Rights Agreement, as amended by the First Amendment; and

WHEREAS, capitalized terms used in this Agreement and not otherwise defined shall have the meanings ascribed to them in Article 3 hereof.

NOW, THEREFORE, for and in consideration of the foregoing and of the mutual covenants and agreements hereinafter set forth, the parties hereto agree as follows:

This Section 1 shall only become effective upon the closing of the Anticipated Merger and, until such time, the provisions of Section 2 of the First Amended and Restated Investor Rights Agreement shall remain in full force and effect.

The Company hereby agrees to take such actions as are necessary, and each Institutional Investor and D&O Holder agrees to vote its shares of XBKS Common Stock and take such other actions as are necessary, so as to nominate for election, and, in the case of the Institutional Investor and D&O Holders, elect, reelect, vote or replace as a Director of the Company, one (1) individual designated by BankCap at the 2016 annual meeting of shareholders of the Company, at the 2017 annual meeting of the shareholders of the Company and at any other meeting of the shareholders of the Company prior to the 2018 annual meeting of the shareholders Company at which the shareholders will be voting on the election of Directors. The Director designated by BankCap pursuant to this Section 1.2 shall be referred to as the “BankCap Director.” If at any time, however, BankCap, together with its affiliates, are no longer the beneficial owners of shares of XBKS Common Stock representing at least 5% of the outstanding capital stock of the Company, the BankCap Director shall promptly offer his resignation to the then current Chairman of the Board of Directors of the Company.

2

For so long as BankCap is entitled to designate and maintains a BankCap Director pursuant to Section 1.2, the Company hereby agrees to take such actions as are necessary, and each Institutional Investor and D&O Holder agrees to vote its shares of XBKS Common Stock, and take such other actions as are necessary, for the removal of the BankCap Director upon the request of BankCap and for the election to the Board of Directors of a substitute designated by BankCap in accordance with Section 1.2 hereof.

For so long as BankCap is entitled to designate and maintains a BankCap Director pursuant to Section 1.2, the Company hereby agrees to take such actions as are necessary, and each Institutional Investor and D&O Holder agrees to vote its shares of XBKS Common Stock and take such other actions as are necessary, in such manner as shall be necessary or appropriate to ensure that any vacancy on the Board of Directors of the Company caused by the resignation, removal or death of the BankCap Director shall be filled only in accordance with Section 1.2 hereof.

For the avoidance of doubt, the provisions of this Article 1 shall not apply to any Holder other than Institutional Investors and the D&O Holders.

No later than thirty (30) calendar days after the date of this Agreement, BankCap shall promptly prepare and file with the appropriate regulatory agencies all applications, notices, petitions and filings required to be filed by BankCap in order to receive the merger consideration from consummation of the Anticipated Merger and otherwise consummate the transactions contemplated by the Merger Agreement (the “Required Approvals”). The Company and HRB shall, subject to applicable law, have the right to review in advance, and, to the extent practicable, consult with BankCap on, any filing made with, or written materials, submitted to, any regulatory agency or governmental entity in connection with the the Required Approvals or the transactions contemplated by the Merger Agreement. BankCap agrees that it will consider in good faith the Company’s and HRB’s views in connection with such filings. BankCap further agrees to furnish to the Company and HRB a final copy of each filing made with any regulatory agency in connection with the Required Approvals or the transactions contemplated by the Merger Agreement, subject to applicable laws governing the confidentiality of such information. BankCap shall use its reasonable best efforts to obtain as promptly as practicable the Required Approvals and take no action that would reasonably be expected to adversely affect or delay the ability to obtain any necessary approvals of any regulatory agency or other governmental entity required for the transactions contemplated by the Merger Agreement.

3

The Company agrees that it shall enforce and not waive the conditions to closing contained in Section 7.1(c) of the Merger Agreement, including as they relate to BCP Fund, unless it has obtained the prior written consent of BCP Fund to such waiver. In addition, the Company acknowledges and agrees that BCP Fund would be irreparably harmed if this Section 2.2 is not performed in accordance with its specific terms and that any breach of this Section 2.2 by the Company could not be adequately compensated in all cases by monetary damages alone. Accordingly, the Company agrees that, in addition to any other right or remedy to which BCP Fund may be entitled at law or in equity, BCP Fund shall be entitled to enforce this Section 2.2 by a decree of specific performance and to obtain temporary, preliminary, and permanent injunctive relief to prevent breaches or threatened breaches, without posting any bond or giving any other undertaking.

The capitalized terms contained in this Agreement shall have the following meanings unless otherwise specifically defined:

“Act” means the Securities Act of 1933, as amended.

“BankCap” means BankCap Partners Fund I, L.P., a Delaware limited partnership, together with its wholly-owned subsidiary BCP Fund.

“D&O Holders” means those eleven Holders who were directors or executive officers of the Predecessor as of the date of the First Amendment and who became directors or executive officers of the Company at the effective time of the Merger.

“Institutional Investors” means those Holders who purchased in the Xenith Private Placement not less than 20% of the outstanding shares of Xenith Common Stock as of the date of the Xenith Private Placement, giving effect to the shares of Xenith Common Stock issued to the Holders on the date of the Xenith Private Placement, but excluding all then outstanding options and warrants.

“Person” means any individual, partnership, limited liability company, joint venture, corporation, trust, unincorporated organization, government or department or agency of a government.

“Registrable Securities” means (i) any shares of Xenith Common Stock held by any Holder as of June 26, 2009, (ii) any additional shares of Xenith Common Stock or XBKS Common Stock acquired by any Holder after June 26, 2009 (other than shares acquired upon exercise of employee stock options or similar employee awards but including any shares of Xenith Common Stock or XBKS Common Stock issued upon exercise of any other convertible securities such as warrants), (iii) any equity securities of FBS issued to the Holders in exchange for their shares of Xenith Common Stock in connection with the Merger, and (iv) any equity securities of the Company issued as a distribution with respect to or in exchange for or in replacement of any of the securities referred to in clauses (i), (ii) or (iii) above; provided, however, that Registrable Securities shall not include any securities that have been previously sold pursuant to a registration statement filed under the Act or under Rule 144 promulgated

4

under the Act, or which have otherwise been transferred in a transaction in which the transferor’s rights under this Agreement are not assigned, or, shares that are eligible for sale by a Holder pursuant to Rule 144(k) under the Act.

“Xenith Private Placement” means the private placement of shares of Xenith Common Stock completed on June 26, 2009.

If the Anticipated Merger is not consummated and the related Merger Agreement is terminated in accordance with its terms, this Agreement shall automatically become null, void and of no effect without any further action required by the parties.

This Agreement constitutes the entire agreement among the parties hereto with respect to the matters provided for herein, and it supersedes all prior oral or written agreements, commitments or understandings with respect to the matters provided for herein between or among such parties. This Agreement may be amended or modified in any respect, or any term hereof may be waived, with the written consent of the Institutional Investors who own at least two-thirds (2/3rds) of the outstanding Registrable Securities on a fully-diluted basis then held by all Institutional Investors; provided that no provision hereof may be amended if it would have the effect of materially adding to the obligations of the Company without the written consent of the Company; provided further that no provision hereof may be amended if it would adversely affect the rights or obligations of the Holders (other than Institutional Investors) without the written consent of Holders (other than Institutional Investors) who own at least two-thirds (2/3rds) of the outstanding Registrable Securities on a fully-diluted basis then held by all Holders (other than Institutional Investors).

No delay or failure on the part of any party hereto in exercising any right, power or privilege under this Agreement or under any other instruments given in connection with or pursuant to this Agreement shall impair any such right, power or privilege or be construed as a waiver of any such right, power or privilege. No single or partial exercise of any such right, power or privilege shall preclude the further exercise of such right, power or privilege, or the exercise of any other right, power or privilege.

This Agreement shall forthwith become wholly void and of no effect as to any Holder (including any permitted assignee of such Holder), at such time as such person no longer owns any securities constituting Registrable Securities.

No rights or obligations of any Holder may be transferred under this Agreement and any transferee of any shares of XBKS Common Stock or Registrable Securities shall not be entitled to any rights or obligations under this Agreement; provided that this Agreement shall be binding upon any successor in interest (including successors by merger) to the Company.

5

This Agreement, the rights and obligations of the parties hereto, and any claims or disputes relating thereto, shall be governed by and construed in accordance with the laws of the Commonwealth of Virginia (excluding the choice of law rules thereof).

All notices, demands, requests, or other communications which may be or are required to be given, served, or sent by any party to any other party pursuant to this Agreement shall be in writing and shall be hand delivered, faxed, sent by overnight courier or mailed by first-class, registered or certified mail, return receipt requested, postage prepaid, addressed as follows:

Xenith Bankshares, Inc.

One James Center

901 E. Cary Street, Suite 1700

Richmond, Virginia 23219

Facsimile No.: 804-433-2194

Attention: T. Gaylon Layfield, III

with a copy (which shall not constitute notice) to:

Hunton & Williams LLP

Riverfront Plaza, East Tower

951 E. Byrd Street

Richmond, Virginia 23219

Facsimile No: 804-788-8218

Xenith Bankshares, Inc.

One James Center

901 E. Cary Street, Suite 1700

Richmond, Virginia 23219

Facsimile No.: 804-433-2194

Attention: Corporate Secretary

Each party may designate by notice in writing (given in accordance with the terms hereof) a new address to which any notice, demand, request or communication may thereafter be so given, served or sent. Each notice, demand, request, or communication which shall be hand delivered, sent or mailed, in the manner described above, shall be deemed sufficiently given, served, sent, received or delivered for all purposes at such time as it is delivered to the addressee

6

(with the return receipt, the delivery receipt or a facsimile confirmation being deemed conclusive, but not exclusive, evidence of such delivery) or at such time as delivery is refused by the addressee upon presentation.

HRB is hereby made an express third party beneficiary with respect to the obligations of BankCap contained solely in Section 2.1 of this Agreement, and shall be entitled to enforce the provisions of Section 2.1 to the same extent as if it were a party hereto.

To facilitate execution, this Agreement may be executed in as many counterparts as may be required; and it shall not be necessary that the signatures of, or on behalf of, each party, or that the signatures of all persons required to bind any party, appear on each counterpart; but it shall be sufficient that the signature of, or on behalf of, each party, or that the signatures of the persons required to bind any party, appear on one or more of the counterparts. All counterparts shall collectively constitute a single agreement. It shall not be necessary in making proof of this Agreement to produce or account for more than one of the counterparts of this Agreement containing the respective signatures of or on behalf of, all of the parties hereto.

[Signatures Appear on Following Pages]

7

Signature Page to Second Amended and Restated Investor Rights Agreement

8

Signature Page to Second Amended and Restated Investor Rights Agreement

9