Exhibit 10.96

AMENDMENT NO. 5

TO MASTER REPURCHASE AGREEMENT

Amendment No. 5 to Master Repurchase Agreement, dated as of April 14, 2015 (this “Amendment”), among Credit Suisse First Boston Mortgage Capital LLC (the “Buyer”), PennyMac Operating Partnership, L.P. (the “Seller”) and PennyMac Mortgage Investment Trust (the “Guarantor”).

RECITALS

The Buyer, the Seller and the Guarantor are parties to that certain Master Repurchase Agreement, dated as of September 28, 2012 (as amended by Amendment No. 1, dated as of May 8, 2013, Amendment No. 2, dated as of December 31, 2013, Amendment No. 3, dated as of January 10, 2014, and Amendment No. 4, dated as of October 31, 2014, the “Existing Repurchase Agreement”; and as further amended by this Amendment, the “Repurchase Agreement”) and the related Pricing Side Letter, dated as of September 28, 2012 (as amended, restated, supplemented or otherwise modified from time to time, the “Pricing Side Letter”). The Guarantor is a party to that certain Guaranty (as amended, restated, supplemented or otherwise modified from time to time, the “Guaranty”), dated as of September 28, 2012, by the Guarantor in favor of Buyer. Capitalized terms used but not otherwise defined herein shall have the meanings given to them in the Existing Repurchase Agreement and Guaranty, as applicable.

The Buyer, the Seller and the Guarantor have agreed, subject to the terms and conditions of this Amendment, that the Existing Repurchase Agreement be amended to reflect certain agreed upon revisions to the terms of the Existing Repurchase Agreement. As a condition precedent to amending the Existing Repurchase Agreement, the Buyer has required the Guarantor to ratify and affirm the Guaranty on the date hereof.

Accordingly, the Buyer, the Seller and the Guarantor hereby agree, in consideration of the mutual promises and mutual obligations set forth herein, that the Existing Repurchase Agreement is hereby amended as follows:

SECTION 1. Definitions. Section 2 of the Existing Repurchase Agreement is hereby amended by:

1.1 deleting the definition of “Mortgage Loan” in its entirety and replacing it with the following:

“Mortgage Loan” means any closed Agency Mortgage Loan or Non-Agency QM Mortgage Loan, which is a fixed or floating-rate, one-to-four-family residential mortgage loan evidenced by a promissory note and secured by a first lien mortgage, which satisfies the requirements set forth in the Underwriting Guidelines and Section 13(b) hereof.

1.2 deleting the definition of “Jumbo Mortgage Loan” in its entirety and replacing it with the following definition of “Non-Agency QM Mortgage Loan”:

“Non-Agency QM Mortgage Loan” means a Mortgage Loan that (a) does not meet the criteria for an Agency Mortgage Loan; (b) meets all applicable criteria as set forth in the Underwriting Guidelines and (c) is otherwise acceptable to Buyer in its sole discretion.

-1-

1.3 deleting all references to “Jumbo Mortgage Loan” in their entirety and replacing them with “Non-Agency QM Mortgage Loan”.

1.4 deleting all references to “Aged Jumbo Loan” in their entirety and replacing them with “Aged Non-Agency QM Mortgage Loan”.

1.5 adding the following definition in its proper alphabetical order:

“Agency Mortgage Loan” has the meaning assigned to such term in the Pricing Side Letter.

SECTION 2. Conditions Precedent. This Amendment shall become effective as of the date hereof (the “Amendment Effective Date”), subject to the satisfaction of the following conditions precedent:

2.1 Delivered Documents. On the Amendment Effective Date, the Buyer shall have received the following documents, each of which shall be satisfactory to the Buyer in form and substance:

(a) this Amendment, executed and delivered by duly authorized officers of the Buyer, the Seller and the Guarantor;

(b) Amendment No. 8 to that certain Pricing Side Letter, executed and delivered by duly authorized officers of the Buyer, the Seller and the Guarantor; and

(c) such other documents as the Buyer or counsel to the Buyer may reasonably request.

SECTION 3. Representations and Warranties. Seller hereby represents and warrants to the Buyer that it is in compliance with all the terms and provisions set forth in the Repurchase Agreement on its part to be observed or performed, and that no Event of Default has occurred and is continuing, and hereby confirms and reaffirms the representations and warranties contained in Section 13 of the Repurchase Agreement.

SECTION 4. Limited Effect. Except as expressly amended and modified by this Amendment, the Existing Repurchase Agreement shall continue to be, and shall remain, in full force and effect in accordance with its terms.

SECTION 5. Counterparts. This Amendment may be executed by each of the parties hereto on any number of separate counterparts (including by facsimile or .pdf), each of which shall be an original and all of which taken together shall constitute one and the same instrument.

SECTION 6. Severability. Any provision of this Amendment which is prohibited or unenforceable in any jurisdiction shall, as to such jurisdiction, be ineffective to the extent of such prohibition or unenforceability without invalidating the remaining provisions hereof, and any such prohibition or unenforceability in any jurisdiction shall not invalidate or render unenforceable such provision in any other jurisdiction.

-2-

SECTION 7. GOVERNING LAW. THIS AMENDMENT SHALL BE GOVERNED BY, AND CONSTRUED IN ACCORDANCE WITH, THE LAWS OF THE STATE OF NEW YORK WITHOUT REFERENCE TO THE CHOICE OF LAW PROVISIONS THEREOF.

SECTION 8. Reaffirmation of Guaranty. The Guarantor hereby ratifies and affirms all of the terms, covenants, conditions and obligations of the Guaranty and acknowledge and agree that the term “Obligations” as used in the Guaranty shall apply to all of the Obligations of Seller to Buyer under the Pricing Side Letter and the Repurchase Agreement, as amended hereby.

[Remainder of page intentionally left blank]

-3-

IN WITNESS WHEREOF, the parties have caused their names to be signed hereto by their respective officers thereunto duly authorized as of the day and year first above written.

Signature Page to Amendment No. 5 to Master Repurchase Agreement