Exhibit 10.1

[Execution]

AMENDMENT NO. 15 TO LOAN AND SECURITY AGREEMENT

AMENDMENT NO. 15 TO LOAN AND SECURITY AGREEMENT, dated as of March 31, 2010 (this “Amendment No. 15”), entered into by and among Wachovia Bank, National Association, successor by merger to Congress Financial Corporation (Florida), in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, LLC, a Delaware limited liability company formerly known as Supreme International, Inc. (“Supreme”), Jantzen, LLC, a Delaware limited liability company formerly known as Jantzen, Inc. (“Jantzen”), Perry Ellis Menswear, LLC, a Delaware limited liability company formerly known as Perry Ellis Menswear, Inc. (“Perry Ellis Menswear”), Perry Ellis Europe Limited, a private limited company incorporated in England and Wales formerly known as Farah Manufacturing (U.K.) Limited (“Perry Europe”), Salant Holding, LLC, a Delaware limited liability company formerly known as Salant Holding Corporation (“Salant Holding” and together with Supreme, Jantzen, Perry Europe and Perry Ellis Menswear, each individually “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PEI Licensing”), Jantzen Apparel, LLC, a Delaware limited liability company formerly known as Jantzen Apparel Corp. (“Jantzen Apparel”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, a Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), Perry Ellis Shared Services Corporation, a Delaware corporation (“PE Shared Services”), Winnsboro DC, LLC, a Delaware limited liability company (“Winnsboro”), Tampa DC, LLC, a Delaware limited liability company (“Tampa DC”), Perry Ellis International Group Holdings Limited, a private company incorporated under the laws of Ireland having its principal place of business in the Bahamas (“Group Holdings”) and Perry Ellis Real Estate, LLC, a Delaware limited liability company formerly known as Perry Ellis Real Estate Corporation (“PE Real Estate” and, together, with Parent, PEI Licensing, Jantzen Apparel, Supreme I, Supreme II, Supreme Realty, Group Holdings, PE Shared Services, Winnsboro, Tampa DC, and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

W I T N E S S E T H :

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1, 2002, by and among Agent, Lenders, Borrowers and Guarantors, as amended by Amendment No. 1 to Loan and Security Agreement,

dated June 19, 2003, Amendment No. 2 to Loan and Security Agreement, dated September 22, 2003, Amendment No. 3 to Loan and Security Agreement, dated December 1, 2003, Amendment No. 4 to Loan and Security Agreement, dated February 25, 2004, Amendment No. 5 to Loan and Security Agreement, dated July 1, 2004, Amendment No. 6 to Loan and Security Agreement, dated as of September 30, 2004, Amendment No. 7 to Loan and Security Agreement, dated as of February 26, 2005, Amendment No. 8 to Loan and Security Agreement, dated as of September 30, 2005, Amendment No. 9 to Loan and Security Agreement, dated as of February 24, 2006, Amendment No. 10 to Loan and Security Agreement, dated as of August 28, 2006, Amendment No 11 to Loan and Security Agreement, dated as of November 29, 2006, Amendment No. 12 and Consent to Loan and Security Agreement, dated as of December 6, 2006, Amendment No. 13 to Loan and Security Agreement, dated as of October 30, 2008 and Amendment No. 14 to Loan and Security Agreement, dated as of October 27, 2009 (as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and/or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”);

WHEREAS, Borrowers and Guarantors have requested that Agent and Lenders agree to make certain amendments to the Loan Agreement, and Agent and Lenders are willing to agree to such amendments, subject to the terms and conditions set forth in this Amendment No. 15; and

WHEREAS, by this Amendment No. 15, Agent, Lenders, Borrowers and Guarantors desire and intend to evidence such amendments;

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions.

(a) Additional Definitions.

(i) “Amendment No. 15” shall mean Amendment No. 15 to Loan and Security Agreement, dated as of March 31, 2010, by and among Agent, Lenders, Borrowers and Guarantors.

(ii) “Amendment No. 15 Effective Date” shall mean the first date on which all of the conditions precedent to the effectiveness of Amendment No. 15 shall have been satisfied and/or waived.

(iii) “Bank of America” shall mean Bank of America, N.A. and its successors and assigns.

(iv) “PrimeRevenue” shall mean PrimeRevenue, Inc. and its successors and assigns.

2

(v) “Prime Revenue Program Documents” shall mean the (A) Accounts Receivable Purchase Agreement, by and among Bank of America, Jantzen, Supreme, Perry Ellis Menswear and Salant Holding, (B) Supplier Agreement, by and among Jantzen, Supreme, Salant Holding, Perry Ellis Menswear and PrimeRevenue, (C) Release of Certain Accounts Receivable, dated March 31, 2010, by Agent in favor of Bank of America and (D) all other agreements, documents and instruments executed and/or delivered in connection with the foregoing.

(b) Interpretation. For purposes of this Amendment No. 15, unless otherwise defined herein, all capitalized terms used herein which are defined in the Loan Agreement shall have the meanings given to such terms in the Loan Agreement.

2. Eligible Accounts. Notwithstanding anything to the contrary in the Loan Agreement, no Account of any Borrower or Guarantor in respect of which Kohl’s Corporation (“Kohl’s”) is the account debtor (each a “Kohl’s Account” and collectively, the “Kohl’s Accounts”) shall be deemed to constitute “Eligible Accounts”, except to the extent that Agent may determine in its sole discretion that such Kohl’s Accounts meet the criteria for “Eligible Accounts” set forth in the Loan Agreement.

3. Kohl’s Accounts. Section 5 of the Loan Agreement is hereby amended to include the following additional Section 5.3:

“5.3 Kohl’s Accounts.

(a) Each Borrower and Guarantor hereby acknowledges and agrees that with respect to the sale of any Kohl’s Accounts by or on behalf of any Borrower to Bank of America, effective upon the purchase by Bank of America of such Account (each such Kohl’s Account, a “Purchased Account”), the security interest of Agent and Lenders in such Purchased Account shall be deemed automatically released by Agent and Lenders without any further action by Agent and Lenders, and Agent and Lenders agree to take such further actions as may be reasonably requested by Bank of America to further effectuate or evidence such release.

(b) Each Borrower and Guarantor hereby acknowledges and agrees that, to the extent that any Borrower or Guarantor repurchases any of the Purchased Accounts from Bank of America, (i) the first priority perfected security interest and lien of Agent and Lenders therein shall automatically attach thereto, (ii) all Purchased Accounts so repurchased shall constitute Collateral and (iii) such Purchased Accounts so repurchased shall not constitute Eligible Accounts.”

3

4. Sales of Assets. Section 9.7(b) of the Loan Agreement is hereby amended to include the following additional section (xi):

“and (xi) the sale, pursuant to the Prime Revenue Program Documents as in effect on the date hereof, from time to time, of Accounts owing to any Borrower by Kohl’s; provided, that, (A) all amounts payable by Bank of America to any Borrower or Guarantor under the Prime Revenue Program Documents shall be paid to a cash management account subject to a Deposit Account Control Agreement for application to the Obligations in accordance with Section 6.4 hereof and (B) the Prime Revenue Program Documents shall be in full force and effect and each Borrower or Guarantor party thereto shall be in compliance with the material terms and conditions thereof and no breach of such terms or default or event of default thereunder shall exist or have occurred.”

5. Prime Revenue Program Documents. Section 9 of the Loan Agreement is hereby amended to include the following additional Section 9.21:

“9.21 Prime Revenue Program Documents. Borrowers and Guarantors shall not agree to any amendment, waiver, or other modification to any of the Prime Revenue Program Documents without the prior written consent of Agent. Borrowers shall furnish Agent all materials notices or demands in connection with the Prime Revenue Program Documents received by any Borrower or Guarantor or on its behalf, promptly after the receipt thereof, or sent by any Borrower or Guarantor or on its behalf, concurrently with the sending thereof, as the case may be.”

6. Reporting. Borrowers and Guarantors agree that together with each delivery of a Borrowing Base Certificate to Agent pursuant to the terms of the Loan Agreement, Borrowers shall report all Kohl’s Accounts offered for sale pursuant to the Prime Revenue Program Documents and such other information regarding such Kohl’s Accounts as Agent may reasonably request from time to time.

7. Representations, Warranties and Covenants. Borrowers and Guarantors, jointly and severally, represent, warrant and covenant with and to Agent and Lenders as follows, which representations, warranties and covenants shall survive the execution and delivery hereof:

(a) this Amendment No. 15 and all other documents, agreements and instruments executed by any Borrower or Guarantor in connection herewith (together with this Amendment No. 15, the “Amendment Documents”) have been duly authorized, executed and delivered by all necessary action on the part of each Borrower and Guarantor which is a party hereto and, if necessary, their respective stockholders, and are in full force and effect as of the date hereof, and the agreements and obligations of Borrowers and Guarantors contained herein and therein constitute legal, valid and binding obligations of Borrowers and Guarantors enforceable against them in accordance with their terms except as such enforceability may be limited by (i) bankruptcy, insolvency, reorganization, moratorium or similar laws of general applicability affecting the enforcement of creditors’ rights and (ii) the application of general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law);

4

(b) neither the Amendment Documents nor any of the Prime Revenue Program Documents nor the transactions contemplated thereby are in contravention of any applicable law, or the terms of any agreement to which any Borrower or Guarantor is a party or by which any property of any Borrower or Guarantor is bound; and

(c) as of the date hereof, no Default or Event of Default exists or has occurred and is continuing.

8. Conditions Precedent. The terms and provisions of this Amendment No. 15 shall only be effective upon the satisfaction of each of the following conditions precedent in a manner satisfactory to Agent:

(a) Agent shall have received executed counterparts of this Amendment No. 15, duly authorized, executed and delivered by Borrowers and Guarantors and such Lenders as are required under the Loan Agreement to approve the transactions contemplated by this Amendment No. 15;

(b) Agent shall have received true, correct and complete executed copies of each of the Prime Revenue Program Documents in form and substance satisfactory to Agent, the receipt of each of which, in satisfactory form, is hereby acknowledged by Agent; and

(c) No Default or Event of Default shall exist or have occurred and be continuing.

9. Effect of this Amendment. This Amendment No. 15 and the other Amendment Documents constitute the entire agreement of the parties with respect to the subject matter hereof and thereof, and supersede all prior oral or written communications, memoranda, proposals, negotiations, discussions, term sheets and commitments with respect to the subject matter hereof and thereof. Except as expressly provided herein, no other changes or modifications to the Financing Agreements are intended or implied, and in all other respects the Financing Agreements are hereby specifically ratified, restated and confirmed by all parties hereto as of the effective date hereof. To the extent that any provision of the Loan Agreement or any of the other Financing Agreements are inconsistent with the provisions of this Amendment No. 15, the provisions of this Amendment No. 15 shall control.

10. Further Assurances. Each Borrower and Guarantor shall execute and deliver such additional documents and take such additional action as may be reasonably requested by Agent to effectuate the provisions and purposes of this Amendment No. 15.

11. Release of Claims. No Borrower or Guarantor has any actual or potential claim or cause of action against Agent or any Lender with respect to any matters relating to the Financing Agreements and related transactions through the date hereof, and hereby waives and releases any right to assert same.

5

12. Governing Law. The rights and obligations hereunder of each of the parties hereto shall be governed by and interpreted and determined in accordance with the internal laws of the State of Florida (but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of Florida).

13. Binding Effect. This Amendment No. 15 shall be binding upon and inure to the benefit of each of the parties hereto and their respective successors and assigns.

14. Counterparts. This Amendment No. 15 may be executed in any number of counterparts, but all of such counterparts shall together constitute but one and the same agreement. In making proof of this Amendment No. 15, it shall not be necessary to produce or account for more than one counterpart thereof signed by each of the parties hereto. Delivery of an executed counterpart of this Amendment No. 15 by telecopier or other method of electronic transmission shall have the same force and effect as delivery of an original executed counterpart of this Amendment No. 15. Any party delivering an executed counterpart of this Amendment No. 15 by telecopier or other method of electronic transmission also shall deliver an original executed counterpart of this Amendment No. 15, but the failure to deliver an original executed counterpart shall not affect the validity, enforceability, and binding effect of this Amendment No. 15 as to such party or any other party.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

6

IN WITNESS WHEREOF, the parties hereto have caused this Amendment No. 15 to be duly executed and delivered by their authorized officers as of the day and year first above written.

Signature Page to Amendment No. 15

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

Signature Page to Amendment No. 15

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

Signature Page to Amendment No. 15

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

Signature Page to Amendment No. 15

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

Signature Page to Amendment No. 15

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

Signature Page to Amendment No. 15