Exhibit 10.1

AMENDED AND RESTATED

CONSULTING AGREEMENT

This Amended and Restated Consulting Agreement (the “Amended and Restated Agreement”) is effective as of April 28, 2011 (the “Effective Date”), by and between Casual Male Retail Group, Inc., a Delaware corporation (the “Corporation”), with its principal executive offices located at 555 Turnpike Street, Canton, Massachusetts 02021, and Jewelcor Management, Inc., a Nevada corporation (the “Independent Contractor”), having its principal executive offices located at 100 North Wilkes-Barre Boulevard, Wilkes-Barre, Pennsylvania 18702.

RECITALS

WHEREAS, the stated term of the Consulting Agreement dated October 28, 1999 between the Corporation and the Independent Contractor (the “Original Consulting Agreement”) ended on April 28, 2000; and

WHEREAS, the Corporation confirmed the retention of the Independent Contractor, effective as of April 29, 2000 (the “Agreement”) to continue as a consultant to the Corporation to assist the Corporation in developing and implementing a strategic plan for the Corporation and for other related consulting services to which the parties may agree, as described in Section 3 (the “Services”); and

WHEREAS the Independent Contractor agrees to perform the Services for the Corporation under the terms and conditions set forth in this Amended and Restated Agreement, it being expressly understood that the Independent Contractor shall perform Services as an independent contractor and nothing contained herein shall be construed to be inconsistent with this relationship or status; and

WHEREAS, given the passage of time and the numerous amendments to the Agreement, the parties desire to enter into this Amended and Restated Agreement;

NOW, THEREFORE, for and in consideration of the mutual promises and covenants set forth in this Amended and Restated Agreement, and for other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the Corporation and the Independent Contractor hereby agree as follows:

SECTION ONE

Representations and Warranties of the Independent Contractor

The Independent Contractor represents, warrants, covenants and agrees that:

(a) the Independent Contractor is a corporation duly organized, validly existing and in good standing under the laws of the State of Nevada and is duly qualified and in good standing as a foreign corporation in each jurisdiction where its performance of Services requires such qualification;

(b) the Independent Contractor has all necessary power and authority to execute and deliver this Amended and Restated Agreement and to perform all of its obligations under this Amended and Restated Agreement;

(c) this Amended and Restated Agreement has been duly and validly authorized, executed and delivered by the Independent Contractor, and constitutes the valid and binding obligation of the Independent Contractor, and is enforceable against the Independent Contractor in accordance with its terms; and

(d) the execution, delivery and performance by the Independent Contractor of this Amended and Restated Agreement does not (I) violate or conflict with any provision of the Independent Contractor’s charter or By-Laws; (2) violate, conflict with, or result in a breach or termination of (or require any consent or approval under) any agreement, license, arrangement or understanding, whether written or oral, to which the Independent Contractor, its agents or employees (or anyone of them) is a party; or (3) violate any law, judgment, decree, order, rule or regulation applicable to the Independent Contractor, its agents or employees (or anyone of them).

SECTION TWO

Representations and Warranties of the Corporation

The Corporation represents, warrants, covenants and agrees that:

(a) the Corporation is a corporation duly organized, validly existing and in good standing under the laws of the State of Delaware;

(b) the Corporation has all necessary power and authority to execute and deliver this Amended and Restated Agreement and to perform all of its obligations under this Amended and Restated Agreement;

(c) this Amended and Restated Agreement has been duly and validly authorized, executed and delivered by the Corporation, and constitutes the valid and binding obligation of the Corporation, and is enforceable against the Corporation in accordance with its terms; and

(d) the execution, delivery and performance by the Corporation of this Amended and Restated Agreement does not (1) violate or conflict with any provision of the Corporation’s Certificate of Incorporation or By-Laws; (2) violate, conflict with, or result in a breach or termination of (or require any consent or approval under) any agreement, license, arrangement or understanding, whether written or oral, to which the Corporation is a party; or (3) violate any law, judgment, decree, order, rule or regulation applicable to the Corporation.

2

SECTION THREE

Nature of the Services

From time to time, at the request of the Board of Directors of the Corporation (the “Board”) or a Committee of the Board, the Independent Contractor will be responsible, on the Board”s behalf, for overseeing the direction and execution of transactions concerning the strategic direction of the Corporation, including:

In addition to the foregoing, at the request of the Board or a Committee of the Board, the Independent Contractor shall provide day-to-day advisory and consulting services to the management and Board or the Committee with respect to the Company’s on-going operations and the achievement of its planned objectives.

In accordance with the terms and conditions of this Agreement, the Independent Contractor shall, to the extent requested from time to time by the Board or a Committee of the Board, perform consulting services for the benefit of the Corporation with respect to all of the above-listed matters and shall perform such additional services as may be agreed to by both parties.

SECTION FOUR

Compensation and Benefits

Subject to the provisions of this Section 4, the fixed non-bonus consideration to be furnished to the Independent Contractor (or to Seymour Holtzman if the Independent Contractor so directs) by the Corporation for the Services rendered by the Independent Contractor under this Amended and Restated Agreement shall consist of annual compensation of $551,000, payable in equal monthly installments.

4.1 Subject to Section 16 hereof, the Corporation shall reimburse the Independent Contractor, within thirty (30) days following receipt of documentation that satisfies the Corporation’s travel and expense reimbursement policies, but in no event later than the last day of the calendar year following the year in which the expense is incurred and documentation is submitted, an amount in cash equal to the actual and direct cost of all reasonable out-of-pocket expenses incurred by the Independent Contractor in the rendering of Services under this Amended and Restated Agreement. The amount of expenses eligible for reimbursement during one calendar year may not affect the expenses eligible for reimbursement in any other calendar year and the right of the Independent Contractor to

3

reimbursement is not subject to liquidation or exchange for any other benefit. The Independent Contractor hereby acknowledges that it has received in writing, read and understands the Corporation’s travel and expense reimbursement policies in effect as of the Effective Date.

4.2. The Independent Contractor is not eligible for participation in any current executive incentive or bonus program of the Corporation. However, the Board, at its own discretion, may choose to award a bonus to Independent Contractor if the Board determines that the Independent Contractor’s performance in carrying out its duties (as described in Section 3 of this Amended and Restated Agreement) has been outstanding, substantially exceeding the Board’s expectations. In no event shall any discretionary bonus payable to the Independent Contractor pursuant to this Section be paid later than the fifteenth (15^th) day of the third (3^rd) month following the year in which the Services were rendered with respect to which the discretionary bonus was determined.

SECTION FIVE

Duration

The term of this Amended and Restated Agreement shall expire on April 29, 2013 (the “Expiration Date”).

Commencing April 29, 2012, this Agreement shall automatically be extended for an additional one (1) year term on each anniversary date of the Effective Date unless either party shall give the other at least ninety (90) days written notice prior to such anniversary date that it will not extend the Term.

SECTION SIX

Complete Performance

The Independent Contractor agrees to fully perform its Services throughout the term of this Amended and Restated Agreement and further agrees to perform such Services in a manner that fulfills the spirit and purpose of this Amended and Restated Agreement. For the purposes of this Section 6 only, it shall be assumed by the parties that the compensation paid to the Independent Contractor for its Services from the Effective Date through the Expiration Date is earned at the per diem rate of $1,575 (the “Per Diem Rate”). If the Independent Contractor were to fail or refuse to completely perform its Services hereunder as a result of or based upon circumstances that are within the Independent Contractor’s control, the Corporation shall be entitled, upon written demand (the “Penalty Notice”), to receive from the Independent Contractor 150% of the Per Diem Rate to have been earned by the Independent Contractor from the date of the Penalty Notice to the Expiration Date.

4

SECTION SEVEN

Place of Work

It is understood that the Services shall be rendered primarily from the Independent Contractor’s offices in Wilkes-Barre, Pennsylvania and Palm Beach, Florida, but that any approved agent or employee of the Independent Contractor shall, upon request of the Corporation, travel to the Corporation’s executive offices located at 555 Turnpike Street, Canton, Massachusetts, or such other places as may be designated by the Corporation.

SECTION EIGHT

Time Devoted To Work

In performing the Services, the hours that approved agents and employees of the Independent Contractor work on any given day shall be entirely within the Independent Contractor’s control and the Corporation shall rely upon the Independent Contractor to determine the number of hours as is reasonably necessary to fulfill the spirit and purpose of this Amended and Restated Agreement.

SECTION NINE

Status of Independent Contractor

The Independent Contractor and the Corporation acknowledge and agree that the Independent Contractor shall perform the Services hereunder as an “independent contractor” and not as agent or employee of the Corporation, and nothing herein shall be construed to be inconsistent with this relationship or status. Except as provided herein, it is agreed between the parties hereto that the Independent Contractor is solely responsible for all labor and expenses in connection with the performance of every obligation of the Independent Contractor hereunder. The Independent Contractor assumes the responsibility for furnishing the Services hereunder and shall withhold and pay when due all employment taxes required by federal, state and local laws, including, without limitation, all social security and withholding taxes, and contributions for unemployment and compensation funds. The Independent Contractor acknowledges and understands that the Corporation will not maintain worker’s compensation, health or liability insurance on behalf of the Independent Contractor.

SECTION TEN

Materials and Equipment

Except as provided herein, the Independent Contractor shall furnish, at its own expense, all materials and equipment necessary to carry out the terms of this Amended and Restated Agreement.

5

SECTION ELEVEN

Work Standards

The Independent Contractor shall adhere to professional standards and shall perform all Services required under this Amended and Restated Agreement in a manner consistent with generally accepted procedural standards.

SECTION TWELVE

Copyrights and Patents

The Corporation shall own all copyrights and/or patents developed by the Independent Contractor while performing the Services provided under this Amended and Restated Agreement. All improvements, discoveries, ideas, inventions, concepts, trade names, trademarks, service marks, logos, processes, products, computer programs or software, subroutines, source codes, object codes, algorithms, machines, apparatuses, items of manufacture or composition of matter, or any new uses therefore or improvements thereon, or any new designs or modifications or configurations of any kind, or work of authorship of any kind, including without limitation, compilations and derivative works, and techniques (whether or not copyrightable or patentable) conceived, developed, reduced to practice or otherwise made by the Independent Contractor, or any of the Independent Contractor’s agents or employees, and in any way related to the rendering of Services under this Amended and Restated Agreement, shall become property of the Corporation. The Independent Contractor agrees to assign, and hereby does assign (and hereby agrees to cause its agents and employees to assign), to the Corporation any and all copyrights, patents and propriety rights in any such invention to the Corporation, together with the right to file and/or own wholly without restrictions applications for United States and foreign patents, trademark registration and copyright registration, and any patent, or trademark or copyright registration issuing thereon.

SECTION THIRTEEN

Privileged and Confidential Information

13.1 The Corporation and the Independent Contractor acknowledge that the Corporation has acquired and developed, and will continue to acquire and develop, information related to its business and its industry which is secret and confidential in character and is and will continue to be of great and unique value to the Corporation and its subsidiaries and affiliates. The term “confidential information” as used in this Amended and Restated Agreement shall mean all trade secrets, propriety information and other data or information (and any tangible evidence, record or representation thereof), whether prepared, conceived or developed by an employee of the Corporation or received by the Corporation from an outside source (including the Independent Contractor), which is in the possession of the Corporation, which is maintained in confidence by the Corporation or any subsidiary or affiliate of the Corporation or which might permit the Corporation or any subsidiary or affiliate of the Corporation or any of their respective customers to obtain a competitive advantage over

6

competitors who do not have access to such trade secrets, proprietary information, or other data or information, including, without limitation, information concerning the Corporation’s seasonal product line plans, store and brand image and trade dress developments and strategies, business plans, real estate leasing terms, conditions and plans, occupancy costs, customers, suppliers, designs, advertising plans, marketing plans, merchandising plans, market studies and forecasts, competitive analyses, pricing policies, employee lists, and the substance of agreements with landlords, tenants, subtenants, customers, suppliers and others. The term “confidential information” also includes information that the Corporation has in its possession from third parties, that such third parties claim to be confidential or proprietary, and which the Corporation has agreed to keep confidential. However, the term “confidential information” as used in this Amended and Restated Agreement shall not include information that is generally known to the public or in the trade as a result of having been disclosed by the Corporation in a press release or in a filing by the Corporation with the U.S. Securities and Exchange Commission. The Independent Contractor shall keep and maintain all confidential information in complete secrecy, and shall not use for itself or others, or divulge to others, any knowledge, data or other information relating to any matter which is confidential information relating to the Corporation obtained by the Independent Contractor as a result of its Services, unless authorized in writing by the Corporation in advance of such use or disclosure. All written information made available to the Independent Contractor by the Corporation, which concerns the business activities of the Corporation, shall be the Corporation’s property and shall, if requested in writing by the Corporation, be delivered to it on the termination or expiration of this Amended and Restated Agreement.

13.2 The Independent Contractor acknowledges that money alone will not adequately compensate the Corporation for breach of any confidentiality agreement herein and, therefore, agrees that in the event of the breach or threatened breach of such agreement, in addition to other rights and remedies available to the Corporation, at law, in equity or otherwise, the Corporation shall be entitled to injunctive relief compelling specific performance of, or other compliance with, the terms hereof, and such rights and remedies shall be cumulative.

SECTION FOURTEEN

Indemnification

14.1 The Independent Contractor shall defend, indemnify and hold harmless the Corporation (including, without limitation, the Corporation’s successors, assigns, subsidiaries, affiliates and contractors and their respective officers, directors, employees, agents and other representatives) from and against all liabilities, losses, claims, actions, damages, expenses (including but not limited to attorney’s fees), suits and assessments (whether proven or not) based upon or arising out of damage or injury (including death) to persons or property caused by the Independent Contractor in connection with the performance of Services, or based upon any violation of any applicable statute, law, ordinance, code or regulation. The Independent Contractor shall also defend, indemnify and hold harmless the Corporation against all liability and loss in connection with, and shall assume full responsibility for, payment of all federal, state, or local income taxes imposed or required under applicable laws with respect to Services performed and compensation paid the Independent Contractor under this Amended and Restated Agreement.

7

14.2 Notwithstanding anything contained in the preceding paragraph, the Corporation shall defend, indemnify and hold harmless the Independent Contractor (including, without limitation, any officers, directors, employees, agents and other representatives of the Independent Contractor and the Independent Contractor’s successors, assigns, subsidiaries, affiliates and contractors and their respective officers, directors, employees, agents and other representatives) to the fullest extent allowed by the law from and against all liabilities, losses, claims, actions, damages, expenses (including but not limited to attorney’s fees), suits and assessments (whether proven or not) based upon or arising out of damage or injury (including death) to persons or property caused by the Corporation in connection with the Corporation’s performance of its obligations under this Amended and Restated Agreement (including, but not limited to, claims based upon the material supplied to the Independent Contractor by the Corporation and utilized by the Independent Contractor in performing the Services), or based upon any violation of any applicable statute, law, ordinance, code or regulation. In the event that the Independent Contractor is made, or threatened to be made, a party to any legal action or proceeding by reason of the fact that the Independent Contractor is or was an independent contractor of the Corporation or serves or served any entities in any capacity at the Corporation’s request, the Independent Contractor shall be indemnified by the Corporation and to the extent permitted by law, such indemnification shall include payment by the Corporation, in advance of the final disposition of a civil or criminal action, suit or proceedings, of expenses incurred by the Independent Contractor (including, but not limited to attorney’s fees) in defending such action, suit or proceeding upon receipt of any undertaking by or on behalf of the Independent Contractor to repay such payment if it shall be ultimately concluded that the Independent Contractor is not entitled to be indemnified by the Corporation. The Corporation may accept any such undertaking without reference to the financial ability of the Independent Contractor to make such repayment.

SECTION FIFTEEN

Compliance with Laws

The parties agree that all obligations to be performed by the parties under this Amended and Restated Agreement shall be performed in compliance with all then applicable federal, state and local laws and regulations.

SECTION SIXTEEN

Approvals

16.1 In addition to approvals required by other Sections of this Amended and Restated Agreement, the Independent Contractor shall seek to obtain the Corporation’s written approval in advance of all expenditures in excess of four thousand dollars ($4,000.00) incurred in connection with the rendering of Services and for which the Independent Contractor seeks reimbursement from the Corporation. In addition, all estimates presented to the Corporation by the Independent Contractor for the Corporation’s consideration and/or approval shall be carefully prepared and shall be based upon reasonable assumptions using the Independent Contractor’s best judgment.

8

16.2 All approvals by the Corporation must be in writing and shall be sought from the President and Chief Executive Officer of the Corporation, or such other person that the Board may designate in writing from time to time. As of the date of this Amended and Restated Agreement the President and Chief Executive Officer of the Corporation is David Levin. If the Corporation fails to approve in writing any matter submitted for approval within fifteen (15) days from the date of its submission, then the matter submitted for approval shall be deemed to be disapproved.

SECTION SEVENTEEN

Notices

All notices and other communications required or permitted to be given under this Amended and Restated Agreement by one party to another shall be in writing and the same shall be deemed effective when delivered (i) in person, (ii) by United States certified or registered first class or priority mail, return receipt requested, (iii) by nationally-recognized overnight delivery or courier service, or (iv) by facsimile transmission 781-828-3221 for the Corporation, and 570-820-7014 for the Independent Contractor), and addressed to the party’s principal offices set forth on page one of this Amended and Restated Agreement, or at such other address or facsimile telephone number as may be designated in writing by such party to the other in accordance with the requirements of this Section 17.

SECTION EIGHTEEN

Governing Law

The place of this Amended and Restated Agreement, its status, or forum is at all times in the County of Norfolk, Commonwealth of Massachusetts, in which County and Commonwealth all matters, whether sounding in contract or in tort relating to the validity, construction, interpretation, and enforcement of this Amended and Restated Agreement, shall be determined. This Amended and Restated Agreement shall be construed and enforced according to the laws of Massachusetts without regard to its principles of conflicts of laws. Any action on the Amended and Restated Agreement or arising out of its terms and conditions shall be instituted and litigated in the courts of the Commonwealth of Massachusetts. In accordance, the parties submit to the jurisdiction of the courts of the Commonwealth of Massachusetts. The prevailing party in any such litigation shall be entitled to recover its reasonable attorneys’ fees in addition to any damages that may result from a breach of this Amended and Restated Agreement.

9

SECTION NINETEEN

Miscellaneous

This Amended and Restated Agreement may not be modified, amended, or waived, except by a writing executed by both parties hereto. This Amended and Restated Agreement, and all attached or referenced schedules, exhibits and attachments, constitutes the full and entire understanding and agreement between the two parties with regard to the subject matter hereof and supersedes all prior agreements and understandings, whether written or oral, relating to the subject matter. The section headings herein are for convenience of reference only, are not part of this Amended and Restated Agreement and shall have no effect on the interpretation of this Amended and Restated Agreement or the provisions hereof. Neither this Amended and Restated Agreement nor any interest therein, or claim thereunder, shall be assigned or transferred by the Independent Contractor to any party or parties. If any provision of this Amended and Restated Agreement shall to any extent be invalid or unenforceable, such invalid or unenforceable provision shall be reformed to the extent required to make it valid and enforceable to the maximum extent possible under law, and the remainder of this Amended and Restated Agreement shall not be affected thereby, with each provision hereof being valid and enforceable to the fullest extent permitted by law. This Amended and Restated Agreement shall be binding upon, and inure to the benefit of, the parties and their respective successors and permitted assigns. This Amended and Restated Agreement may be executed in one or more counterparts, each of which shall be deemed an original and all of which together shall constitute one and the same agreement.

IN WITNESS WHEREOF, the parties have signed, sealed and delivered this Amended and Restated Consulting Agreement in duplicate, each of which is deemed an original, as of the Effective Date.

10