Exhibit 99.1

CASUAL MALE RETAIL GROUP, INC.

REPORTS FOURTH QUARTER AND FISCAL 2010 RESULTS

Fiscal 2010 EPS $0.32 vs. $0.14 Last Year

Guidance for Fiscal 2011 EPS $0.40-$0.45

CANTON, MA, (March 17, 2011) — Casual Male Retail Group, Inc. (NASDAQ: “CMRG”), the largest retailer of big & tall men’s apparel and accessories, today reported operating results for the fourth quarter and fiscal year ended January 29, 2011 (“fiscal 2010”).

Fourth Quarter Highlights (4QFY10 vs. 4QFY09)

Fiscal 2010 Highlights (FY10 vs. FY09)

David Levin, President and CEO, stated, “With essentially a flat sales base, we were able to double our operating income to $16.2 million from $8.0 million last year and significantly grow earnings per share to $0.32. In 2011, we are expecting continued improvement to earnings per share in the range of $0.40-$0.45. We anticipate our customer visits will slowly recover from the economic downturn and produce comparable sales growth of 4.0%-4.5%, gross profit margins will remain strong and improve slightly, and SG&A will increase only modestly. Furthermore, based on the strong initial performance of the DXL test stores in 2010, and overwhelming positive customer feedback, we are pursuing expansion of DXL with an additional 10-14 locations and the launch of a new, functionally rich DXL shopping website by the second quarter of 2011 providing a multi-channel DXL experience to our loyal customer base.”

Sales

In the fourth quarter, total sales increased 0.7% to $111.5 million and comparable sales increased 2.9% as compared to the fourth quarter of fiscal 2009. For fiscal 2010, total sales decreased 0.4% to $393.6 million and comparable sales increased 1.5% as compared to fiscal 2009.

The comparable sales increase of 1.5% for fiscal 2010 consisted of a 0.5% increase in sales from the Casual Male XL business and a 2.7% increase in the Rochester business. On a comparable basis, sales from the Company’s direct businesses increased by 3.2% and sales from its retail business increased 1.1%.

While traffic was down 4.2% for fiscal 2010, it was a substantial improvement from the -9.7% and -9.2% experienced in fiscal 2009 and fiscal 2008, respectively. Traffic trends were flat during the fourth quarter of 2010. In addition, the Company continued to see improvements each quarter from its sales productivity in its retail channel as measured by improved conversion rate (the percentage of store customers who make a purchase) and dollars spent per customer transaction.

Gross Profit Margin

In the fourth quarter, gross profit margin decreased 100 basis points to 45.4%. The decrease was the result of a 150 basis point decrease in merchandise margin, partially offset by a 50 basis point improvement in fixed occupancy costs. The decrease in merchandise margin was due to the impact of liquidating markdowns associated with closing certain stores.

For fiscal 2010, gross profit margin increased 166 basis points to 45.8%. The increase was driven by a 96 basis point improvement in merchandise margins and a 70 basis point improvement in fixed occupancy costs.

SG&A

SG&A expenses, on a dollar basis, decreased 4.7% to $41.1 million in the fourth quarter of fiscal 2010 as compared to last year’s fourth quarter. As a percentage of sales, SG&A expenses declined 210 basis points to 36.8%. This decrease was primarily due to the timing of bonus accruals compared to last year.

For fiscal 2010, SG&A expenses remained flat at $150.9 million as compared to fiscal 2010. The Company continued to realize cost savings from several of its 2009 cost reduction initiatives, such as reductions in store payroll and lower distribution costs. These cost savings absorbed increases in expenses principally related to the four new DXL™ stores.

Interest Expense

Net interest expense for the fourth quarter of fiscal 2010 was flat when compared to the fourth quarter of fiscal 2009.

For fiscal 2010, net interest expense decreased $0.4 million to $0.7 million as compared to $1.1 million in fiscal 2009. The continued reduction in interest costs was due to a total repayment of both the Company’s long-term debt as well as its credit facility. The Company’s average outstanding borrowings under its credit facility were $5.2 million during fiscal 2010 as compared with average borrowings of $32.1 million in fiscal 2009.

Provision for Income Taxes

Net income for fiscal 2010 was positively impacted by $0.8 million, or $0.02 per diluted share, due to the reversal of previously reserved tax matters.

Cash Flow

Cash flow from operating activities was $19.0 million for fiscal 2010 as compared to $30.8 million for fiscal 2009. Free Cash Flow (as defined below) in fiscal 2010 was $10.0 million as compared to $26.2 million in fiscal 2009. The decrease in free cash flow for fiscal 2010 as compared to fiscal 2009 of $16.2 million is due to a swing of $20.4 million in operating assets and liabilities and an increase in capital expenditure of $4.4 million. These decreases were partially offset by an increase in cash flow from operations before changes in operating assets and liabilities of $8.6 million. It is important to note that the swing of $20.4 million primarily relates to working capital changes. Specifically, an increase in inventory and, due to the Company’s excess cash position in fiscal 2010 as compared to 2009, a reduction in accounts payable primarily due to the timing of cash payments.

Balance Sheet & Liquidity

During the fourth quarter of fiscal 2010, the Company amended and restated its current credit facility with Bank of America, N.A., as Agent. The new agreement provides for a maximum committed borrowing of $75 million, which, pursuant to an accordion feature, may be increased to $125 million upon the request of the Company and the agreement of the lender(s) participating in the increase. The maturity date of the amended and restated credit facility is November 10, 2014.

Also during the fourth quarter, the Company prepaid, with no penalty, the remaining balance outstanding under its long-term notes of $2.7 million. As a result, at January 29, 2011, the Company had no outstanding debt and full availability under its credit facility of approximately $63.0 million.

Inventories increased 3.2% to $92.9 million at the end of fiscal 2010 from $90.0 million at the end of fiscal 2009. The increase in inventory is primarily due to a shift in the Company’s merchandise receipts to receive selected merchandise earlier, especially imports, to avoid out of stock positions and delays to the stores.

Destination XL

During fiscal 2010, the Company has opened its first four Destination XL™ (also called DXL™) stores located in the following test markets: Schaumburg, IL; Memphis, TN; Houston, TX; and Las Vegas, NV.

For fiscal 2010, the four DXL stores collectively have met the Company’s expectations with sales increases of over 20%, when compared to the prior year’s predecessor stores from each of the respective DXL store market area. Once a DXL store has fully matured, which the Company estimates to be 36 months, EBITDA (Earnings Before Interest, Taxes, Depreciation and Amortization) margins are expected to be approximately 30%. Based on the strong performance of these four stores thus far, the Company anticipates opening an additional 10-14 new DXL stores in fiscal 2011. Depending on the real estate and market demographics for each of these store locations, the Company expects the size of each store to vary between 6,000 to 12,000 square feet, to accommodate each market. If the Company continues to have success with the DXL stores in fiscal 2011, the long-range plan currently envisions at least 75-100 DXL store locations by fiscal 2015.

Fiscal 2011 Outlook

For the fiscal year ending January 28, 2012, the Company is initially projecting the following, inclusive of the opening of 10-14 DXL stores:

Non-GAAP Measures

In addition to financial measures prepared in accordance with generally accepted accounting principles (GAAP), the above discussion refers to non-GAAP free cash flow of $10.0 million for fiscal 2010, $26.2 million for fiscal 2009 and estimated non-GAAP free cash flow of $16.0 million for fiscal 2011. The presentation of non-GAAP free cash flow is not a measure determined by GAAP and should not be considered superior to or as a substitute for net income or cash flows from operating activities or any other measure of performance derived in accordance with GAAP. In addition, all companies do not calculate non-GAAP financial measures in the same manner and, accordingly, “free cash flows” presented in this release may not be comparable to similar measures used by other companies. The Company calculates free cash flows as cash flow from operating activities less capital expenditures and less discretionary store asset acquisitions, if applicable.

Investors are invited to listen to a broadcast of the Company’s conference call to discuss its fiscal 2010 earnings results. The conference call will broadcast live today, Thursday, March 17, 2011 at 9:00 a.m. Eastern Time and can be accessed at www.casualmalexl.com and then clicking on the investor relations icon. The call will be archived online within one hour after its completion. Participating in the call will be David Levin, President and Chief Executive Officer, and Dennis Hernreich, Executive Vice President, Chief Operating Officer and Chief Financial Officer.

During the conference call, the Company may discuss and answer questions concerning business and financial developments and trends. The Company’s responses to questions, as well as other matters discussed during the conference call, may contain or constitute information that has not been disclosed previously.

Casual Male Retail Group, Inc., the largest retailer of big and tall men’s apparel with operations throughout the United States, Canada and Europe, operates 440 Casual Male XL retail and outlet stores, 4 Destination XL stores, 16 Rochester Clothing stores, and direct-to-consumer businesses which include several catalogs and e-commerce sites. The Company is headquartered in Canton, Massachusetts, and its common stock is listed on the NASDAQ Global Market under the symbol “CMRG.”

Certain information contained in this press release, including the Company’s expectations regarding fiscal 2011, constitutes forward-looking statements under the federal securities laws. The discussion of forward-looking information requires management of the Company to make certain estimates and assumptions regarding the Company’s strategic direction and the effect of such plans on the Company’s financial results. The Company’s actual results and the implementation of its plans and operations may differ materially from forward-looking statements made by the Company. The Company encourages readers of forward-looking information concerning the Company to refer to its prior filings with the Securities and Exchange Commission, including without limitation, its Annual Report on Form 10-K filed on March 19, 2010, that set forth certain risks and uncertainties that may have an impact on future results and direction of the Company.

Forward-looking statements contained in this press release speak only as of the date of this release. Subsequent events or circumstances occurring after such date may render these statements incomplete or out of date. The Company undertakes no obligation and expressly disclaims any duty to update such statements.

For additional information, contact:

Jeff Unger

Vice President of Investor Relations

Casual Male Retail Group, Inc.

561-482-9715

jeffunger@usa.net

[tables to follow]

CASUAL MALE RETAIL GROUP, INC.

CONSOLIDATED STATEMENTS OF OPERATIONS

(In thousands, except per share data)

CASUAL MALE RETAIL GROUP, INC.

CONSOLIDATED BALANCE SHEETS

January 29, 2011 and January 30, 2010

(In thousands)